6.2 Calculating Classification Metrics

Classification models almost never get every case right. Labels are noisy, features overlap, and the learner is probabilistic. Learning objective AI-3.3.1 (K3) is not “list four metric names.” It is calculate common ML functional performance metrics from a given confusion matrix. On the exam this is treated as a two-point calculation item: you will be handed counts, you will apply the syllabus formulas, and you will interpret what the numbers mean for risk.

The four cells you must never mix up

For a binary problem, every prediction falls into one of four buckets:

• True positive (TP): actual positive, predicted positive. The screening test flags a patient who really has the condition; the filter files a message that really is spam.
• False positive (FP): actual negative, predicted positive. A healthy patient is flagged; a legitimate invoice is dumped in junk.
• False negative (FN): actual positive, predicted negative. A sick patient is cleared; a phishing mail lands in the inbox.
• True negative (TN): actual negative, predicted negative. A healthy patient is cleared; a normal mail stays in the inbox.

A confusion matrix is just a 2×2 arrangement of those four counts. Figures often put actual classes on one axis and predicted classes on the other. The axes may be swapped from one textbook or dashboard to the next. That presentation change does not invent new categories. You still extract TP, FP, FN, and TN, then plug them into the formulas. If an exam figure looks “upside down” compared with the last tutorial you watched, relabel the four cells before you divide. Do not invent a fifth cell.

Formulas exactly as the CT-AI v2.0 syllabus states them

Write these until they are automatic. Precision and recall are used as percentages (0–100) when you compute F1-score the way the syllabus writes it; using 0–1 proportions and then scaling yields the same F1-score.

• Accuracy = (TP + TN) / (TP + TN + FP + FN) * 100%
  Accuracy is the percentage of all classifications that were correct — both positives and negatives.
• Precision = TP / (TP + FP) * 100%
  Precision is the proportion of predicted positives that were actually positive. It answers: when the model raises an alarm, how sure can I be?
• Recall (also called sensitivity) = TP / (TP + FN) * 100%
  Recall is the proportion of actual positives that were predicted positive. It answers: how confident can I be that positives are not being missed?
• F1-score = 2 * (Precision * Recall) / (Precision + Recall)
  F1-score is the harmonic mean of precision and recall, ranging from 0 to 100. A value near 100 means both precision and recall are high, so false positives and false negatives have little impact. A low F1-score means the model struggles to identify positives — missing true cases, generating many false alarms, or both.

Accuracy can be high while F1-score is poor. That is the trap this K3 item is built to catch.

Worked example 1: medical screening on an imbalanced set

A clinic screens 1,000 people for a rare condition. Only 20 people truly have it (2% prevalence). A “model” that predicts negative for everyone scores:

• TP = 0, FP = 0, FN = 20, TN = 980
• Accuracy = (0 + 980) / 1000 * 100% = 98%
• Recall = 0 / (0 + 20) * 100% = 0%
• Precision is undefined if you never predict positive (TP + FP = 0), which is already a testing smell: the positive class was never exercised.

Ninety-eight percent accuracy looks like a celebration slide. The system missed every sick patient. Testers who stop at accuracy would ship a useless screener.

Now a real detector flags 100 people. The confusion matrix is:

Check the total: 18 + 2 + 82 + 898 = 1,000.

• Accuracy = (18 + 898) / 1000 * 100% = 916 / 1000 * 100% = 91.6%
• Precision = 18 / (18 + 82) * 100% = 18 / 100 * 100% = 18%
• Recall = 18 / (18 + 2) * 100% = 18 / 20 * 100% = 90%
• F1-score = 2 * (18 * 90) / (18 + 90) = 3240 / 108 = 30

Read that out loud. Accuracy is still in the nineties. Recall is high: 90% of the truly sick people were flagged. Precision is 18%: five out of six alarms are false. F1-score is only 30 because the harmonic mean punishes the weak precision. In screening, missing a true case (FN) may be the unacceptable harm, so a team might still prefer recall and accept a pile of follow-up tests. They must not quote 91.6% accuracy as if the positive predictions were trustworthy.

Worked example 2: spam filter where false alarms bury real mail

A mailbox has 200 labeled messages. 60 are spam; 140 are legitimate. The filter’s matrix is:

• TP = 36 (spam caught)
• FP = 4 (legitimate mail marked spam)
• FN = 24 (spam that reached the inbox)
• TN = 136 (legitimate mail left alone)

Total: 36 + 4 + 24 + 136 = 200.

• Accuracy = (36 + 136) / 200 * 100% = 172 / 200 * 100% = 86%
• Precision = 36 / (36 + 4) * 100% = 36 / 40 * 100% = 90%
• Recall = 36 / (36 + 24) * 100% = 36 / 60 * 100% = 60%
• F1-score = 2 * (90 * 60) / (90 + 60) = 10800 / 150 = 72

Here false positives are expensive: an auto-deleted invoice or a missed customer email is worse, for many users, than deleting one extra phishing note by hand. Precision is the metric that matches that harm model. Recall of 60% means two in five spam messages still arrive; that may be acceptable if the alternative is burying payroll notices. F1-score 72 sits between the two: it is a compromise statistic, not a substitute for naming which error type the product can tolerate.

If you reversed the costs — a filter for a high-risk fraud queue where missing spam-like fraud is catastrophic — you would lean the other way, toward recall, the same way the clinic did. The formulas do not choose the product risk; they quantify the two error directions so you can.

Side-by-side so the imbalance lesson sticks

Accuracy ranked the always-negative screener first. That ranking is why CT-AI makes you calculate the rest.

How to sit a two-point calculation item

1. Label TP, FP, FN, TN from the figure even if predicted/actual are swapped.
2. Write the four syllabus formulas before plugging in numbers.
3. Compute accuracy, precision, and recall as percentages.
4. Compute F1-score from those precision and recall percentage values unless the item clearly uses 0–1 proportions — the syllabus form is the percentage form, and both agree if you are consistent.
5. Sanity-check: precision uses FP in the denominator; recall uses FN. Swapping FP and FN is the most common arithmetic way to lose both points.
6. Interpret: if classes are imbalanced, do not let accuracy be the only sentence in your answer.

You will see library helpers in hands-on work. The exam still expects you to calculate by hand from the four counts. Practice until 2 × (P × R) / (P + R) is muscle memory.