10.3 Label Correctness Testing

Labels as the teacher in supervised learning

Supervised learning treats labels as the teacher. If the teacher is wrong, the model still learns: it learns the wrong association. Label correctness testing checks whether those teachers are accurate and consistent. Inaccurate or inconsistent labels undermine performance and generalization. A fraud tag applied to legitimate late-night grocery orders will teach the model that night shopping is suspicious. A missed pedestrian box in a driving dataset will teach the model that some people are background.

This section is about how testers find those defects. A larger network will not reliably average them away. The wrong label is a persistent target.

Expert review

Domain experts or trained annotators review a sample of labelled data against written guidelines. The sample is not a casual scroll. It has a size, a sampling plan, and a recording form: guideline ID, agree or disagree, defect type. Experts catch systematic mistakes that metrics miss ("we labelled 'stopped at a light' as 'parked'"). They also expose guideline holes. Expert review is expensive, so it is most valuable early, when the labelling guide is still moving, and later on a risk-weighted sample rather than on every row.

Write the guidelines so a second expert could apply them without a hallway conversation. If two seniors need a meeting to decide whether a cardboard cut-out counts as a pedestrian, the guide is not finished, and expanding the annotator pool will multiply the confusion.

Multiple annotation and inter-annotator agreement

Give the same items to two or more annotators independently, then compare. This is a form of back-to-back testing of annotators (see 6.1.10). Disagreements are defects that need investigation, not noise to ignore.

Measure inter-annotator agreement (IAA) with Cohen's Kappa or simple percentage agreement. Percentage agreement is intuitive: 87 of 100 pairs match, so agreement is 87%. It does not credit how often they would agree by chance. If 90% of images are "no pedestrian," two lazy annotators who always click "no pedestrian" look excellent on raw percent. Cohen's Kappa subtracts chance agreement: (observed agreement minus expected-by-chance agreement) divided by (1 minus expected-by-chance agreement). Kappa near 0 is chance-level; negative is worse than chance; values closer to 1 show agreement beyond chance.

Worked sketch: 100 images, most of them empty road. Annotators A and B both default to "empty" on the easy majority and only collide on the hard minority. Percent agreement can still look high while Kappa is mediocre. Low IAA is a signal to diagnose cause, not a signal to average the labels and move on:

• Bad labelling guidelines — vague class boundaries, missing examples, conflicting rules.
• Ambiguous data — a box that might be a bag or a crouched person; audio that might be two overlapping intents.
• Poor annotation — fatigue, inadequate training, incentive to go fast.

Do not "fix" low IAA by blindly taking a majority vote until you know which of the three you have. Majority vote on a bad guideline just cements the bad rule. Majority vote on genuinely ambiguous items may need a new class, a "cannot decide" bin, or a tighter photo spec rather than a forced winner.

Risk-based prioritization

Expert review and multiple annotation should not spend equal effort on every row. Risk-based prioritization picks samples by:

• Likelihood of mislabeling: ambiguous items, points near a boundary between classes, rare subclasses, poor sensor conditions.
• Impact on success or safety: a mislabelled pedestrian at night in an emergency-braking dataset outranks a mislabelled empty sky; a mislabelled dosage class outranks a mislabelled marketing-propensity flag.

Testers write the prioritization rule down. "We double-annotate all night-time frames with a person-sized blob near the curb, and 2% of the rest" is a test design. "We looked at some images" is not. Risk-based sampling is how a small expert budget still covers the labels that can hurt people or sink the product.

Label distribution comparison

When a similar dataset exists (prior year, sibling site, or a public set that actually covers the same task), compare label distributions. If last year's depot file is 8% delayed and this year's extract is 41% delayed with no operational story, you may have a labelling process change, a filter bug, or systematic mis-tags. Distribution comparison does not name the wrong rows, but it tells you to go looking. It is cheap enough to run on every new dump.

Automated rule-based tests

Some label structures admit hard rules. For object detection, a common example is: bounding boxes must not overlap (when the project forbids overlapping objects) and must not extend beyond the image. A box with xmax larger than image width is a machine-checkable defect. Other tasks have other rules: a span label in text cannot start after it ends; mutually exclusive classes cannot both be true; a polygon must be closed. Encode the rules and run them on every label, not on a sample. Rule tests do not replace experts for semantic mistakes ("that is a bicycle, not a motorcycle"), but they catch geometric and schema-level label defects at ML scale.

Treat the rule as part of the project's label contract. If two valid objects are allowed to occlude, "no overlap" is the wrong rule and will generate false failures. The test still belongs in the pipeline; the predicate follows the guideline, not a generic vision blog post.

Model loss analysis

During training, loss on a point is large when the model's prediction deviates substantially from the assigned label. High-loss points are candidates for mislabelling: the model cannot learn the given tag. Caveats matter. High loss can also mean a rare but correct case, a too-small model, or a corrupted feature. Use high loss to rank items for human review, not as an automatic relabel. The test idea is that the training process itself is a flashlight aimed at labels the current model finds impossible to fit.

A practical workflow is: train a first model on the current labels, export the highest-loss N examples per class, and send that slice to expert review. Many teams find a handful of inverted tags this way that IAA never saw, because both annotators made the same systematic mistake.

Model confidence

After you have a trained model, low prediction confidence on a point may mean the item is mislabeled, ambiguous, or outside the training distribution. Those three causes are not the same. A clean, well-centred photo of a new packaging design can be low-confidence because it is new (distribution), not because yesterday's annotator erred. Combine confidence with other evidence: if two annotators already disagreed and confidence is low, ambiguity is likely; if annotators agreed and loss is high, consider a guideline error or an unusual but valid case.

Do not equate low confidence with "the label is wrong." Do not equate high confidence with "the label is right." A confidently wrong model on a consistently wrong label is how systematic labelling bugs become production behaviour.

Combine approaches over the lifecycle

No single method is enough. A workable sequence:

1. Experts and a written guide early, before a crowd of annotators multiplies a bad rule.
2. Multiple annotation and IAA to test whether the guide is usable.
3. Risk-based extra review on safety-critical and boundary items throughout.
4. Automated rules on every batch as soon as label structure is stable.
5. Distribution checks whenever a new dump arrives.
6. Loss and confidence once models exist, feeding a review queue rather than silently rewriting labels.

Using the combination is itself the test strategy: each method is strong against a different failure mode. Guideline defects show up in IAA and expert review. Geometry defects show up in rules. Silent class flips show up in distributions. Individual howlers that survived the crowd show up in loss and confidence.