13.1 Testing for Overfitting and Underfitting

Why fit is a testing problem, not only a training notebook

Learning objective AI-6.1.8 is a K2 explain item: show how testers detect overfitting and underfitting, and how they recognize the third outcome, a right-fitting model. CT-AI v2.0 (ISTQB, 17 April 2026) puts this work inside model testing, not as an optional chart a data scientist might glance at after the last epoch. The syllabus is explicit about when the checks happen: during training, evaluation, and tuning. Those three activities together are model generation. If you wait until a frozen production model is already serving traffic, you have skipped the windows where a too-flexible network memorized quirks or a too-simple model never learned the job.

Every machine learning system (MLS) you test lands in one of three buckets. Overfitting means the model learned the training sample too well, including accidental noise, so it generalizes poorly to new data. Underfitting means the model never captured the underlying structure, so it is weak on data it was shown and weak on data held back. Right-fitting means the model learned the pattern that matters, with only a small gap between behavior on data used in model generation and behavior on data it has not seen. Your exam job is to name the three outcomes, name the three activities where you test for them, and name the evidence that separates overfit from underfit from right-fit.

The three outcomes in tester language

Overfitting: noise instead of pattern

Overfitting occurs when a model learns the training data too well. That phrase is not praise. The model captures noise in that sample rather than the underlying pattern that should apply to new cases. A claims-triage classifier that memorizes that last year's fraud ring always used the phrase "urgent wire today" will fire on that phrase and miss a new ring that uses different wording. Training metrics look excellent because the noise lived in the training file. Generalization to unseen data is poor.

That is why testing for overfitting is not "look at training accuracy and stop." Training success is the trap. A model that recites the training set is not a model that will survive next month's operational mix.

Underfitting: too simple, or the features were never there

Underfitting occurs in two syllabus-named ways, and multiple-choice items like to offer only one. First, the model is too simple to capture the structure of the data: a linear separator where the true relationship bends, a tiny decision tree that cannot represent a policy with several interacting conditions, a network with too little capacity for the task. Second, the training data does not contain features that reflect an important relationship between inputs and outputs. If the true driver of a medical-risk score is a lab ratio nobody collected, extra epochs will not invent that relationship. The observable result is the same either way: poor ML functional performance on both the training dataset and the validation dataset.

Right-fitting: a small generalization gap

A right-fitting model is the remaining outcome. It performs well on training data, on the validation data used during evaluation and tuning, and on an independent test set. The generalization gap — the drop from scores seen during model generation into truly unseen cases — stays small. Right-fit is not "perfect on every metric." It is "good against the agreed ML functional performance criteria, without the tell-tale splits that mark overfit or underfit."

When you test: training, evaluation, and tuning

Do not treat fit testing as a single gate after someone announces that training is finished. During training, you watch whether errors are actually falling. During evaluation, you measure ML functional performance on data that was not used to update weights in that step. During tuning — architecture choices, hyperparameters, feature sets — you can overfit the validation set itself by hunting for a configuration that only wins on that slice. That is why CT-AI still demands a separate test dataset that was not used during training (and, in a well-run workflow, was not used to steer tuning either). The three-way data split from earlier in the syllabus is the instrument. This learning objective is about the defects that split is meant to reveal.

If a team retunes after peeking at the test set, the test set has become another validation set. You can still run numbers. You can no longer claim you measured generalization on data that never influenced model generation.

How testers detect overfitting

To test for overfitting, evaluate the model's ML functional performance on a separate test dataset that was not used during training. That sentence is the detection method. Two extra requirements sit on the test set.

First, independence. If the rows you call "test" leaked into training or into the tuning loop, you are not measuring generalization. You are measuring a slightly shuffled training score.

Second, uncommon examples. The test dataset should include less common examples that were unlikely to have been used during training. Frequent cases can look fine even when the model only memorized common patterns. Rare but valid cases — an unusual product-return reason, an uncommon lab combination, a nighttime photo of a traffic sign, a loan applicant with a thin but legitimate credit file — are where memorization fails and a right-fitting model should still behave reasonably.

The decision rule is comparative: the model might be overfitting if it performs significantly worse on the test dataset than on the validation dataset. Notice the pairing. The contrast is test versus validation, not "training looked nice." Validation already participated in evaluation and tuning. A model can look acceptable on validation and still collapse on a true hold-out, especially when uncommon cases finally appear. "Significantly worse" is a tester judgment against the project's agreed metrics and thresholds, not a universal percentage the exam expects you to invent.

A worked picture: a spam filter reports 0.97 F1 on validation after a week of hyperparameter search. On a held-out test set that includes new phishing templates, odd character encodings, and low-volume sender domains, F1 drops to 0.71. That gap, on data the training loop never saw, is the overfitting signal. If instead F1 is 0.96 on validation and 0.95 on a similarly constructed test set, you do not have that signal. Other risks may remain. This is not the overfit pattern.

How testers detect underfitting

Underfitting is detected by evaluating ML functional performance metrics such as accuracy, precision, recall, or F1 score. If those metrics are consistently low on both the training set and the validation set, the model is underfitting. Both sides matter. Low validation with high training is a different story (often overfit during the training run). Low on both means the model never got a useful grip on the task.

Ask two follow-up questions when both sides are poor. Is the model too simple for the structure of the data? Or did data preparation omit features that carry an important input-output relationship? Testers do not have to retrain the model themselves, but they do have to report which of those two stories the evidence supports. A fraud model that never sees "claim amount divided by policy limit" cannot learn that ratio's effect. Extra training time will not create a missing column. A model with too few parameters cannot represent a policy that depends on several interacting conditions, even when every needed column is present.

Learning curves as a visual test

Visual inspection of learning curves is a second underfitting detector. Plot training error and validation error as training progresses. If training and validation errors remain high and relatively close together, without significant improvement, the picture is underfitting. The model is not pulling away from a bad start. A right-fitting run usually shows errors falling, then leveling, with a modest gap between the two curves. An overfitting run often shows training error still falling while validation error flattens or rises — a growing gap. For this learning objective, memorize the underfit curve in words: high, close, not improving.

Do not wait until the last epoch to glance at a single pair of numbers. The "without significant improvement" clause is about trajectory. A short, noisy start can look high and close; continued training that still does not move is the finding you log.

Putting a K2 answer on the page

A complete exam answer names five things. One: the three outcomes — overfit, underfit, right-fit. Two: testing during training, evaluation, and tuning, not only after freeze. Three: overfitting as noise-not-pattern with poor generalization, detected on an independent test set that includes uncommon examples, with a significant drop versus validation. Four: underfitting as too-simple or missing-features, with poor metrics on both training and validation, plus the high-and-close learning-curve signature. Five: right-fit as good train, validation, and test scores with a small generalization gap.

Exam traps for AI-6.1.8

• Testing for fit only after deployment. The syllabus requires the check during training, evaluation, and tuning.
• Declaring overfit because training accuracy is high. You need the independent test comparison, including uncommon examples.
• Comparing only training versus test and ignoring that validation is the syllabus's contrast partner for the "significantly worse" rule.
• Calling a model right-fit merely because training and validation errors are close. If both are high and stuck, that closeness is underfitting.
• Treating underfitting as "the model is always too simple." Missing features of an important relationship is an equally valid cause.
• Reusing the test set for tuning, then claiming you measured generalization.
• Confusing this LO with drift testing. Fit is about how the model learned from the datasets you have now. Drift is about operational data changing later.