6.1 Training, Validation and Test Datasets

A machine learning model has no built-in sense of “I already used this row to make a decision.” If you fit a classifier, pick its hyperparameters, and then quote a score on the same labeled examples, you are grading homework while the answer key is still on the desk. Learning objective AI-3.2.3 (K2) asks you to contrast the three roles labeled data plays while a supervised model is developed: training, validation, and test (hold-out). This is a design and process topic for testers, not a statistics derivation. Later syllabus material covers sample-size formulas; you are not expected to memorize those formulas here.

Three equivalent randomly selected sets

Logically, developing an ML model needs three equivalent collections, typically drawn at random from one representative labeled pool:

• A training dataset is used to train (fit) the model. The learning algorithm is allowed to see these feature vectors and their labels while it updates parameters — weights in a network, split thresholds in a tree, and similar internals.
• A validation dataset is used for evaluating and subsequently tuning the model. You compare candidate architectures, regularization strengths, learning rates, or early-stopping points here. In any one trial the model is not trained on those rows, but people and search scripts do use the validation scores to make choices. That is still a form of peeking, which is why validation is not the final score.
• A test dataset, also called the hold-out dataset, is used to test the tuned model. After you have committed to a recipe, you run that frozen model on hold-out examples you have not used for fitting or for choosing the recipe. That is the independent functional-performance check.

“Equivalent” is doing real work in that sentence. The three slices should look like draws from the same real-world distribution: same feature schema, similar class mix, similar noise, similar missingness. Random selection from a single representative dataset is the usual way to pursue that. If the source pool is already skewed — only weekday traffic, only one hospital, only one camera — no clever split repairs the mismatch with production.

Think of a spam filter with 10,000 labeled messages. You shuffle, then cut three slices. Each slice should still contain a mix of newsletters, invoices, phishing, and personal notes. If the “test” slice accidentally holds all the phishing mail, a terrible model can look excellent, or a good model can look broken, for reasons that have nothing to do with the algorithm.

How much data goes in each slice

When there is an abundance of suitable data, how much you spend on training versus evaluation versus testing typically depends on:

• The expected complexity of the model (a deep net with millions of parameters usually needs more training examples than a tiny linear classifier before it stops underfitting).
• The algorithm used to train it (some methods are data-hungry; some saturate earlier).
• Resources: RAM, disk space, computing power, network bandwidth, and calendar time. A split that looks elegant on a whiteboard can be unusable if each training run takes a weekend or the feature store cannot ship another terabyte.
• The desired confidence in the resultant model. More held-out cases generally give a stabler estimate of functional performance, but every example you hold out is an example the learner does not see during fitting.

There is no single official percentage you must recite on the exam. A diagram labeled “70 / 15 / 15” is a scenario, not a law. Testers should ask why a team chose a split, whether the slices stayed equivalent, and whether the hold-out stayed untouched — not whether the ratios match a blog post.

When labeled data is limited

A rigid three-way split on a small pool can leave too little training data for the intended model. The practical failure mode is underfitting: the learner never sees enough variety to capture the patterns you care about, so even training metrics stay weak, and production behavior is worse.

A common strategy, when it is feasible, is:

1. Set aside a small final hold-out test set and lock it in a vault. Nobody uses it for fitting, for model selection, or for “just one more look.”
2. Treat everything else as a combined training-and-validation pool.
3. Estimate tuning quality on that pool with k-fold cross-validation, where k is a user-specified integer, commonly 5 or 10.

Walking through k-fold without turning it into a formula sheet

Suppose you have 1,000 labeled defect images. You park 100 as hold-out. The remaining 900 become five folds of 180 images each (k = 5).

• Fold 1: train on folds 2–5 (720 images), validate on fold 1 (180).
• Fold 2: train on folds 1 and 3–5, validate on fold 2.
• Repeat until each fold has been the validation slice once.

You now have five validation scores — accuracy, F1-score, or whichever functional metric the risk analysis named. Average them. That average is a more stable picture of “how this recipe behaves on data that was not used in that particular fit” than a single 80/20 cut that happened to be lucky or unlucky.

Stratified sampling is the usual way to assign rows to folds when classes are imbalanced or the dataset is small. Stratified here means each fold keeps approximately the same class mix as the pool. If 4% of images are cracks, you do not want one fold with 0% cracks and another with 12%. Random assignment without stratification can do exactly that on small or skewed data.

After cross-validation identifies hyperparameters (depth, learning rate, regularization, decision threshold policy, and similar knobs), you typically train a final model on the entire training-and-validation pool — all data except the hold-out — using those chosen hyperparameters. Then you evaluate once on the hold-out. That single number is the unbiased (relative to your tuning process) functional-performance assessment the syllabus is pointing at.

What if you cannot afford a hold-out

If extreme scarcity makes even a small hold-out infeasible, the average cross-validation performance is optimistically biased. You already used every example both to train in some folds and to validate in others, and you likely chose the recipe that looked best on those same examples. That average can still be useful for comparing recipes, but it cannot serve as an unbiased estimate of how the shipped model will behave on new cases.

Two other resampling methods appear in the same limited-data toolkit:

• Leave-one-out cross-validation is the special case of k-fold where k equals the number of samples. Each example is a singleton validation fold; the model trains on everyone else. It spends training data very aggressively and is computationally heavy.
• Bootstrap techniques resample with replacement to form training draws and evaluate on examples left out of a given draw. Treat this as a named alternative you can contrast, not as a set of equations to derive on exam day.

The leakage rule testers exist to enforce

Do not leak test (hold-out) data into training or tuning. Leakage is often mundane rather than theatrical:

• Duplicate images sitting in both the training folder and the hold-out folder.
• Time-series rows shuffled so that “future” sensor windows teach the model to predict the past.
• Feature engineering fitted on the full file (scaling, imputation, rare-category maps) before the split, so hold-out statistics leaked into the transformers.
• Early stopping, threshold picking, or “we tweaked one more time” decisions that used hold-out dashboards.
• Data augmentation that copies hold-out images into the training stream.

From a tester’s chair, contrasting the three sets means asking: what was this slice allowed to influence? Training may change parameters. Validation may change choices. The hold-out may change nothing except the number you report. If a pipeline cannot answer those three sentences, the split is a slideshow, not a control.

A compact k-fold picture you can narrate on the exam

You do not need a statistical derivation. You do need the control flow: remaining data after the hold-out is cut into k folds; each fold is the validation slice once; metrics are averaged; the winner’s hyperparameters go into one final fit on everything except hold-out; hold-out is read once.

If someone reports “our CV F1 is 0.91, so we skipped a test set,” you should hear optimistic bias. If someone says “we used k = 5, then trained the winner on the hold-out too, then quoted that same hold-out score,” you should hear leakage. If class imbalance is severe and folds were cut with a plain shuffle, you should ask whether stratified assignment was used so rare positives were not accidentally missing from some validation folds.

Abundance versus scarcity is the contrast AI-3.2.3 is built around. Plenty of data: three equivalent random slices, with sizes driven by complexity, algorithm, resources, and how much confidence you need in the final check. Limited data: protect a small hold-out if you can; use k-fold (often 5 or 10) on the rest; stratify when imbalance or small n would make random folds unrepresentative; retrain the chosen recipe on the pool; test once. No hold-out: say out loud that the CV average is optimistically biased, and know that leave-one-out (k = n) and bootstrap exist as further resampling tools — without turning the item into a sample-size calculation.