12.3 Drift Testing

Why operational models go stale

Learning objective AI-6.1.7 is K2: explain how drift testing is used on operational machine learning systems. The setting is not the first training run. It is the live system after deployment, when the world continues to move and the locked-in model does not automatically move with it. CT-AI v2.0 (ISTQB, 17 April 2026) splits that movement into two forms of drift. Mixing them up is the most common way to lose the item.

A model is a snapshot of relationships that were present in the data used to train and evaluate it. Operational inputs arrive later. People change how they shop, attackers invent new phishing kits, seasons rotate, and lawmakers rewrite what "allowed" means. Drift testing is how testers and operators notice that the snapshot no longer describes the live stream. Chapter 5 already warned that after deployment you should monitor operational input properties for changes that might indicate data drift from the original training distributions. This section is the model-testing counterpart: what you call the two forms, and which test design needs fresh labels.

Data drift: the inputs changed

Data drift occurs when the statistical properties of the operational input data change over time. The input data is now significantly different from the data the model was trained on. Typical drivers are shifts in user behavior and seasonality. The syllabus example is concrete: a spam filter encounters new types of phishing attacks that did not exist during its training.

Stay on the input side of the pair. The meaning of "spam" may still be "unsolicited deceptive mail." What changed is the mix of features arriving at the model: new URL patterns, new attachment types, new languages, more evening traffic than the training week contained, a January shopping vocabulary that September training never saw. The labels' semantics can be stable while the input distribution is not. That is still data drift.

Other pictures help on exam day. A vision model trained on summer street scenes starts receiving winter images with snow and different clothing. A voice model trained on headset audio starts receiving far-field kitchen noise. A fraud model trained on card-present retail starts receiving a wave of mobile-wallet taps. In each case you can talk about input statistics — frequencies, ranges, class mix of raw features — without yet claiming that the correct answer for a given feature vector has been rewritten.

Concept drift: the correct answer for those inputs changed

Concept drift occurs when the relationship between the input data and the correct output changes over time. The model's originally learned patterns or rules no longer reflect the current reality. The syllabus example is again concrete: due to new financial regulations, a transaction type previously considered low-risk might now be classified as high-risk. The meaning of the data has changed. The model's learned decision boundaries become outdated, and predictive accuracy declines even if the raw transaction fields look familiar.

This is not "we saw a new phishing template." This is "the same template, or the same transaction record, now ought to receive a different label." A medical coding model meets a new clinical guideline that recodes a procedure. A moderation model meets a policy change that now forbids a category it used to allow. A credit model meets a law that redefines a protected calculation. The input schema can be unchanged. The correct output function moved. That is concept drift.

If you remember only one contrast, remember this: data drift is about the operational INPUT distribution; concept drift is about the mapping from input to CORRECT output. Both can appear together. A regulation change can also change customer behavior, so input statistics and label meanings shift at once. Testers still name which form they are claiming, because the evidence and the fix differ. Retraining on more recent phishing examples addresses a data-drift story. Relabeling historical transactions to the new regulation and then retraining addresses a concept-drift story. Using the wrong story wastes the retraining cycle.

Why the distinction matters in a test strategy

Data drift often shows up first in input monitors: feature histograms, embedding distances, unexpected category codes, sudden language shifts. You can raise a suspicion before anyone labels a new batch. Concept drift often needs fresh truth: someone has to say what the output should be now. If you only watch inputs, you can miss a silent policy change that leaves the feature mix looking "normal" while every decision is legally wrong. If you only watch accuracy against old labels, you can miss that the old labels are the thing that expired. Drift testing programs that serve operational MLS therefore plan for both forms, and they pick dynamic or static techniques according to whether current ground truth is available.

Dynamic drift testing: you need current ground truth

Dynamic drift testing relies on feedback from the users, which provides the current ground truth. That current ground truth is compared with the model's output. The difference between the two is determined and compared against a threshold. If the difference exceeds the threshold, you have evidence of drift (and you then investigate which form). If it stays under the threshold, the live mapping still looks close enough to what users currently treat as correct.

The threshold is a test-design choice. Too tight, and ordinary noise in ratings looks like a crisis. Too loose, and a real policy change hides inside "acceptable error." Document the threshold with the same seriousness you would document an accuracy criterion: what signal, over what window, at what volume of feedback.

User feedback can be direct or indirect. The syllabus uses a film recommendation system for both.

• Direct feedback: the user rates a recommendation. Stars, thumbs, "not interested," an explicit correctness flag on a predicted spam folder — anything where the user states a judgment of the output. That rating is current ground truth for that impression.
• Indirect feedback: extracted from data on the films the user has watched. Nobody filled a score card. Behavior is the proxy: watched to the end, abandoned after ten seconds, clicked through, returned the item. Indirect feedback is cheaper at scale and noisier. People watch films for many reasons. You still get a current signal without a survey.

Dynamic testing is the right family when you can obtain that live truth stream. Recommendation, moderation with user appeals, fraud with later chargebacks, and clinical systems with later confirmed diagnoses all produce some form of delayed label. The delay matters: chargebacks arrive weeks later, so the "current" truth is current to the feedback clock, not to the millisecond of the prediction. Your comparison window has to respect that lag or you will flag drift that is only latency in the truth pipeline.

Dynamic testing does not require you to know in advance whether you are looking at data drift or concept drift. A spike in disagreement with users is the alarm. Root-cause analysis then asks: did the inputs change (new titles, new user cohorts, seasonality), or did the meaning of a good recommendation change (a policy that now forbids a genre), or both?

Static drift testing: no current ground truth required

Static drift testing is not dependent on the current ground truth. Instead it compares the statistical properties of the input and predicted-output data distributions using a test such as Kolmogorov-Smirnov. A significant difference in either of these distributions is an indicator that drift has occurred.

Read the last sentence in pieces. You compare distributions, not individual labeled cases. You look at inputs and at predicted outputs — the model's own current answers, not a fresh human label. A significant difference is an indicator, not a courtroom proof of which form of drift you have. Kolmogorov-Smirnov (KS) is the named example: a nonparametric comparison of two samples that asks whether they appear to come from the same distribution. In practice you might compare last month's operational feature distribution with the training or baseline window, and you might compare the distribution of predicted classes or scores across the same windows. If either comparison lights up, you investigate.

Static testing is what you still have when users do not rate, do not click, and do not send chargebacks — a batch scoring job, an embedded detector with no UI, a model whose predictions are consumed by another system. It is also what you run continuously as a cheap tripwire beside dynamic tests, because distribution checks do not wait for labels. The limitation is the flip side of the benefit: without current ground truth you can see that inputs look different, or that the model is emitting a different mix of predictions, and still not know whether the correct answers moved. A data-drift-like input shift and a concept-drift-like silent policy change can both move predicted-output histograms. Static testing says "look here." It does not, by itself, relabel the world.

Building an operational drift-testing loop

A practical loop for an operational MLS uses both families.

1. Baseline the training or go-live window: input statistics and predicted-output statistics you are willing to treat as the reference.
2. Static watches on live inputs and live predictions (KS or equivalent). Significant difference → ticket.
3. Dynamic watches wherever feedback exists: direct ratings and indirect behavioral traces, difference versus threshold.
4. Triage each ticket: data drift (new phishing, seasonality, behavior), concept drift (regulation, policy, meaning of the label), pipeline defect, or a threshold that was simply too tight.
5. Act with the matching remedy: more representative recent inputs, relabeling to the new concept, retraining, or a temporary reject/unknown policy while the concept is under review.
6. Retest after the change, because yesterday's baseline is no longer the baseline.

That loop is how drift testing is used, which is what K2 asks you to explain. It is not a training-time substitute for representativeness testing, and it is not adversarial testing. Adversarial examples can appear with a stable distribution. Drift can wreck accuracy without any human-imperceptible perturbation. Keep the techniques separate on the exam, then use both in a real model-test strategy.

Exam traps for AI-6.1.7

• Calling new phishing types concept drift. Unless the meaning of spam changed, that syllabus example is data drift (operational inputs).
• Calling a regulation that recategorizes low-risk to high-risk data drift. That syllabus example is concept drift (input-to-correct-output relation; decision boundaries outdated).
• Claiming dynamic testing can run without user feedback. It depends on current ground truth from feedback, compared with model output against a threshold.
• Swapping direct and indirect: rating a recommendation is direct; films actually watched are indirect.
• Claiming static testing needs fresh labels. It does not depend on current ground truth; it compares input and predicted-output distributions (Kolmogorov-Smirnov), and a significant difference indicates drift.
• Treating a significant KS result as a complete diagnosis. It is an indicator; you still separate data drift from concept drift in the investigation.