11.1 ML Model Risks and Mitigations
Key Takeaways
- Testers choose ML model test techniques from the identified risk rather than running every technique on every release.
- Bias and unfair outcomes are mitigated with bias testing; unethical behavior is mitigated with ethical system testing; adversarial examples are mitigated with adversarial testing.
- When a cheap expected result is missing, metamorphic testing, back-to-back testing, and A/B testing are the usual oracle-problem mitigations.
- Model updates that add defects are checked back-to-back; updates that lower metrics are checked with A/B testing; immediate deploy failure is a smoke test; later breakage is regression testing.
- Security, safety, privacy, and harmful-output risks are mitigated with red teaming; unexpected inputs are probed with adversarial testing and fuzz testing.
Why testers match techniques to risks
Machine learning models fail in ways that a conventional function rarely does. A classifier can look strong on a dashboard and still treat one group of people worse than another. A reinforcement-learning agent can hit a numeric target while damaging equipment. A freshly trained checkpoint can beat last month's validation file and then collapse when this week's traffic mix arrives.
The ISTQB Certified Tester AI Testing (CT-AI) v2.0 syllabus treats those failures as model risks. Learning objective AI-6.1.1 (K2) asks you to give examples of test approaches used for risk mitigation of ML models. The verb is give examples, not apply every row of a table to every release. Testers pick techniques from the risk. Budget, labeled data, compute, and access to production all limit what you can run. If the dominant threat is unfair outcomes, you invest in bias testing. If the dominant threat is a brittle serving contract, you invest in API testing. The rest of the catalogue stays available until a risk appears that needs it.
Keep three risk families in mind. Functional risks include bias, overfitting, and adversarial vulnerability. Non-functional risks include weak AI robustness and poor performance efficiency. Deployment risks include a rollout that will not start, a silent metric drop, or a regression in an old user journey. The map in this section is a menu. The product in front of you tells you which rows matter today.
Unfairness, ethics, and adversarial examples
A biased or unfair model delivers systematically different quality of service for people or groups in ways the organization cannot justify. Bias is not a synonym for accuracy is low. Accuracy can look healthy while false-decline rates explode for a neighborhood that happens to correlate with a protected class. Bias testing is the matching mitigation. You slice metrics by relevant groups or proxies, compare error types rather than only averages, and inspect whether the training data encodes historical discrimination.
An unethical model may be statistically tidy and still be the wrong system to operate. A health chatbot that invents dosages, a biometric pipeline used beyond the consent that was collected, or a generator that produces sexual content involving minors is not a metric miss. Ethical system testing examines purpose, misuse, consent, and foreseeable harm. It checks whether real behavior matches the story told to users, buyers, and regulators.
Adversarial examples are inputs—often tiny perturbations of legitimate examples—that cause incorrect predictions. A few stickers on a stop sign, a crafted overlay on a face, or a carefully chosen token sequence can flip a decision. Adversarial testing searches for those inputs so the team can add defenses, reject suspicious traffic, or retrain. You will study the technique in more depth later; for this objective you need the pairing: adversarial examples as the risk, adversarial testing as the mitigation.
Fit, drift, side-effects, and reward hacking
An overfitted model memorizes training quirks and then fails on fresh cases. Training scores look excellent; independent scores look worse. An underfitted model never captured the signal: both training and test scores stay mediocre. Mitigations are overfitting tests and underfitting tests. Compare splits, watch whether a too-simple model plateaus far below the required metric, and treat a perfect training score with a weak independent score as an overfitting story.
Data drift is a change in the incoming feature world: a new camera sensor, generated cover letters, different fraud devices. Concept drift is a change in the meaning of the target: what counted as default or spam last year is not what counts now. Both are mitigated with drift testing on operational data—watching feature distributions and recent labeled performance so you can retrain, throttle, or roll back.
Side-effects are harms the objective never priced in. A routing model that shortens trips by sending trucks through a school zone at dismissal can pass a narrow accuracy test and still be unacceptable. Side-effects testing looks for collateral damage to safety, privacy, neighboring systems, and physical equipment.
Reward hacking is the loophole version of the same idea, especially in reinforcement learning. Pay an agent for distance cleaned and it may spin in a circle. Pay a recommender for clicks and it may learn outrage. Reward hacking testing hunts high-reward, low-intent trajectories: constraint violations, proxy-metric inflation, and strategies that would look indefensible if described in public.
Interfaces, missing oracles, and weak requirements
A model API defect is a contract failure around the served model: swapped class indices after export, preprocessing that does not match training, NaNs returned as success, or an endpoint that still loads last Tuesday's weights. API testing checks schema, status codes, authentication, versioning, and the path from raw payload to prediction.
Failure to achieve required accuracy, recall, precision, F1-score, or similar measures is a functional-performance risk. The mitigation is ML functional performance testing: measure the right metric on an independent set and compare it with the acceptance criterion. For probabilistic systems, that comparison is statistical, which section 11.3 develops.
Functional incorrectness and non-functional defects are broader than one number. The model may be inconsistent, fragile, or quietly wrong in unlabeled regions. MLS often add the test oracle problem: there is no cheap, trusted expected output for every case. You cannot hand-label a million images tonight. Mitigations that still produce verdicts include metamorphic testing (follow-up cases derived from a relation, such as a lossless rotation that should not delete a detected tumor), back-to-back testing (compare two implementations, pipelines, or versions on the same inputs), and A/B testing (compare treatments on traffic). When a question mentions a missing oracle, think of that trio.
Poor system requirements starve later tests. Slogans such as be fair or work like a senior analyst do not tell you when to fail the build. Mitigation starts with requirements review—pushing for users, out-of-scope uses, measurable thresholds, and data constraints—and with red teaming, which tries to break the product in ways the happy-path specification never listed. Thin specifications also invite charter-based exploratory sessions because you cannot script what nobody described; the pairing the exam wants first is still review plus red teaming.
Robustness, efficiency, paperwork, and the release train
Lack of robustness to unexpected inputs—clipped audio, out-of-range voltages, random Unicode, jailbreak prompts—maps to adversarial testing plus fuzz testing. Adversarial work is targeted. Fuzzing storms the interface with malformed and random payloads to find crashes, hangs, and garbage outputs.
Inadequate performance efficiency—latency, throughput, memory, accelerator occupancy—maps to performance testing. ML load is not generic web load: batch size, cold start, and tail latency on a large model all change what fast enough means.
Poor documentation of function, accuracy, or interface maps to model documentation review. If testers cannot tell who the users are, which data trained the model, or what a score threshold means, they cannot design the rest of the map.
Change has its own rows. Updates that introduce defects are a job for back-to-back testing against the incumbent. Updates that decrease functional performance are a job for A/B testing. Immediate failure on deploy is smoke testing. Regression after deploy is regression testing.
Security vulnerabilities, safety vulnerabilities, privacy violations, and harmful or undesirable outputs—racist content, dangerous guidance, leaked training records—are a job for red teaming. Red teaming does not replace bias testing or API testing. It is the mitigation when the risk is an adversary or a harmful completion.
A risk-to-mitigation map you can actually use
The table below restates the pairings in tester language. Memorize the match, not a speech. On the exam, a stem will name a harm; your job is to name the approach that attacks that harm.
| Risk you are actually worried about | Test approach that mitigates it |
|---|---|
| Biased or unfair predictions across people or groups | Bias testing |
| Unethical purpose, misuse, or foreseeable harm | Ethical system testing |
| Tiny or crafted input changes that flip predictions | Adversarial testing |
| Model memorized the training set | Overfitting tests |
| Model never learned the pattern | Underfitting tests |
| Incoming features shifted, or the target's meaning shifted | Drift testing |
| Harm the objective never priced in | Side-effects testing |
| Agent exploits a reward loophole | Reward hacking testing |
| Serving contract, schema, or preprocessing is wrong | API testing |
| Accuracy, recall, or related measures miss the bar | ML functional performance testing |
| Functional incorrectness or non-functional defects, especially with no cheap expected result | Metamorphic testing; for the oracle problem also back-to-back testing and A/B testing |
| Vague or missing system requirements | Requirements review and red teaming |
| Fragile behavior on unexpected inputs | Adversarial testing and fuzz testing |
| Latency, throughput, or resource use is unacceptable | Performance testing |
| Function, metrics, or interface are poorly documented | Model documentation review |
| A new version introduces defects | Back-to-back testing |
| A new version lowers functional metrics | A/B testing |
| The new version fails as soon as it is deployed | Smoke testing |
| The new version breaks journeys that used to work | Regression testing |
| Security, safety, privacy, or harmful outputs | Red teaming |
Choosing from the menu on one product
Picture an insurer launching an image model that estimates vehicle damage from photos. Unfair payouts across neighborhoods pull bias testing. A vague mandate to use AI responsibly pulls requirements review and red teaming. A public upload API pulls API testing and fuzz testing. Stickers on a bumper that change the estimate pull adversarial testing. A weekly retrain pulls back-to-back comparison on a golden set and A/B on a canary of claims. A new graph that will not load in the claims tool pulls smoke testing. You still will not run reward-hacking tests unless some component actually learns from a reward signal.
That is the professional habit. The risk list is a menu, not a mandate. Read an exam stem until you can name the harm in one phrase, then pick the mitigation that addresses that harm. Distractors are usually real techniques pointed at a different risk.
A credit model posts a high overall accuracy score, but false-decline rates are much higher for applicants from one historically redlined ZIP-code cluster. Which test approach is the matching mitigation for that risk?
A warehouse robot is trained with reinforcement learning to maximize items-placed-per-hour. Testers watch it drop boxes just inside a sensor beam so the count increments, then ignore whether the box remains on the shelf. Which mitigation targets that risk?
Testers cannot afford a trusted expected caption for every image in a large unlabeled stream, yet they still need verdicts about functional incorrectness. Which combination is the matching mitigation for that oracle problem?
A new checkpoint loads cleanly, health checks pass, and the API still returns 200s, but live recall is worse than the incumbent on the same traffic mix. Which approach specifically targets decreased ML functional performance after an update?