14.1 ML Development Risks and Mitigations
Key Takeaways
- ML development testing targets tools, configuration, and deployment mechanisms rather than the learned model weights themselves.
- Incorrect TensorFlow or PyTorch API use is mitigated with API testing; a defective framework metric helper is mitigated by reviewing evaluation code and cross-checking results against manual benchmarks.
- Defective allocation of data to training, validation, and test sets is mitigated with a data allocation review, not with extra training epochs.
- A deployed model that is no improvement over the current model is mitigated with shadow testing; an incompatible operational environment is mitigated with smoke testing plus MLS deployment testing.
- Because learning is stochastic, a single run can mislead; ML functional performance testing with repeated measurement is the listed mitigation for incorrect interpretation of results.
When an ISTQB Certified Tester AI Testing (CT-AI) v2.0 item mentions machine learning (ML) development testing, it is not asking whether your classifier beat last week's F1 score. Learning objective AI-7.1.1 (K2) asks you to give examples of test approaches that mitigate risks introduced by ML development tools, configuration choices, and deployment mechanisms rather than by the learned parameters of the model itself. The model can be innocent. The factory that built it can still be defective.
That factory includes libraries such as TensorFlow and PyTorch, the evaluation helpers those libraries ship, package builds and GPU wheels, hyperparameter defaults, the code that partitions data into training, validation, and test sets, and exporters that rewrite a graph for a phone neural processing unit or an edge compiler. If the factory is wrong, every artifact that leaves it inherits the fault. Testers who only retrain and re-score miss the exam's point and miss the production bug.
Why this chapter is not more model testing
Model testing asks whether the learned function meets ML functional performance criteria on data the learner should not have memorized. Development-risk testing asks whether the toolchain, the configuration, and the shipping path are trustworthy. The syllabus therefore reuses techniques you meet elsewhere—API testing, bias testing, smoke testing, performance testing, usability testing, security testing, ML functional performance testing, back-to-back testing, A/B testing, reviews, and shadow testing—but it attaches them to development failure modes.
A useful exam habit is to read the risk first. If the stem describes a wrong TensorFlow call, do not answer collect more labels. If the stem describes a metric helper that disagrees with a confusion matrix you computed by hand, do not answer change the learning rate. Match the risk cluster to the listed mitigation.
A working map of risks to mitigations
The table below restates the pairings in tester language. Treat it as a retrieval aid, not as a claim that one technique is enough for every project.
| Potential development risk | Mitigation testers should name |
|---|---|
| Incorrect or unintended use of library or framework application programming interfaces (APIs), for example TensorFlow or PyTorch | API testing |
| Sub-optimal framework selection | Framework suitability review |
| Algorithm, model, or development framework introducing systemic unfairness | Bias testing |
| Defective framework installation or build | Smoke testing |
| Defective evaluation implementation by the framework | Reviews of framework evaluation code, plus cross-checks against manual benchmarks |
| Poor performance efficiency (the framework is slow to respond) | Performance testing |
| Poor usability of the framework | Usability testing |
| Defect in a library used by the framework | ML functional performance testing and back-to-back testing |
| Defective algorithm implementation | ML functional performance testing and back-to-back testing |
| Security vulnerabilities in the framework | Security testing |
| Poor user documentation for the framework | Framework documentation review |
| Sub-optimal algorithm selection | Algorithm suitability review and A/B testing |
| Sub-optimal hyperparameters such as network structure or learning rate | ML functional performance testing and A/B testing |
| Defective allocation of data to training, validation, and test datasets | Data allocation review |
| Poor selection of evaluation approach, for example k-fold cross-validation | ML functional performance testing |
| Incorrect interpretation of results because learning is stochastic | ML functional performance testing |
| Deployment defect from generating a modified version for a target platform | Smoke testing, ML functional performance testing, and A/B testing |
| Deployed model incompatible with the operational environment | Smoke testing and machine learning system (MLS) deployment testing |
| Deployed model is no improvement over the current model | Shadow testing |
Framework, API, install, and documentation cluster
Incorrect API use is a development defect even when the research idea is sound. Example: a training engineer calls a PyTorch loss with reduction set to mean while class weights were prepared for sum, so rare-defect classes barely move the gradient. Another example: a TensorFlow tf.data pipeline shuffles the validation split, so epoch-end metrics jitter and cannot be compared. API testing probes contracts: shapes, dtypes, training-versus-inference flags, device placement, and error paths when an input tensor is empty.
Framework suitability review happens before you pour weeks into training. A research-only library that cannot export to the factory's NVIDIA Jetson runtime, or a framework whose license blocks your deployment channel, is sub-optimal framework selection. Testers challenge that choice with a review against operational constraints, not with another notebook cell.
Smoke testing after a defective install or build answers a crude question: does the stack run at all? A CUDA wheel compiled against the wrong driver, a CPU-only binary on a GPU node, or a broken pip extra should fail a smoke test in minutes. There is no accuracy story until the framework executes.
Poor usability of the same stack is mitigated with usability testing: can a competent engineer configure mixed precision without silent fallback to float32? Poor user documentation is mitigated with a framework documentation review. Obsolete flags in a README are a development risk because they produce wrong runtime behavior.
Security testing targets vulnerabilities in the framework: unsafe deserialization in model loaders, debug servers bound to all interfaces, or compromised wheels. The weights file can be clean while the runtime is not.
Performance testing in this cluster is about a slow framework, not about F1. If an upgrade adds 40 milliseconds of Python wrapper time per inference, or training throughput collapses, you are testing tool efficiency.
When the algorithm, model, or framework introduces systemic unfairness—for example a default tokenizer that corrupts names from one language, or a sampling helper that drops a demographic slice—the mitigation is bias testing, not a faster GPU.
Evaluation-code cluster: independent calculation as the oracle
The exam wants a specific picture: the framework's evaluation implementation is defective, and testers mitigate that with reviews of the evaluation code and cross-checks against manual benchmarks. Do not treat the dashboard as ground truth.
Worked example 1 — inflated accuracy. A four-class industrial defect detector reports framework accuracy 0.91 on test. A tester exports raw predictions and rebuilds the confusion matrix by hand over 120 distinct images. Rows are the actual class, columns the predicted class:
| Actual \ Predicted | A | B | C | D | Actual total |
|---|---|---|---|---|---|
| A | 36 | 4 | 0 | 0 | 40 |
| B | 2 | 26 | 0 | 2 | 30 |
| C (rare) | 14 | 6 | 10 | 0 | 30 |
| D | 0 | 0 | 1 | 19 | 20 |
| Predicted total | 52 | 36 | 11 | 21 | 120 |
Read the per-class counts straight off the matrix: true positives on the diagonal, false negatives across the rest of the row, false positives down the rest of the column.
- Class A: 36 true positives, 16 false positives, 4 false negatives
- Class B: 26 true positives, 10 false positives, 4 false negatives
- Class C (rare): 10 true positives, 1 false positive, 20 false negatives
- Class D: 19 true positives, 2 false positives, 1 false negative
Sanity-check the fixture before you trust any metric derived from it. Actual totals sum to 40 + 30 + 30 + 20 = 120, matching the image count. The diagonal sums to 36 + 26 + 10 + 19 = 91, which cannot exceed 120. Every misclassified image is a false negative for its true class and a false positive for the class it was predicted as, so those two totals must agree: false negatives are 4 + 4 + 20 + 1 = 29 and false positives are 16 + 10 + 1 + 2 = 29. A fixture whose diagonal exceeds the item count, or whose false-positive and false-negative totals disagree, is arithmetically impossible and cannot contradict anything.
Accuracy from the matrix is 91 / 120 ≈ 0.758, not 0.91. Code review shows the helper iterated a loader that still applied training augmentations and counted each augmented view as an extra test item, mostly easy Class A copies. The metric implementation, not the learner, produced the headline number.
Worked example 2 — macro-F1 disagreement. Using the same matrix, recall for Class C is 10 / (10 + 20) = 0.333. Precision for Class C is 10 / (10 + 1) ≈ 0.909. Class-C F1 is 2 × 0.909 × 0.333 / (0.909 + 0.333) ≈ 0.49. Repeat that per class and average: A ≈ 0.78, B ≈ 0.79, C ≈ 0.49, D ≈ 0.93, so macro-F1 ≈ 0.75. If the framework's macro-averaged F1 prints 0.88, the helper is probably averaging only over classes present in the current batch, or substituting accuracy for F1. A tester computes macro-F1 from the full matrix as a manual benchmark and files a defect on the evaluation path.
Worked example 3 — ranking metric fixture. For a binary fraud scorer, construct ten labeled examples with known scores, sort them, and compute area under the ROC curve with the trapezoid rule. If the framework AUC differs on that fixture, the bug is in the tool. Independent calculation is the mitigation the syllabus is pointing at.
Algorithm, hyperparameter, split, and stochastic cluster
Defective algorithm implementation and a defect in a library the framework uses (a wrong kernel in a PyTorch operator, for example) are mitigated with ML functional performance testing and back-to-back testing: same inputs through the suspect stack and through a reference or last-known-good version; compare outputs and scores.
Sub-optimal algorithm selection is a choice, not a crash. Mitigate with an algorithm suitability review (does a linear model even have capacity for this interaction?) and A/B testing of two candidate learners under a frozen data contract.
Sub-optimal hyperparameters—depth, width, learning rate, regularization—are mitigated with ML functional performance testing and A/B testing. A single heroic run at learning rate 0.1 that later diverges is not evidence.
Defective allocation of data to train, validation, and test sets (patient IDs on both sides of a split, time-series shuffled as independent rows, target leakage through a future-looking feature) is mitigated with a data allocation review. Testers inspect identifiers, timestamps, and stratification counts. More epochs will not un-leak the split.
Poor evaluation approach, such as ordinary k-fold on temporally dependent samples, produces optimistic numbers. ML functional performance testing on a forward-chaining window exposes the collapse.
Because learning is stochastic, seed 7 can look like a breakthrough and seed 13 like a regression. Incorrect interpretation of results is mitigated with ML functional performance testing that repeats runs and reports spread, not with a single screenshot.
Export and operational compatibility
Generating a modified version for a target platform (quantization, compiler, reduced graph) can inject a deployment defect. Mitigate with smoke testing, ML functional performance testing, and A/B testing of the exported artifact against the training-time model.
If the deployed model is incompatible with the operational environment (wrong tensor layout, missing runtime, unsupported GPU instruction set), use smoke testing plus the MLS deployment testing forms in the next section.
If the deployed model is no improvement over the current model, shadow testing is the named mitigation: the candidate sees live requests in parallel, user-visible answers stay on the current model, and you compare.
A training loop called the PyTorch backward path twice on a graph that was not retained, silently corrupting gradients. Which mitigation addresses incorrect or unintended use of framework APIs such as TensorFlow or PyTorch?
A framework helper reports test accuracy 0.94. Independent calculation from a confusion matrix on distinct images yields 0.81 because the helper counted augmented copies. What is the mitigation for a defective evaluation implementation by the framework?
After export, a candidate model can run in production, but product owners need evidence it is actually better than today's live model before switching users. Which mitigation matches the risk that the deployed model is no improvement over the current model?
Patient records from the same encounter appear in both the training file and the test file. Which mitigation matches defective allocation of data to training, validation, and test datasets?