8.4 Risk-Based Testing of ML Systems
Key Takeaways
- Risk-based testing applies to all systems; most AI regulations require a risk-based approach, and AI-based systems pose unique risks.
- There is no standard ML risk taxonomy; one useful split follows the ML workflow: development project risks, input-data product risks, and model product risks.
- Development and project risks such as algorithm selection, evaluation approach, and framework security are detailed in Chapter 7.
- Input-data product risks such as bias, pipeline defects, and unrepresentative data are detailed in Chapter 5; model product risks such as missed metrics, overfitting, and adversarial examples are detailed in Chapter 6.
- A second split uses ISO/IEC 25059 quality characteristics; specialized MLS techniques include data pipeline testing, adversarial testing, and algorithm or model suitability review.
Risk-based testing is the default, not an AI extra
Risk-based testing applies to all systems, with or without AI. You already prioritize tests by likelihood and impact. CT-AI v2.0 insists on that discipline because most regulatory frameworks, published or in development, require a risk-based approach to developing and managing AI-based systems. The EU AI Act is one example of that expectation; it is not the only one. Learning objective AI-4.3.2 is K2: explain how risk-based testing is applied to machine learning systems.
AI-based systems pose unique risks, so testing them differs from testing non-AI systems. A conventional defect might crash a page. An MLS defect might silently deny credit, invent medical advice, or leak a training record through a prompt. Impact is often statistical, delayed, or distributed across a population, which is why a purely scripted happy path is not a strategy.
Unique does not mean mysterious. It means the failure modes travel through data, induced behavior, and probabilistic outputs, so the test strategy must name those paths instead of copying a CRUD application's risk list and stopping there.
There is no standard ML risk taxonomy
There is no standard way to categorize ML risks. That sentence is itself an exam point. Do not invent a fake ISO number for the ML risk list. The syllabus then offers two useful splits you are expected to explain.
Split 1 — follow the ML workflow
ML-specific risks attach to development of the MLS (project risks) and to the MLS itself (product risks). One way to categorize them is to divide the ML workflow into three areas.
Development (project risks) — concerned with the ML algorithm, development of the model, and the ML development framework. Example project risks include sub-optimal algorithm selection, poor selection of evaluation approach, and framework security vulnerabilities. See Chapter 7. Testers contribute by reviewing whether the team picked a model class the data can support, whether the evaluation set is independent, and whether notebook dependencies are pinned and scanned. A beautiful dashboard that reports accuracy on the training set is an evaluation-approach risk, not a user-interface risk.
Input data (product risks) — concerned with providing training data to support ML and production data used by the model in its operational environment. Example product risks include biased training data, data-pipeline defects, and unrepresentative training data. See Chapter 5. A pipeline that drops a rare but high-impact class is a product risk even if the algorithm choice was sound. Unrepresentative data is not cured by a more fashionable architecture.
Model (product risks) — concerned with the generated ML model. Example product risks include failure to achieve required ML functional performance measures, an overfitted model, and susceptibility to adversarial examples. See Chapter 6. Overfitting is a model product risk even when the pipeline and the framework were fine.
Memorize the mapping for the exam: development and project → Chapter 7; input-data product → Chapter 5; model product → Chapter 6. Items love to swap those destinations.
Split 2 — ISO/IEC 25059 quality characteristics
Another way to categorize risks associated with AI is according to the quality characteristics defined in ISO/IEC 25059, which extends ISO/IEC 25010 for AI systems. From a testing perspective those characteristics shape objectives and acceptance criteria. AI-specific characteristics you met in Chapter 2 include AI functional correctness, functional adaptability, user controllability, transparency, AI robustness, intervenability, and societal and ethical risk mitigation.
A risk register can be a matrix of workflow area versus quality characteristic. Unrepresentative training data (Chapter 5) threatens fairness under societal and ethical risk mitigation and also threatens AI functional correctness for a minority slice. Weak intervenability is a product risk even when recall looks strong: if no operator can stop a harmful action in time, the quality characteristic has failed.
Specialized techniques that exist because the risks are ML-specific
Several forms of testing address these risks and are designed for MLS. The syllabus names, as instances:
- Data pipeline testing
- Adversarial testing
- Review of algorithm/model suitability
Chapters 5, 6, and 7 cover several of these in depth. Your job at AI-4.3.2 is to explain how risk-based testing is applied: pick a split, name example risks, and name a technique that mitigates them. You are not asked here to calculate an adversarial perturbation or to run a full pipeline suite.
Use the techniques where the register says impact is high. Data pipeline testing answers input-data product risk. Adversarial testing answers model product risk of susceptibility to adversarial examples. Algorithm and model suitability review answers development project risk of sub-optimal algorithm selection. Mixing those up on the exam is a common fail.
Worked risk register for an imaging MLS
A hospital deploys a locked imaging model that flags likely fractures on X-rays, with a radiologist still issuing the report. The strategy meeting lasts forty minutes and produces the table below — that is risk-based testing applied, not a slogan.
| Risk | Split | Example test response |
|---|---|---|
| Team picks a tiny CNN because the framework tutorial used it | Development / project (Chapter 7) | Algorithm and model suitability review against image resolution and class imbalance |
| Training studies come from one scanner vendor | Input data product (Chapter 5) | Representativeness testing; pipeline tests for vendor-tag handling |
| Nightly DICOM feed drops a required metadata field | Input data product (Chapter 5) | Data pipeline testing in a production-like integration environment |
| Hold-out recall looks strong; live recall drops on pediatric films | Model product (Chapter 6) | Additional model testing on the under-served slice; ML functional performance monitoring |
| Stickers on the cassette fool the detector | Model product (Chapter 6) | Adversarial testing of input perturbations |
| Quantized edge model on the clinic PC disagrees with the server model | Conventional system test, triggered by model-change risk | Confirmation testing after compression (see 8.3) |
| A GenAI report-drafting add-on could invent findings | Product risk across safety and misinformation | Red teaming (4.2.2) plus acceptance criteria on AI functional correctness |
Notice the pattern: unique AI risks do not replace component or system tests; they prioritize extra ML levels and specialized techniques where impact is high. A low-risk internal clustering demo may skip heavy adversarial testing. A high-risk clinical or credit system should not.
When regulators ask for a risk-based approach, they are asking to see this kind of register, the tests it drove, and the residual risk you accepted — not a claim that the team ran accuracy. CT-AI v2.0 expects you to explain that application, using either the workflow split or the ISO/IEC 25059 split, and to name the specialized techniques that belong with those risks.
Which statement about categorizing ML risks does CT-AI v2.0 make?
Sub-optimal algorithm selection, a poor evaluation approach, and ML-framework security vulnerabilities are examples of which risk group, and where does CT-AI v2.0 send you for detail?
Biased training data, data-pipeline defects, and unrepresentative training data are examples of which risk group in the ML-workflow split?
Which trio does CT-AI v2.0 give as examples of testing forms specifically designed for machine learning systems when addressing these risks?