8.3 Test Levels for Machine Learning Systems

Two levels that exist only because the system learned from data

Conventional software has component, integration, system, and acceptance testing. A machine learning system (MLS) still needs those. It also needs two levels that exist because behavior was induced from data rather than fully specified in code. Business outcome BO6 asks you to recognize the scope and importance of the two test levels specific to the testing of machine learning systems. Those two levels are:

1. Input data testing (syllabus Chapter 5)
2. ML model testing (syllabus Chapter 6)

If an exam item asks which test levels are specific to MLS, the answer is those two — not system testing, which remains a conventional level applied to a new kind of product. Learning objective AI-4.3.1 is K2: summarize the test levels used to develop machine learning systems. You must be able to place a given activity on this map, not invent a sixth ML-only level.

Input data testing concerns the training data used to train the model and the production data used by the MLS to generate a prediction in the operational environment. Testers who only profile the training set miss drift, schema breaks, and unrepresentative live features. Testers who only glance at live traffic miss labels, leakage, and bias baked into the fit. Chapter 5 is where bias testing, pipeline testing, representativeness, and dataset constraints live. This section only places the level on the map.

ML model testing concerns the models, which are the ultimate output of the ML workflow (syllabus 3.1.2). You evaluate ML functional performance metrics, overfitting, robustness to adversarial examples, and similar model product risks. Chapter 6 owns those techniques. This section's job is to stop you from pretending that a good F1 score on a notebook is the same as a tested product.

The syllabus is explicit: not all ML risks are addressed by these two levels. You still need conventional levels, scaled to perceived risk.

Component testing of the non-AI pieces

Component testing applies to non-AI components such as the user interface, data pipeline, and communication components. The upload form that rejects a 12-column CSV, the feature-store job that casts timestamps, the message queue that delivers inference requests — those are still units and components. Test them with the techniques you already know from ISTQB Foundation: equivalence partitioning, boundary values, API contracts. Do not wait for the model to absorb a pipeline bug; the model will happily learn the bug.

A practical tell: if the defect would still exist if you replaced the model with a stub that returns a constant, it is a component (or integration) defect, not an ML-model-testing miss.

Component integration — including AIaaS APIs

Component integration testing includes verifying that inputs from the data pipeline are received as expected by the model, and that predictions generated by the model are exchanged with the relevant system components (for example the user interface) and used correctly. A classic defect: the model emits a probability in the unit interval and the UI treats it as a percentage, or the pipeline sends Fahrenheit while the model was trained in Celsius.

When AI is provided as a service (AIaaS), API testing of the provided service is performed as part of component integration testing. You test authentication, schema, error codes, rate limits, and payload contracts for the remote model the same way you would test any critical dependency — plus ML-specific checks such as whether a model-version header is present so you can reproduce a prediction.

Worked example. A credit-decision UI calls an internal feature service, then an AIaaS scoring API, then a rules service that applies a policy cap. Component integration tests send a frozen feature vector, assert the AIaaS response schema, assert the UI shows the same band the policy cap computed, and assert that a 503 from the AIaaS produces a defined fallback, not a blank page. Those tests do not replace model testing on a hold-out set; they prove the sockets still fit after someone changes a field name.

System testing — especially after you shrink the network

System testing includes confirmation testing that ML functional performance from initial model testing is not adversely affected when the model is embedded in a complete system. This is especially important when the ML model has been deliberately changed, for example by compressing a deep neural network (DNN) to reduce its size. Quantization, pruning, and distillation can drop accuracy, change calibration, or alter latency in ways a notebook metric never showed.

System testing also covers non-functional quality of the whole AI-based system, for example performance efficiency of the time required to deliver a prediction. The number that matters is user-visible prediction latency of the assembled product, not the isolated model-predict call. Filters, feature lookup, and UI rendering all sit on that clock.

If you only re-run the Chapter 6 metric suite on the uncompressed checkpoint, you have not done this confirmation. Embed the compressed artifact, hit it through the real path, and compare against the same ML functional performance criteria you accepted in model testing.

System integration testing

System integration testing focuses on interfaces and data exchanges between the AI-based system and external systems or services, using an environment representative of operational conditions. Think core banking, electronic health records, identity providers, batch file drops from partners, and third-party data vendors. A model that scores well on a CSV from last quarter can fail when the partner's nightly file arrives with a new enum or a delayed timestamp.

Representative does not mean production data of real people copied into a laptop. It means production-like schemas, volumes, timing, and failure modes, with sanitized or synthetic payloads where privacy requires it.

Acceptance testing, especially for AIaaS

Acceptance testing, especially where AI is used as a service, may be needed to determine suitability of the service for the intended system and whether ML functional performance criteria have been achieved. Contractual acceptance for AIaaS is not the demo looked clever. It is measurable: recall on a held-out operational sample, latency percentiles, availability, data-residency constraints, and a documented model-update process.

Walkthrough: placing tests on a loan MLS

A lender trains a gradient-boosted model on historical applications.

• Input data testing: training extract versus live application stream; sensitive-attribute proxy checks; pipeline tests for encoding.
• ML model testing: hold-out metrics, overfitting gap, simple adversarial perturbations of income fields.
• Component: UI field validation; pipeline job; notification service.
• Component integration: pipeline to model to UI; if scoring is AIaaS, API contract tests live here.
• System: the model is quantized for the branch laptop; confirmation that approval-rate and error metrics still meet the notebook baseline; p95 prediction latency of the whole path.
• System integration: overnight bureau file and identity service in a production-like staging network.
• Acceptance: product owner accepts AIaaS suitability and the ML functional performance criteria in the contract.

BO6 is satisfied only if the first two levels exist as first-class work, not as a footnote inside system test. Conventional levels then catch the embedding, the neighbors, and the contract.