9.3 Data Pipeline Testing

A machine learning system is not only a model file. It is a data pipeline: the chain of ingestion, transformation, validation, storage, and routing that turns raw events into the tables or tensors a model can consume. If that chain is wrong, the model is scoring a fiction. ISTQB CT-AI version 2.0 learning objective AI-5.1.3 (K2) asks you to summarize the forms of data pipeline testing. The story is layered. You do not jump from a design sketch to production A/B testing and hope. You review the design, test components, integrate those components, test the assembled pipeline as a system, integrate it with the rest of the world, then keep testing after go-live — including the unglamorous work of configuration management.

Effective pipeline testing does two jobs at once. It supports reliability and performance of data-driven systems, and it protects data quality across the ML workflow. A transform that silently recodes a category, a sensor interface that drops night batches, or a config that points production at last month's schema will manufacture every input-data risk from section 9.1. Pipeline testing is how you catch those manufacturing defects before they become mysterious model behavior.

Design reviews come first

The layered approach begins with data-pipeline design reviews during the design phase. Reviewers ask whether sources, schemas, retention, error paths, and security boundaries make sense before anyone writes a brittle script. Typical defects found here are mismatched grain — daily facts joined to hourly facts as if they were twins — a missing dead-letter path for rejects, or a plan to train on a field that operations is not allowed to send. Design review is cheap compared with discovering those issues after a model is already in a customer path. On the exam, if the stem is still on the whiteboard, the form of testing is design review, not fault injection.

Component tests: ingestion, transforms, and sensor interfaces

Component testing covers the pieces that capture and reshape data: ingestion components, transformation scripts, and sensor interfaces. Techniques include code review, static analysis, and hardware-specific tests that confirm reliable capture from a device, not just from a mocked file. A camera clock that drifts, a controller that reports a sentinel as a real temperature, or a mobile software kit that truncates location to city scale will not show up in a unit test that reads a perfect fixture. Hardware-specific tests exist because sensors lie in hardware-specific ways.

Component tests also validate transformation logic, exercise data validation rules, confirm error handling, and probe vulnerabilities that could let someone introduce malware or poisoned data. Ask what happens when a file is truncated, when a field is duplicated, when an extra column appears, when a scriptable payload rides inside a CSV cell, or when an uploader swaps a file for one that looks similar. Error handling that logs and continues can be a defect if continuing means training on half a record. Think of this layer as both correctness of the recipe and hostility of the ingredients.

What a strong component test actually asks

A transformation that computes a rolling average should be tested for window size, for missing points inside the window, and for the moment a unit change would silently rescale the average. A validation rule that rejects negative ages should be tested with zero, with a legal maximum, and with a string that used to be a number. An ingestion service should be tested for duplicate delivery, delayed delivery, and a sender that authenticates but then posts a payload in the wrong schema. These are still component tests. They become integration tests only when two independently built pieces must agree on the meaning of the bytes.

Component integration: the seams

Component integration testing checks that data still mean the same thing as they cross internal interfaces. This is where teams discover that component A emits degrees Celsius and component B assumed Fahrenheit, that timestamps lost their time zone, that a null and an empty string diverged, or that a feature store keyed records by user while the trainer keyed them by session. Integration tests exist to find interface mismatches and incorrect assumptions between components. If you only tested each script with its author's favorite fixture, you have not tested the pipe. Exam stems that mention a silent unit change at a hand-off, or two teams reading the same field differently, are describing this layer.

System testing of the assembled pipeline

System testing assesses the fully assembled pipeline as one object, not as a pile of scripts.

• Smoke testing confirms basic end-to-end function: a known good batch enters and a plausible output appears. If smoke fails, stop. Deeper tests will only multiply noise.
• Functional testing checks specified requirements such as transformations and routing. Did the derived features compute as specified? Did rejected rows go to the quarantine you named, not to the training set? Did the right model version receive the right feature vector?
• Non-functional testing evaluates load, scale as volumes grow, and security measures that protect integrity and confidentiality. A pipeline that works for a laptop demo and dies at month-end close is not operationally ready. Security tests here protect the data in motion and at rest, including whether an untrusted upload can ride into the training path.
• Fault injection measures robustness by sending defective data and watching whether integrity holds. Can the pipeline reject poison, quarantine a schema break, and keep a clean path alive? Fault injection is not the same as ordinary functional testing with valid inputs. It is a deliberate attempt to feed the pipe garbage, hostility, or corruption and see whether quality survives.
• Back-to-back testing compares the operational pipeline with the training pipeline to see whether they still implement the same transformations. Many production surprises are not the model drifted. They are training used a notebook, production used a rewritten job, and the jobs are cousins rather than twins. If the two pipes disagree on a feature, every later accuracy number is a comparison of different worlds.

System integration with the rest of the estate

System integration testing verifies interaction with external systems and services: data sources, storage platforms, monitoring tools, and downstream consumers such as ML models. A pipeline can pass isolated system tests and still fail when the warehouse enforces a new constraint, when the monitoring agent samples every hundredth record, or when the model server expects a different serialization than the pipeline emits. Test the contracts, not only the happy logs. If monitoring never sees rejects, operations will learn about poison from customers. If storage silently truncates a column, the model will learn a shorter world than the scientist intended.

Testing in production

Testing in production is performed on the operational system because some defects only appear with live volume, live attackers, and live schema creep.

• Back-to-back testing against previous versions checks that behavior is consistent or improved, not quietly worse. A new job that is faster but recodes a rare category is not a success.
• A/B testing of new pipeline iterations compares a candidate path with a baseline so you can see improvement without assuming the live stream is harmless. This is how you try a new parser, a new window, or a new validation strictness on a slice of traffic.
• Continuous monitoring watches model behavior, pipeline performance, and defects in real time: lag, reject rates, distribution shifts, and security events. Monitoring is a test that never clocks out. It is also how you notice that yesterday's healthy pipe is today's slow, leaky, or hostile one.

Production tests are not a substitute for the layers below. They are the layers that keep working after release. Teams that skip design and component tests and hope monitoring will catch everything usually catch the incident after users do.

Configuration management reviews

Configuration management reviews confirm that the correct pipeline code versions, configurations, and datasets are used across training, testing, and production. The defect pattern is painfully ordinary: a scientist trains on one snapshot, a tester uses a sanitized subset, and production points at a different bucket with a different codebook. Reviews trace those pointers on purpose. If you cannot say which code, which config, and which data produced a model, you cannot reproduce a failure and you cannot claim the evaluation meant anything. This review is still pipeline testing. It is testing the identity of the pipe, not only its functions.

Training pipelines and operational pipelines are different test problems

Test emphasis should follow the pipeline's purpose. Training pipelines are often exploratory. They change weekly, they ingest experimental features, and their first duty is data integrity: do not teach the model garbage, poison, or a silently recoded label. Operational pipelines must be robust. Their testing priorities emphasize reliability, performance, and maintainability — surviving load, failing closed, and remaining changeable without tribal knowledge. A strategy that treats a research notebook and a twenty-four-hour scoring pipe as the same object will under-test one of them. Name the purpose first, then choose how deep each layer must go. Exploratory training work still needs integrity checks. Live scoring work still needs more than a happy-path notebook test.

On a K2 exam item, map the clue to the layer. A whiteboard schema fight is design review. A drifting sensor is a component and hardware-specific test. A Celsius-versus-Fahrenheit hand-off is component integration. A month-end volume collapse is non-functional system testing. Defective batches used as weapons are fault injection. Notebook-versus-job disagreement is back-to-back. A warehouse or monitor that never joined the lab tests is system integration. A live iteration compared with last week's job is production A/B or back-to-back. A model whose training snapshot nobody can name is configuration management. Keep the layers distinct and you will keep the answers distinct.