13.2 A/B Testing for ML Systems
Key Takeaways
- A/B testing compares two variants on the same inputs to decide which is better; it is statistical work that typically needs multiple runs.
- It is a classic marketing split and is widely used in production e-commerce by diverting different consumers to different functionality.
- A/B testing treats the existing system as a partial oracle for the test-oracle problem; it does not generate test cases or tell you how to design tests, though operational inputs are often used.
- For AI updates, agreed ML functional performance metrics are required so testers can confirm the updated variant is as good as or better than the previous one.
- The same approach covers simple classifiers, complex systems such as smart-city routing, and self-learning changes that must improve or revert; name t-test, z-test, chi-squared, and Mann-Whitney U.
What A/B testing is for on an MLS
Learning objective AI-6.1.9 is a K2 explain item: show how A/B testing is used on machine learning systems. The core move is simple to say and easy to cheapen on an exam. You take two variants of a program — call them A and B — present them with the same inputs, and compare their responses to decide which variant is better. "Better" is not a vibe. It is a statistical conclusion. The syllabus calls A/B testing a statistical testing approach that typically requires comparing results from multiple test runs before you treat a difference as real.
That framing already separates A/B testing from a one-shot demo. A single commute-time sample, a single day's click-through rate, or a single batch of classifier scores can bounce. Multiple runs, under comparable conditions, are how you keep luck from looking like a model improvement.
The classic split, then the production split
The picture most people already know is marketing. Two promotional offers go to a list divided in half. Half the list gets offer A. Half gets offer B. The success of each offer — opens, clicks, purchases — decides which offer you keep. CT-AI wants that picture on the page because it is the same design you will apply to models: two treatments, comparable audiences or inputs, a success measure, a decision.
Many e-commerce and web-based companies run the same idea in production. They divert different consumers to different functionality so they can see which behavior customers prefer. Variant A might be the current recommender. Variant B might be a newly trained ranker. Shoppers still arrive with real intents. The system assigns them to a variant. You measure the outcome you care about: conversion, add-to-cart, refund rate, or a model-specific ML functional performance metric. Production diversion is not a toy. It is how A/B testing gets operational inputs without you having to invent a complete expected-result oracle for every shopper.
A partial answer to the test-oracle problem
A/B testing is one approach to the test oracle problem. In conventional testing, an oracle tells you whether an output is correct. For many MLS tasks there is no cheap, complete oracle: the "right" recommendation, the "right" route, the "right" next document to retrieve. A/B testing typically uses the existing system as a partial test oracle. Variant A is already live. You do not claim A is perfect. You claim A is the baseline you are allowed to compare against. If B is not as good as A on the agreed measures, B does not replace A.
Partial is the honest word. The existing system can be biased, stale, or itself overfit. A/B testing does not magically certify absolute truth. It answers a relative question: is this variant better than that one, on these inputs, by these metrics, with enough runs to trust the difference?
What A/B testing does not do
Two negatives are examinable. A/B testing does not generate test cases. A/B testing provides no guidance on how the tests should be designed. Teams often incorporate operational inputs into the tests — live traffic, logged requests, last week's commutes — but the technique does not hand you a coverage model, a boundary table, or a metamorphic relation. If your operational mix never includes an edge case, A/B testing on that mix will not invent it.
That limitation is why A/B testing sits beside other model tests rather than replacing them. Adversarial examples, metamorphic follow-ups, and representativeness checks still have jobs. A/B testing's job is the comparative, statistical one.
Updates to AI-based systems
A/B testing can be used to test updates to an AI-based system, provided there are agreed-upon acceptance criteria, such as ML functional performance metrics. Whenever the system is updated, A/B testing is used to determine that the updated variant performs as well as, or better than, the previous variant. "As well as" matters. An update that is statistically indistinguishable from the incumbent on the agreed metrics can still be a pass if the reason for the change was maintainability, cost, or a non-functional property you are measuring separately. An update that is worse on the agreed ML functional performance metrics is a fail, even if someone liked the new confusion matrix in a notebook.
Without agreed metrics you cannot run the test you think you are running. "The new model feels sharper" is not an acceptance criterion. Precision, recall, F1, calibration error, business conversion, or average commute time can be, if the project named them before the split started.
Simple classifiers and complex systems
The same approach covers a simple classifier and far more complex systems. For a binary fraud classifier, variants A and B might score the same stream of transactions; you compare precision and recall, or a cost-weighted loss, across multiple days. For a smart-city transport routing system, an update meant to improve effectiveness can be tested by comparing average commute times for two variants on consecutive weeks. Consecutive weeks is a practical way to hold the city roughly constant while still feeding each variant a large operational diet. It is also a reminder that "same inputs" in the statistical sense can mean comparable operational slices, not a laboratory replay of one frozen file — though a replay of logged journeys is a valid lab form of the same idea.
When you use consecutive weeks, write down the threats to comparability: a holiday, a storm, a transit strike. Statistics do not cancel a one-off shock unless you designed for it. Multiple runs still help. They do not excuse a spoiled week treated as a model verdict.
Self-learning systems: accept the change or revert it
A/B testing is also how you police self-learning systems. When the system makes a change, automated tests run, and the resulting system characteristics are compared with those before the change. If the system is improved, the change is accepted. Otherwise the system reverts to its previous state. That revert path is part of the test design. A self-learning MLS without a measured comparison and a rollback is not "using A/B testing." It is hoping.
Characteristics can be the same ML functional performance metrics you use for a planned update: accuracy on a sliding window, average reward, complaint rate, commute time, click-through. The point is the closed loop. Change happens. Tests compare. Improvement stays. Non-improvement is undone.
Statistical techniques you must be able to name
The syllabus lists the most popular statistical techniques for A/B testing and points to a statistics reference. You are not asked to derive a formula in this learning objective. You are asked to name them:
| Technique | What testers use it for in this context |
|---|---|
| t-test | Compare means of a continuous measure (commute time, score) between variants when sample sizes and variance assumptions fit |
| z-test | Compare means or proportions with larger samples where a normal approximation is used |
| chi-squared test | Compare categorical outcomes (converted or not, class counts) between variants |
| Mann-Whitney U test | Compare two distributions without assuming they are normal; a rank-based alternative when the t-test's assumptions are a poor fit |
On an exam item, a choice that says "A/B testing is not statistical" is wrong. A choice that says you must show a derivation of Mann-Whitney U is outside this LO. A choice that lists only one of the four as "the" A/B test is too narrow. Remember the set: t-test, z-test, chi-squared, Mann-Whitney U.
Multiple runs and a named statistical test work together. Multiple runs give you a sample. The test tells you whether the difference you saw is plausible as chance. Neither replaces agreed metrics. A significant improvement on a metric nobody accepted is still the wrong decision.
Putting a K2 answer on the page
A complete answer states that A/B testing compares two variants on the same inputs to see which is better; that the comparison is statistical and typically needs multiple runs; that marketing splits and production e-commerce diversion are the everyday forms; that the existing system acts as a partial oracle; that the technique does not generate cases or design tests; that AI updates need agreed ML functional performance metrics and must be as good as or better than the previous variant; that the method scales from simple classifiers to systems such as smart-city routing (average commute times on consecutive weeks); that self-learning changes are accepted only when automated tests show improvement, otherwise reverted; and that the popular statistics are t-test, z-test, chi-squared, and Mann-Whitney U.
Exam traps for AI-6.1.9
- Treating A/B testing as a test-case generator or a design technique. It compares variants; it does not invent the cases.
- Running one pass and calling the winner. The syllabus expects multiple runs and statistics.
- Updating an AI system with no agreed ML functional performance metrics.
- Requiring the new variant to be perfect rather than as good as or better than the previous variant.
- Claiming A/B testing works only for email marketing, or only for simple classifiers. Smart-city routing is in the syllabus for a reason.
- Accepting a self-learning change without a revert path when characteristics did not improve.
- Confusing A/B testing with back-to-back testing. A/B testing asks which variant of the same MLS is better. Back-to-back testing uses a reference to detect defects.
Which statement about A/B testing of machine learning systems is true?
When A/B testing is used to check an update to an AI-based system, what must be in place?
After a self-learning MLS changes itself, automated tests compare system characteristics with the previous state. If the change is not an improvement, what should happen?