12.1 Adversarial Testing

Why CT-AI tests for adversarial examples

Learning objective AI-6.1.4 is a K2 summarize item: explain adversarial testing of machine learning systems in tester language, not invent a new attack paper. The activity is distinctive among model tests. You start from a legitimate input the model currently handles correctly, then you deliberately perturb that input. The change is often imperceptible to a human. A person still reads a stop sign as a stop sign, still hears the same spoken phrase, still sees the same loan application. If the model now makes an incorrect prediction, that perturbed input is an adversarial example.

The CT-AI v2.0 syllabus (ISTQB, 17 April 2026) is precise about the test inputs: they often consist of slightly modified versions of legitimate inputs that cause the model to misclassify them. That sentence does a lot of work. Garbage that no human would recognize is a stress input, not an adversarial example in this sense. An ordinary miss on a clean production sample is a functional failure, not an adversarial example. The human-imperceptibility clause is the point: you are showing that the model's decision surface does not match how a person interprets the same scene.

A worked picture, not a formula

Picture a traffic-sign classifier that correctly labels a clear photograph as stop. You add a perturbation so small that a driving instructor looking at the same frame still says "stop sign." The model now outputs yield, or speed limit 45, or some other class. That pair — original plus tiny change, human still correct, model now wrong — is the adversarial example. The same pattern appears in other modalities. Swap a few tokens in a support-email so a person still reads a billing question, and a text classifier routes it as "urgent legal threat." Scale one numeric feature in a credit file by a fraction a human underwriter would ignore, and the score flips from decline to approve.

Those successful perturbed inputs are the examples you log. Unsuccessful perturbations are still test inputs; they simply did not expose the vulnerability on this run. Keep the vocabulary tight on exam day: perturbation is what you did to the input; adversarial example is what you call the input when the incorrect prediction appears.

Accidental encounters versus malicious attacks

Once you can produce adversarial examples in the lab, you have to ask how they appear in production. Two stories sit on the same technical phenomenon.

Accidental adversarial examples arise without an attacker. Camera angle, JPEG compression, unusual lighting, a faded sticker a person still reads, a scanner that slightly warps a form, wind noise on a voice channel — any of these can push an input across a brittle classification boundary. The model never saw that exact combination during training. The world supplied the perturbation for free.

Malicious attacks use the same class of input on purpose. Someone who knows that small perturbations can flip a classifier will craft them to evade a malware detector, a content filter, a facial-recognition gate, or a traffic-sign reader. From the tester's chair the lab activity is similar: you still generate perturbations and you still record whether the prediction is incorrect. What changes is the threat model you document and the safeguard you recommend. CT-AI expects you to name both destinies. Do not claim adversarial testing exists only for red-team theatrics, and do not claim it exists only for messy cameras.

What the test is for

Finding these vulnerabilities is not an end in itself. Identification through adversarial testing lets developers incorporate safeguards and make the model more robust against adversarial examples, whether those examples arrive by accident or by attack. Typical safeguards include additional training on known examples, input preprocessing that strips high-frequency noise, confidence thresholds that refuse to act on low-margin predictions, and operational monitoring that flags sudden oddities in input statistics. You are not required to name a specific research algorithm on the exam. You are required to state the purpose in this order: find the vulnerability, add a safeguard, improve robustness.

Why generating effective examples is technically hard

If every tiny perturbation flipped the label, the model would already be unusable. Effective adversarial examples sit in a thin region: large enough to cross the model's classification boundary, small enough that a person still treats the input as the original class. Finding those points is technically complex. Attack techniques also evolve. A defense that blocked last year's perturbation pattern may fail against this year's. Staying current with evolving attack techniques is an ongoing challenge, which is why adversarial testing is not a one-time gate before first release. Treat it as a repeating model-test activity, especially after retraining, after input-pipeline changes, and after you hear about a new perturbation family in your domain.

Black-box adversarial testing

Black-box testing focuses on the model's input and output behavior without requiring knowledge of its internal workings. You cannot inspect architecture, weights, or the training process of the system under test. The syllabus gives two practical routes. Learn both; exam items like to offer only one as a trap.

Transfer from an equivalent model

Create an equivalent model whose internals you do know. Train or obtain a substitute that solves the same classification task. Craft adversarial test inputs against that substitute using those internals. Then apply the same inputs to the original model. The working assumption is transferability: equivalent models share classification boundaries, so an example that fools the substitute often fools the target. Transfer is a test-design hypothesis, not a proof. You confirm it by executing the examples on the system under test. If they do not transfer, you learned something about the target's boundary, and you do not get to claim a vulnerability you never observed on the real model.

This route exists precisely because many deployed models are reachable only as an API. You still need a story for how you generated the candidates. The equivalent model is that story. The original model remains a black box the whole time.

Brute-force search

The other black-box route is blunt. Generate a vast number of tests and hope that some random tests coincide with an adversarial example. This is expensive and unguided. It can still be the only option when you cannot build a substitute and you have no internals. Treat it as a last-resort search, not as a designed perturbation strategy. On a multiple-choice item, brute-force is still a valid black-box method. It is not the same as transferability, and it is not white-box testing.

White-box adversarial testing

White-box testing uses knowledge of the ML model's internals. Architecture, parameters, and the training process typically make it easier to craft adversarial examples, because you can see which features the model is sensitive to and in which direction a small change moves the predicted class. White-box access does not automatically produce examples; it lowers the cost of searching for them. During model testing you often do have this access even if the later production API hides it. Do not invent a rule that white-box testing requires the vendor's entire source tree. The syllabus names three internals: architecture, parameters, training process.

Manual crafting versus automated generation

Adversarial testing can be done manually by crafting specific adversarial examples, or through automated algorithms that generate large numbers of variations to find effective adversarial inputs. Manual crafting is useful when you have a domain hypothesis: a yellow square sticker in the lower left of a speed-limit sign is a realistic vandalism pattern; a particular Unicode look-alike in a product title is a realistic spam trick. Automation is useful when you need volume and when you want to explore many perturbation magnitudes. On the exam, remember both modes exist. Do not claim that adversarial testing is only an automated research technique, and do not claim a tester must hand-paint every perturbation.

How to report the work so it can change the model

A useful adversarial test report names the original input, the perturbation class (pixel noise, token substitution, feature scaling), whether a human still recognizes the original class, the original prediction, the new prediction, and whether the example transferred from a substitute model. That report is what lets the development team add a safeguard. A pile of unlabeled perturbed files is not a test result. If you used the equivalent-model route, record that the examples were generated on the substitute and executed on the target, so a reader can judge transferability rather than assume it.

Exam traps for AI-6.1.4

• Confusing adversarial examples with any incorrect prediction. Ordinary errors on clean data are functional failures.
• Treating random unrecognizable garbage as an adversarial example. The perturbation is often imperceptible to humans.
• Claiming black-box testing is impossible. Black-box routes exist: equivalent-model transfer and brute-force search.
• Treating transferability as a guarantee. Shared classification boundaries are an assumption you validate on the original model.
• Claiming white-box testing forbids automation, or that automation forbids manual crafting. Both generation modes are in the syllabus.
• Stopping at "we found a weird input." The reason you tested is to drive safeguards and robustness, against both accidental encounters and malicious attacks.