12.2 Metamorphic Testing

What metamorphic testing is for

Learning objective AI-6.1.5 is K3: use metamorphic testing to derive test cases for a given scenario. That is an apply item. Memorizing the definition is not enough. You must start from a source case, name a metamorphic relation, and write the follow-up inputs and expected results. Hands-on objective HO-6.1.6 sits next to this LO in the CT-AI v2.0 syllabus (ISTQB, 17 April 2026); the exam-style skill is the derivation, not a tool click-path.

Metamorphic testing (MT) is a technique in which new follow-up test cases are derived from a previously passed source test case. You generate one or more follow-ups by changing ("metamorphizing") the source using a metamorphic relation (MR). The MR is based on a property of a required function of the test object. It describes how a change in the test case's inputs is reflected in that test case's expected results.

Read that last sentence twice. The oracle you check is not "the one true label of this new image," which you may not have. The oracle is the relation. If the source predicted stop and you rotate the photo a few degrees, the relation may say the prediction should remain stop. You are checking that the pair of outputs obeys the property, not that you independently recomputed the scene from first principles.

MT can be used for most test objects. It applies to functional testing (did the class, score, or route stay in the required relationship?) and to non-functional testing (did latency, memory, or confidence stay in the required relationship after a cheap input change?). In AI and ML work it has been used on image recognition, search engines, route optimization, and voice recognition — the four application families the syllabus names, and the four you should be ready to reuse as examples.

Three relations the exam expects you to name

Testers verify objectives such as:

• Consistency — outputs remain in agreement across related inputs. Two queries that a product owner says are the same information need should not produce contradictory rankings.
• Monotonicity — outputs change directionally with the input. If a risk factor gets worse, the predicted risk should not improve.
• Invariance — outputs remain stable under perturbations that a person treats as irrelevant. A small rotation or a modest brightness change should not invent a new class if the object is still plainly visible.

Those three words — consistency, monotonicity, invariance — are the vocabulary for writing MRs. If an item gives you a story, pick the objective first, then write the follow-up.

When you select MT, and when the source label is missing

MT is particularly useful where generating expected results is problematic because an affordable test oracle is unavailable. That is the situation with some machine learning systems that use big data, and with systems where testers are unclear how the ML model derives its predictions. Traditional expected-result testing wants a trusted label for every new input. Opaque models and huge operational spaces make that expensive or impossible.

MT is typically selected over traditional oracle-based testing when:

• no reliable expected outputs exist because of model opacity or data scale;
• the system is a black box; or
• relational properties (not absolute values) suffice for the confidence you need.

MT is often based on a source test case that passed. It can also help when you cannot generate an expected result for the source itself. Some functions are too complex for a human tester to replicate and use as an oracle. In that situation you still generate follow-ups, run the set, and check the relationships among the outputs rather than their actual values. If those relationships hold, confidence in the program improves even though nobody hand-labeled the true age at death, the true shortest path, or the true relevance rank.

Worked MR when the true value is unknown: age at death

A risk-assessment MLS predicts age at death. You may have no ethical or practical way to know the true age at death for a synthetic person. You can still write a monotonicity MR from domain knowledge: increasing the number of cigarettes smoked should decrease the prediction. Source case S uses 0 cigarettes/day and the model outputs 82 years (you do not claim 82 is true). Follow-up F uses 20 cigarettes/day and the same other features. The relation says the follow-up prediction should be lower than 82, not that it should equal some textbook number. If F comes back as 88, the MR failed. That failure is usable even though both numbers might be "wrong" in an absolute sense.

Worked derivation for a two-point exam item: stop-sign classifier

This is the derivation you should be able to write under exam time. Treat it as a two-point item: one point for invariance follow-ups that keep the class, one point for an MR in which the class should change.

Given source test case S (passed). Input: a photograph of a stop sign, the sign filling the frame, daylight, upright. Output: the classifier predicts class stop. A tester or an independent check has already accepted this case, so it is eligible as an MT source.

Step 1 — name invariance MRs from the required function. For an image recognizer, a required function is: if a person still clearly sees a stop sign, the predicted class should remain stop. Two cheap perturbations preserve that human reading.

• MR-inv-rotation (invariance). A small rotation that leaves the sign readable must not change the predicted class. Follow-up T1: rotate the source photograph by about 8 degrees. Same crop, same sign, no other edits. Expected result: class stop.
• MR-inv-brightness (invariance). A moderate brightness change that leaves the sign readable must not change the predicted class. Follow-up T2: raise (or lower) brightness without blowing out the red octagon or hiding the word STOP. Expected result: class stop.

Execute T1 and T2. If either predicted class leaves stop while a person still sees the sign, the invariance MR failed. That is a relational defect in the model, even if you never had a second labeled dataset.

Step 2 — name an MR in which the expected result should change. Invariance is the wrong relation once the sign is gone. If you crop the image so the sign is no longer in the frame — pavement, sky, or a blank wall — a person would no longer identify a stop sign. The required function now says the predicted class should change; remaining stop would be a false identification of an object that is not there.

• MR-change-crop (expected class changes). Removing the sign region so the sign is gone must change the prediction away from stop. Follow-up T3: crop the source photograph so the octagon and the word STOP are entirely outside the frame. Expected result: class ≠ stop (a different class, or a reject/unknown if the system has that output).

Step 3 — write the three follow-ups as a set, not as three unrelated tests. S passed. T1 and T2 check stability under perturbations a driver would ignore. T3 checks that the model is not glued to stop after the evidence is removed. If you mistakenly apply invariance to T3 ("crop the sign away, still expect stop"), you have written an incorrect MR. The test might "pass" while the model is hallucinating a sign. That is exactly how incomplete or wrong MRs create false confidence.

How testers derive MRs, and how they validate them

You do not harvest MRs from the model's weights. Testers derive MRs from domain knowledge, requirements, or domain properties such as laws of physics. A routing system should not increase travel distance when you remove a closed road from the map and a better open road remains. A voice recognizer should not change the transcript because you applied a volume change that a listener still hears as the same sentence. A search engine should not drop a document from the top set merely because you reordered two equivalent query terms, if requirements say those terms are synonyms.

Validate candidate MRs before you trust a green dashboard:

• Expert review — a domain expert confirms the relation is actually required. A medical expert may reject "brighter X-ray, same diagnosis" if brightness change hides a fracture.
• Reference models — run the same follow-ups on a trusted implementation or a simpler baseline. If the reference obeys the MR and the system under test does not, you have a comparative signal.
• Edge coverage — include the boundaries of the relation. Invariance for "small rotation" is not a license for a 180-degree flip of a directional sign. Monotonicity for cigarettes is not a claim about every other lifestyle variable at once.

Incorrect MRs, for example overlooking complex interactions between variables, or incomplete sets of MRs, lead to false confidence. The stop-sign crop is the teaching case: an invariance-only suite never asks whether the class should change, so a model that always shouts stop looks perfect.

What MT does not detect

MT detects relational flaws. It does not detect all absolute errors. Both the source and the follow-up can be wrong in the same direction and still satisfy a sloppy relation. If the stop-sign photo was actually a red advertisement and the model said stop, and you rotate it and it still says stop, invariance holds and the absolute error survives. That is why the syllabus says MT should be used in combination with other test techniques — labeled functional performance tests, adversarial tests, reviews, and whatever oracle fragments you do have.

A second limitation is the source. Follow-ups inherit the source. If you cannot find a passed source case, you are not in the standard "previously passed source" pattern, and any relations you check are weaker. The hands-on exercise in the syllabus reminds students of that limitation when source cases that passed are unavailable.

Putting a K3 answer on the page

When the exam gives you a scenario, write in this order:

1. Identify the source input and the source result (or state that the source value is unknown and you will only check output relationships).
2. State the required-function property in one sentence (consistency, monotonicity, or invariance — or an MR in which results should differ).
3. Describe the input change.
4. State the expected result of the follow-up relative to the source.
5. Note one validation check (expert review, reference model, or an edge the MR must not over-claim).

That sequence is the whole skill. The stop-sign set (T1 rotation, T2 brightness, T3 crop-until-gone) plus the cigarette monotonicity example cover the syllabus's own illustrations. If you can derive those without looking, you can derive MRs for search, routing, and voice on the same skeleton.

Exam traps for AI-6.1.5

• Treating MT as "invent any new input." Follow-ups are derived from a source via an MR, not sampled at random.
• Checking only absolute labels when the point of MT is the relationship between outputs.
• Applying invariance where the required function says the output should change (the cropped stop sign).
• Claiming MT replaces every other technique. It misses some absolute errors by design.
• Writing an MR from model behavior ("the network seems to ignore brightness, so that must be required"). MRs come from domain knowledge, requirements, or physics, then you test whether the model obeys them.