8.1 Testing Generative AI

Why testers cannot treat GenAI like a query

A conventional search box retrieves. A generative AI (GenAI) system synthesizes: new sentences, pixels, video frames, or audio samples drawn from a learned distribution. ISTQB CT-AI v2.0 therefore treats GenAI as a testing problem with its own oracles, parameters, and failure modes. Learning objective AI-4.2.1 is K2: explain how generative AI can be tested. You will compare evaluation dimensions, name the input factors that explode the test space, and distinguish functional review from non-functional measurement.

You will not, on this syllabus, be examined on using GenAI to write test cases for other software. That craft sits on the separate Certified Tester – Testing with Generative AI (CT-GenAI) syllabus. Treat CT-GenAI as a pointer, not as an in-scope technique for this chapter.

If you keep one sentence for the exam, keep this: testing GenAI means judging correctness, coherence, and creativity (including originality and novelty) of outputs, and checking that functional and non-functional requirements are both met.

Correctness, coherence, and creativity are different verdicts

Correctness is whether the artifact is true for the task and the domain. A benefits chatbot that invents a waiting period that does not exist in the policy is incorrect even if the tone is warm. A generated wiring diagram that swaps live and neutral is incorrect even if the linework is crisp.

Coherence is whether the artifact stays consistent with itself. Text that changes a patient's age between paragraph two and paragraph five is incoherent. A generated floor plan whose staircase does not meet the landing is incoherent. Video whose product label mutates across frames is incoherent. Coherence failures often survive a first-glance looks-fine review; they show up when you read or watch the whole artifact.

Creativity and originality ask whether the output is new enough for the use case without copying protected or unwanted source material. A slogan generator that paraphrases a competitor's trademarked tagline has failed originality even if the sentence is grammatical. A design tool that pastes a well-known cartoon mascot into a living room has failed a domain rule, not a spelling rule.

Exam items will mix the three. A discharge-summary model can be coherent and original while being medically incorrect. An image model can be correct as a labeled anatomy sketch while looking generic. Name the quality that failed; do not collapse everything into the model hallucinated.

Modalities change the oracle

GenAI systems produce text, images, video, and audio. A test strategy that only scores chat transcripts will miss a video model that invents a safety pictogram and an audio model that speaks a disclaimer the transcript never contained.

Write the modality into the test charter. The output is helpful is not an expected result for an image. The bottle label remains readable for three seconds and matches the SKU list is.

Black-box testing: prompts in, judgment out

The syllabus's common approach is black-box testing. Testers feed prompts, images, or partial data into the running system and assess the result (red teaming, covered in the next section, is one aggressive form of this). You do not need weights or training logs. You do need agreed factors:

• Clarity — can the intended user understand the artifact?
• Originality — is it new enough, and not a near-copy of a known work?
• Domain-specific rules — house style, clinical constraints, catalog SKUs, legal disclaimers, accessibility contrast

Black-box evaluation, manual or automated, is particularly useful for end-user applications such as chatbots and design tools, where satisfaction depends on usefulness and plausibility of generated content.

Worked example. An interior-design GenAI accepts a photograph of an empty room plus a text prompt. Charter tests include: (1) name a wood species the catalog does not sell — the render must not invent a SKU; (2) a dusk photograph — lighting in the output must remain physically plausible; (3) a request to place a copyrighted character as a floor lamp — the system must refuse. Those tests exercise domain rules and originality without opening the model.

The input explosion problem

The hard part is not writing one prompt. It is the input explosion problem: inputs are highly diverse and difficult to control. A test case that records only the user sentence is incomplete.

A reproducible GenAI test case therefore looks like {system-prompt version, user prompt, attachments, UI versus API, temperature, max tokens, conversation history}, not {prompt, expected string}. If any of those is missing, you cannot retest after a vendor model swap.

Exam trap: candidates remember temperature and forget the context window. CT-AI v2.0 states that retained conversation affects generated output. A single-turn pass does not prove a 20-turn session is safe or even on-policy.

Pass and fail when the expected result is qualitative

GenAI outputs are often not a single canonical string. In many situations, assessment is manual review, and pass or fail depends on qualitative evaluation criteria written in the requirements. Example: The assistant uses plain language, does not name a drug dose, and always ends with a prompt to contact a licensed clinician. Reviewers score a rubric. That is still testing, provided the rubric existed before execution and disagreements are resolved the same way you resolve any other inspection.

Alternatively, a second GenAI system can judge the first. The syllabus example is concrete: the correctness of a generated image can be checked by an image-recognition system. Treat the judge as a proxy oracle, not as ground truth. The judge may reproduce similar biases or errors. An image generator that under-represents darker skin will miss contrast problems in generated clinical photos; a recognition model trained on the same skewed world will often agree. Dual-model agreement is not independence. Record the judge's version, temperature, and rubric prompt with the same discipline you apply to the system under test, or you cannot tell product drift from judge drift.

Non-functional testing: compute, time, and money

Non-functional testing can be as necessary as functional testing. Resource use during inference and training must support efficient operation and cost control. Measures the syllabus names include CPU and GPU usage, memory consumption, network bandwidth, and response times. Add training cost and inference cost to the plan, because a cheap fine-tune that produces an expensive-to-serve model still fails the business requirement.

Measure latency on the path the user actually hits: gateway, safety filter, model, post-processing. A raw model call of 400 milliseconds plus a 2-second moderation filter is a 2.4-second product. Dropping the filter to make a public score look fast is not meeting the requirement; it is hiding a quality characteristic.

Training-time metrics matter when you own fine-tuning. Inference-time metrics matter on every production call. Keep them in separate rows of the test report so a spike in one is not explained away by a saving in the other.

Benchmarks compare; they do not ship

Benchmark suites are curated datasets and associated tasks that enable consistent comparison across models. They measure aspects such as language understanding, reasoning, and coding. Use them to compare candidate models and to detect regressions after a vendor change. Do not use a public leaderboard as the sole ship criterion. A model that tops a coding benchmark can still fail your hospital's discharge-summary rubric, your bank's citation rule, or your latency budget.

When you adopt a benchmark, freeze the dataset version and the scoring script. We ran a language benchmark is not a result. We ran this snapshot, this prompt template, temperature 0.0, this judge is a result.

Scope line you must not cross on CT-AI

Using GenAI for testing — generating test cases, oracles, or automation — is out of CT-AI v2.0 scope. Follow that thread in CT-GenAI. This chapter tests the GenAI system under test.