12.2 Create Prompt Variants and Compare Performance

Create Prompt Variants and Compare Performance

Quick Answer: A variant is one LLM-node (or agent-instruction) candidate. Change one factor at a time — system text, few-shot examples, or temperature — then batch-run every candidate on the same evaluation dataset and score groundedness, relevance, coherence, fluency, or task accuracy. Promote the winner. Do not A/B on production traffic first.

The second Domain 3 bullet is create prompt variants and compare performance across different prompts. Microsoft’s prompt-flow documentation is the reference implementation of that skill, even as new Foundry work moves the same idea onto agent versions and evaluation runs.

What a variant is

In prompt flow, a variant is a specific version of a tool node with distinct settings. Variants are supported on the LLM tool. A new variant can change prompt content, connection settings (temperature, model deployment), or both. The first node you see is variant_0 and is the default. You Show variants, Clone to variant_1, edit, and Hide variants when you are done. All variants stay folded; the default is what a normal run uses.

Microsoft’s news-summary example makes the factorial obvious:

That table is a teaching grid, not a recommended experiment. If you ship all four at once you cannot tell whether wording or temperature moved the metric. The controlled experiment rule is: change one thing.

The Web Classification sample in the tuning guide is the exam-shaped walkthrough. Starting from variant_0 on classify_with_llm, they create variant_1 with temperature 1 → 0 (classification wants precision) and variant_2 with temperature 0 plus few-shot examples. On summarize_text_content they clone a variant that asks for 300 words instead of 100. They do not flip summary length, few-shot, and temperature in a single clone.

On Foundry agents the same discipline applies with files: v1_instructions.txt, v2_instructions.txt (few-shot added), v3_instructions.txt (grounding/citation clause added). create_version() mints agent:1, agent:2, agent:3. You still change one factor per version if you want a defensible comparison.

Compare with evaluators, not vibes

Microsoft’s own warning: running variants on a few rows and judging with the naked eye cannot reflect real-world diversity, and the output is not measurable. The fix is a batch run (prompt flow Evaluate wizard) or a Foundry evaluation run.

Prompt-flow batch evaluation:

1. Prepare a dataset that represents production (for classification, URLs plus ground-truth labels).
2. Select one node whose variants you will sweep. Other LLM nodes stay on their default variant so you do not confound two knobs.
3. Map columns: ground truth from the dataset, prediction from a flow output.
4. Pick an evaluation method. Classification uses Classification Accuracy Evaluation. RAG and generation use quality evaluators.
5. After the run, Visualize outputs to see metrics per variant and the per-row predictions.
6. Set the winner as the node default (or deploy that agent version).

Domain 4 — covered in Chapter 13 — is the metric catalog you attach now:

• Groundedness — is the answer supported by retrieved context? Model-based groundedness scores 1–5. Groundedness Pro (preview) is a binary pass/fail from Azure AI Content Safety and does not need a judge deployment.
• Relevance — does the answer address the query?
• Coherence — logical flow.
• Fluency — readable language.
• Retrieval / Document Retrieval — did the retriever surface the right chunks (a retrieval problem, not a wording problem)?
• Risk and safety evaluators (hate, sexual, violence, self-harm, protected materials, indirect attack / XPIA) — a “friendlier” variant that fails safety is not a promotion candidate.
• Agent evaluators such as tool call accuracy, task adherence, and intent resolution when the prompt’s job is to drive tools.

Microsoft’s combining guidance for RAG is Retrieval + Groundedness + Relevance + Content Safety. A variant that wins fluency and loses groundedness is a regression. Do not average unrelated metrics into one vanity score unless you documented a rubric with weights.

A/B testing in this chapter means: score variant A and variant B on the same held-out evaluation dataset (and, later, a small non-production or canary slice). Domain 5’s RAG A/B frameworks are for retrieval strategy. Here you are A/B-ing prompts. You do not start by sending 50 percent of members a rewritten system prompt because someone liked it in the playground.

When a flow has variants on two nodes, you cannot run both sweeps in one batch. Microsoft documents that each run selects one LLM node with variants; other nodes use defaults. You sweep classify_with_llm, lock the winner, then sweep summarize_text_content. That is the same one-factor rule at graph scale.

Exam scenario

You maintain a prompt-flow chat flow that drafts appeal letters from retrieved policy text. variant_0 is a short system prompt at temperature 0.7. Legal wants firmer citations. You clone variant_1 that only adds “quote the retrieved clause and append [doc_id]” — same temperature, same few-shot. You batch-run 400 labeled appeals. Groundedness mean moves 3.1 → 4.4; relevance is flat; fluency dips 0.2. Safety evaluators still pass. You set variant_1 as default, commit the Jinja2 file, and open a pull request with the eval HTML attached. You do not also drop temperature to 0 in that same clone, because then you could not tell which change helped.

Common trap

The trap is declaring a winner from five playground chats. A sibling trap is changing system text, few-shot, and temperature in one variant, then attributing the lift to “better prompting.” A third is promoting a variant that raised fluency while groundedness and XPIA scores got worse. A fourth is running production traffic as the first A/B because “the eval set is small” — grow the eval set (Chapter 13) instead of using members as guinea pigs.

Variants are controlled experiments with numbers. The next section puts those files in Git so the winner is the commit you can roll back.