15.4 RAG Relevance Metrics and A/B Testing
Key Takeaways
- Evaluate retrieval separately from generation. A fluent answer can still be ungrounded if the index returned the wrong chunks.
- With labeled qrels, Microsoft Foundry’s Document Retrieval evaluator reports Fidelity, nDCG, XDCG, Max Relevance, and Holes for parameter sweeps (k, chunk size, hybrid versus vector, semantic ranker on or off).
- Without labels, the Retrieval evaluator (LLM-judge, 1–5) scores whether retrieved context could answer the query. After generation, use Groundedness (precision versus context) and Relevance (answer versus query).
- A/B one retrieval parameter at a time, hold the generator, prompt, and temperature constant, and score a frozen labeled eval set—not vibes from the playground.
- Do not A/B retrieval changes on 100 percent of production traffic without a holdout. Keep a control index or query template and a rollback git tag.
RAG Relevance Metrics and A/B Testing
Quick Answer: Split RAG evaluation into process (did Azure AI Search retrieve the right chunks?) and system (did the Microsoft Foundry model answer from those chunks?). Use Document Retrieval metrics (nDCG, Fidelity, Holes) on labeled qrels to tune k, chunking, and hybrid settings. Then score groundedness and relevance. A/B one retrieval parameter, hold the generator constant, and never ship a new index to all production traffic without a holdout.
Domain 5 closes RAG with evaluate and improve RAG with relevance metrics and A/B testing frameworks. Domain 4 already taught quality evaluators; this section is about using them as a retrieval tuning loop, not as a one-off demo.
Two layers: retrieval versus generation
If the retriever misses the policy section, the chat model will either abstain, guess, or quote a nearby but wrong paragraph. Groundedness can look decent if the model faithfully repeats the wrong chunk; relevance to the user still fails. Conversely, a perfect retriever plus a sloppy prompt yields ungrounded prose. Microsoft Foundry therefore ships process evaluators (retrieval quality) and system evaluators (final answer quality).
| Evaluator | Layer | You need | What it tells you |
|---|---|---|---|
| Document Retrieval | Process | retrieval_ground_truth (qrels with labels) + retrieved_documents | Fidelity, nDCG, XDCG, Max Relevance, Holes |
| Retrieval | Process | query + context (no labels) | LLM-judge 1–5 whether chunks could answer the query |
| Groundedness | System | response + context (query recommended) | Precision: no fabricated claims beyond context (1–5, default pass ≥ 3) |
| Groundedness Pro (preview) | System | query, response, context | Strict boolean via Azure AI Content Safety |
| Relevance | System | query + response | Does the answer address the question? (1–5) |
| Response Completeness (preview) | System | ground_truth + response | Recall versus the expected answer |
Recall@k (fraction of relevant documents appearing in the top k) and nDCG@k (normalized discounted cumulative gain—are the best documents ranked highest?) are the classic IR names you should still be able to explain. Foundry’s Document Retrieval evaluator is how you compute a bundle of those ideas without hand-rolling ranking code. Holes counts retrieved IDs that lack a judgment; a high hole ratio means your qrels do not cover what the index is returning, so nDCG is untrustworthy until you label more.
Groundedness is the precision of the answer against retrieved context. Response completeness is the recall of the answer against ground truth. Do not collapse them into a single “accuracy” number on the exam.
Build a labeled eval set
You cannot A/B chunk sizes in the playground. You need:
- A frozen list of real queries (production traces, support tickets, plus adversarial SKUs and paraphrases).
- For each query, qrels: document or chunk IDs with a relevance label (Foundry’s document retrieval sample uses labels on a scale such as 1–5; configure
ground_truth_label_min/maxto match). - Optional answer ground truth for completeness and for spot-checking generation.
- A mapping file that records which index, embedding deployment, k, searchMode, and semantic configuration produced each run.
Keep this dataset in Git next to the skillset. When the corpus changes, refresh labels; do not keep scoring against dead document IDs (those become Holes).
Microsoft documents a parameter sweep: generate retrieval results for candidate settings (vector vs hybrid vs hybrid+semantic, several k values, chunk sizes), then run builtin.document_retrieval to see which setting wins on nDCG and Fidelity. That sweep is the scientific version of “tweak k in Search Explorer until it looks good.”
A/B testing rules for RAG
A retrieval A/B is an experiment with one independent variable:
- Hold constant: generator model and version, prompt/instructions, temperature, safety filters, and the eval query set.
- Change one retrieval lever: chunk size, overlap, embedding model (full re-index), hybrid on/off, semantic ranker on/off, k, vector threshold, searchMode,
maxTextRecallSize. - Score both legs with Document Retrieval (and Retrieval if you lack full qrels), then run generation once per retrieved context and score groundedness/relevance.
- Promote only if retrieval metrics improve and answer-level groundedness does not regress. A higher nDCG that stuffs 40 chunks into the prompt can still wreck groundedness via distractors.
- Roll forward with a holdout. Send 5–20 percent of production traffic (or a canary index alias) to the candidate. Keep the control index and a git tag. If groundedness or incident rate worsens, flip the alias back.
Do not:
- A/B the embedding model and the prompt and the GPT deployment in the same week and call the winner “hybrid search.”
- Use production users as unlabeled judges without logging retrieved IDs—you will not know whether retrieval or generation broke.
- Tune on the same 20 queries you overfit in Search Explorer; keep a held-out query split.
- Flip 100 percent of traffic to a rebuilt index on Friday afternoon with no control.
Exam scenario: nDCG@10 is 0.41 on vector-only k = 10. You stand up index-B with the same 3-small embeddings but hybrid + semantic ranker, k = 50, top = 8 chunks to the model. On the frozen 150-query qrel set, Document Retrieval shows nDCG@10 = 0.63 and fewer Holes. Groundedness stays at a mean of 4.1 because you still send only eight chunks. You route 10 percent of the Foundry agent’s live queries to index-B for a week, watch Foundry quality monitors, then switch the alias. You did not change GPT-4o-mini in the same experiment.
Common trap: Declaring victory because the playground answer “sounds better” after enabling semantic ranker, with no qrels and with a new system prompt in the same session. A second trap: A/B testing on production with both variants writing to the same unversioned index, so you cannot roll back. A third: optimizing only fluency (a Domain 4 quality metric) while retrieval Fidelity is near zero—the model is eloquently guessing.
Operationalize the loop in GitHub Actions the same way you evaluate prompts (Chapter 12): on a pull request that changes the skillset, chunk parameters, or query template, run the eval dataset, fail the PR if nDCG or groundedness drops past a gate, and require a human review of Holes. That is GenAIOps for RAG, and it is what Exam AI-300 is testing when it pairs relevance metrics with A/B testing frameworks.
Why does Exam AI-300 insist that you evaluate RAG retrieval separately from generation?
You have query relevance labels (qrels) for a benefits corpus and want to compare chunk sizes and hybrid versus vector. Which Foundry evaluator is built for that parameter sweep?
How should you A/B a retrieval change such as turning on hybrid search plus semantic ranker?
A team wants to replace the production Azure AI Search index with a newly chunked hybrid index this afternoon for all users. What is the production-safety expectation on AI-300?