2.9 Evaluating Generative Solutions: LLM-as-a-Judge and RAG Evaluation

2.9 Evaluating Generative Solutions: LLM-as-a-Judge and RAG Evaluation

Blueprint reference: Section 2.3, "Evaluating predictive and gen AI solutions (e.g., model evaluation metrics and LLM-as-a-judge)."

This consideration has two halves. The predictive half — precision versus recall, PR-AUC versus ROC-AUC, RMSE versus MAE — is covered in depth in Section 3.10, Model Evaluation Metrics, Transfer Learning and Fine-Tuning. This section covers the generative half, which is newer, less well documented, and where the exam's LLM-as-a-judge wording points.

The one predictive fact worth restating here, because it recurs in generative contexts too: a metric is only meaningful next to a baseline, and evaluation data must be held out from anything used to select prompts or models.

Evaluating Generative Output

Reference-based metrics — when there is a correct output:

• Exact match / F1 for extraction and short-answer tasks.
• ROUGE for summarization (recall of reference n-grams), BLEU for translation (precision of candidate n-grams).
• BERTScore and embedding similarity for semantic rather than lexical overlap.

These are cheap and reproducible but shallow. A summary can score well on ROUGE while being factually wrong, because n-gram overlap does not check truth.

LLM-as-a-judge — when there is no single correct output. A separate model scores the response against an explicit rubric.

Rubric (each scored 1–5, with anchors):
  Faithfulness   — every claim is supported by the provided context
  Relevance      — the answer addresses the question actually asked
  Completeness   — no required element of the answer is missing
  Tone/format    — matches the specified style and structure
Output: per-dimension score plus a one-sentence justification.

Rules the exam expects you to know:

• Write the rubric down. "Rate this 1–10" produces noise; anchored criteria produce signal.
• Calibrate against humans. Score a sample with both the judge and human raters and measure agreement. An uncalibrated judge is an unvalidated instrument.
• Do not let a model grade its own output. Self-preference bias is well documented; use a different model, ideally a stronger one.
• Control for known biases. Judges favour longer answers and are sensitive to option order in pairwise comparisons; randomize position and consider length normalization.
• Pairwise comparison beats absolute scoring when you are choosing between two candidates, because relative judgments are more stable than absolute ones.

Designing the Judge Prompt

A judge is a model with a job description, and vague job descriptions produce vague work. Four elements make the difference between a usable instrument and a random number generator:

1. One dimension at a time. Ask for faithfulness, then relevance, then completeness, as separate scored fields. A single "overall quality" number silently averages incommensurable things and cannot tell you what to fix.
2. Anchored scale points. Define what a 1, a 3, and a 5 look like in words. "5 = every claim is directly supported by a cited passage; 3 = the main claim is supported but supporting details are not; 1 = the answer contains claims absent from the context." Without anchors, two runs of the same judge disagree with each other.
3. A required justification. Making the judge state its reason in one sentence before the score improves consistency and, more importantly, gives a human reviewer something to audit when the score looks wrong.
4. Structured output. Constrain the judge to a schema so scores parse reliably and can be logged as a time series rather than scraped out of prose.

Measuring the Judge Itself

The step teams skip is validating the instrument. The procedure is short and the exam expects you to know it exists:

• Draw a sample — typically 50 to 100 items spanning the score range.
• Have human raters score them against the same rubric.
• Measure agreement between judge and humans. Exact agreement is a weak statistic on an ordinal scale; correlation or a chance-corrected agreement statistic is more informative.
• If agreement is poor, the fix is usually the rubric, not the judge model — ambiguous anchors produce disagreement among humans too, which is itself the diagnostic.
• Re-measure periodically. Traffic changes, and a judge calibrated on last quarter's inputs can quietly drift out of usefulness.

A judge that agrees closely with human raters can then score volumes no human review could reach. A judge that has never been compared to humans is producing numbers of unknown meaning, and reporting them as quality metrics is worse than reporting nothing, because they carry unearned authority.

Evaluating RAG Systems in Two Halves

A retrieval-augmented system has two failure modes and they need separate measurement:

The reason to separate them: an answer that sounds well-grounded but was built on the wrong documents is a retrieval failure, and no amount of prompt tuning fixes it. Conversely, correct documents plus a hallucinated answer is a generation failure. Reporting one blended score hides which component to fix.

Two practical notes on the retrieval half. First, it needs a labelled probe set — queries paired with the documents that should be retrieved — and building one is unglamorous work that teams defer indefinitely. A few dozen well-chosen queries covering the corpus's main topics is enough to detect a regression. Second, citation resolution is a free deterministic check that requires no judge at all: verify that every source the answer cites actually appears in the set of retrieved documents. A citation that resolves to nothing is a fabrication, caught by string matching.

Building an Evaluation Set

• Representative, including hard cases. Sample from real traffic, then deliberately add edge cases, adversarial inputs, and known failure modes.
• Versioned and stable. Changing the eval set between runs invalidates comparison.
• Held out from tuning. An eval set used to select prompts becomes a training set; keep a separate final holdout.
• Sized for the decision. A few hundred well-chosen examples usually distinguishes candidates; tens of thousands of easy examples do not.

Exam Traps

• ROUGE as a factuality check. It measures overlap, not truth.
• A model judging itself. Self-preference bias.
• An unanchored 1-to-10 scale. Produces noise, not measurement.
• One blended RAG score. Measure retrieval and generation separately.
• An uncalibrated judge. Agreement with human raters is what makes it an instrument.
• Selecting prompts on the final holdout. That holdout is now a training set.