7.2 Evaluating Gen AI: Computation Metrics, Rubrics & LLM-as-a-Judge
Key Takeaways
- Gen AI evals on Agent Platform support adaptive rubrics, static rubrics, computation-based metrics, and custom function metrics.
- Adaptive rubrics generate a unique set of pass/fail tests for each prompt, and Google recommends them as the default starting point.
- Computation-based metrics such as exact_match, BLEU, and ROUGE need a ground-truth reference and use deterministic algorithms.
- Response flipping reduces position bias in pairwise judging by swapping baseline and candidate responses in half of the judge calls.
- A judge model should be validated against human ratings using metrics such as balanced accuracy and balanced F1 before its scores are trusted.
The exam guide explicitly names LLM-as-a-judge. Gen AI evaluation matters for model selection (Chapter 4), prompt changes, tuning, migration to new model versions, and production monitoring (Chapter 19). Agent Platform's Gen AI evals (formerly the Gen AI evaluation service) provides the tools.
Step 1: Build the Evaluation Dataset
Evaluation is only as good as its dataset. Gen AI evals supports three ways to build one:
- Upload complete prompts, or a prompt template plus a file of variable values.
- Sample production logs to reflect real usage.
- Generate synthetic data from a prompt template to get many consistent examples.
Include reference answers (ground truth) if you plan to use computation-based metrics. Cover edge cases: long inputs, ambiguous requests, adversarial prompts, multiple languages, and anything that previously failed.
Step 2: Choose Metric Types
| Metric type | How it works | Output | Best for |
|---|---|---|---|
| Adaptive rubrics (recommended) | A judge generates pass/fail tests unique to each prompt, then checks the response against each | Pass rate per response, with a rationale per rubric | Fast, task-relevant evaluation of open-ended outputs |
| Static rubrics | One fixed rubric applied to every prompt | A numeric score (such as 1-5) | Consistent benchmarks on a specific dimension |
| Computation-based | Deterministic comparison with a reference | Score such as 0.0-1.0 | Tasks with a well-defined correct answer |
| Custom function (SDK) | Your Python logic, run locally or in a remote sandbox | Your score | Format checks, business rules, domain validators |
Adaptive rubric example
Prompt: "Write a four-sentence summary of the article, keeping an optimistic tone." Generated rubrics: (1) it summarizes the article, (2) it has exactly four sentences, (3) it keeps an optimistic tone. A response that passes the first two but ends on a negative note fails the third, for a 66.7% pass rate. Comparing two models on the same rubrics shows exactly which requirement each one misses.
Predefined adaptive metrics include GENERAL_QUALITY (the recommended default, which can be guided with natural-language guidelines such as "must not give financial advice"), INSTRUCTION_FOLLOWING, and TEXT_QUALITY, plus multi-turn and agent metrics (for example, hallucination and tool_use_quality).
Static rubric and template metrics
GROUNDING checks factual consistency against provided source text, which is essential for RAG. SAFETY checks policy violations such as hate speech or dangerous content. Template metrics such as FLUENCY are also available. You can define custom static rubrics with your own criteria and rating scale, but Google recommends trying GENERAL_QUALITY with guidelines first.
Computation-based metrics
| Metric | Measures | Typical task |
|---|---|---|
| exact_match | Share of responses identical to the reference | Short factual answers, labels, IDs |
| BLEU | Matching n-grams against a reference | Translation |
ROUGE (rouge_1, rouge_l) | N-gram or longest-common-subsequence overlap | Summarization |
Overlap metrics reward wording similarity, not correctness. A correct paraphrase can score low, and a fluent wrong answer that reuses reference words can score high. Pair them with rubric-based or human checks.
Custom function metrics
A custom function receives each instance (prompt, response, reference, and other data) and returns a score, for example "is the output valid JSON with all required fields?" Remote custom functions run in a sandbox with no network access, a 1-minute execution limit, and a 1.5 GB limit on code plus loaded data.
Step 3: Pointwise vs. Pairwise Judging
- Pointwise: score each response on its own against criteria. Good for tracking absolute quality over time.
- Pairwise (side-by-side): a judge compares a baseline and a candidate response and picks the better one or a tie. Good for "is the new prompt or model better than the current one?"
LLM-as-a-Judge: Making Scores Trustworthy
Using a model as the judge scales far beyond human review, but judges have biases. Agent Platform provides configuration options:
| Option | Purpose |
|---|---|
| System instructions for the judge | Set the evaluator role and rules, such as "You are an expert claims auditor" |
| Response flipping | In pairwise judging, swap baseline and candidate positions in half of the calls to reduce position bias |
| Multi-sampling | Call the judge several times per item and aggregate, which improves consistency |
| Tuned judge model | Use a judge tuned on your domain's human ratings |
Validate the judge against humans
Before trusting a judge, collect human ratings for a sample and compare. Add columns such as {metric}/human_rating (pointwise) or {metric}/human_pairwise_choice (pairwise) and compute agreement:
- Balanced accuracy and balanced F1 for two-class results (pass/fail, A/B).
- Multi-class balanced accuracy and F1 for 1-5 scales or A/B/tie.
- Confusion matrices to see how the judge disagrees. For example, it may call too many ties.
If agreement is weak, refine the rubric or guidelines, add judge system instructions, enable multi-sampling, or tune the judge.
Matching Metrics to Common Gen AI Tasks
| Task | Primary metrics |
|---|---|
| Classification or extraction with fixed labels | exact_match or custom field-level accuracy functions |
| Translation | BLEU plus a fluency or rubric check |
| Summarization | ROUGE for overlap, GROUNDING for faithfulness, rubrics for length and tone |
| RAG question answering | GROUNDING, adaptive rubrics, and a custom check for citations |
| Chat assistant | GENERAL_QUALITY and INSTRUCTION_FOLLOWING, multi-turn quality, SAFETY |
| Agent with tools | tool_use_quality, hallucination, final response quality |
Putting It Together: Evaluation-Driven Development
- Define success criteria with stakeholders, such as "cites policy, under 120 words, no financial advice."
- Build a dataset from production samples plus curated edge cases.
- Use GENERAL_QUALITY with guidelines, plus GROUNDING for RAG and a custom JSON validity function.
- Validate the judge on 200 human-rated items.
- Run baseline vs. candidate pairwise comparisons with response flipping for every prompt, model, or tuning change.
- Gate releases on pass rates in CI/CD (Chapter 17), and sample production traffic for ongoing evaluation (Chapter 19).
BigQuery ML also offers SQL-native evaluation: ML.EVALUATE on remote LLM models returns BLEU and ROUGE-L for text generation (Chapter 2), and AI.EVALUATE scores TimesFM forecasts.
A team compares a new prompt (candidate) with the current prompt (baseline) using a pairwise LLM judge and notices the judge favors whichever response appears first. Which configuration addresses this?
A support-answer generator has no single correct wording, and each customer question carries different requirements (steps, tone, policy limits). Which Gen AI evals approach does Google recommend as the default starting point?
Before relying on an LLM judge to gate releases, how should a team confirm the judge's scores are trustworthy?