13.2 Groundedness, Relevance, Coherence, and Fluency

Groundedness, Relevance, Coherence, and Fluency

Quick Answer: Groundedness = the response is supported by context (needs retrieved text). Relevance = the response addresses the query. Coherence = logical flow. Fluency = language quality. Scores are typically 1–5 with default pass threshold 3. Similarity and F1 need ground_truth. LLM judges need a deployment. Groundedness is not relevance.

The official AI-300 skill is implement AI quality metrics (groundedness, relevance, coherence, fluency). Microsoft Foundry’s catalog implements those as built-in evaluators. You already mapped columns in section 13.1. This section is which metric to trust for which failure.

Quality is a panel, not one number

A claims agent can be fluent (beautiful English) and incoherent (contradicts itself), relevant (talks about deductibles) and ungrounded (invents a $0 deductible the handbook never stated), or grounded (quotes a stale chunk) and irrelevant (ignores the actual question). Promoting a prompt because fluency went from 3 to 5 while groundedness fell from 5 to 2 is a regression. Chapter 12 already said that; here are the evaluators that make it measurable.

Microsoft’s RAG guidance treats groundedness as precision (nothing extra that is not in context) and response completeness (preview) as recall versus ground_truth (nothing critical missing). Retrieval evaluators (builtin.retrieval, builtin.document_retrieval) score the search step, not the final sentence — Domain 5 goes deeper on RAG tuning; you still need to know that a bad retrieval score explains a later groundedness failure.

Recommended RAG application combo from the built-in catalog: Retrieval + Groundedness + Relevance + content safety. That is the exam bundle, not fluency alone.

Groundedness is not relevance

Groundedness asks: given this context, did the model stick to it? If the handbook says the waiting period is 90 days and the model says 30, groundedness fails even if 30 days would be a "relevant" sounding benefits answer. If the model recites the 90-day paragraph in perfect prose but the member asked about network providers, relevance fails even if every sentence is in the chunk.

Provide query + response + context for the best groundedness results; Microsoft says query is optional but improves scoring. Concatenate chunks. For agents, context can be omitted when tool-call results are in the trace and the evaluator can extract them — do not rely on that on a dataset evaluation of plain strings. Groundedness Pro is the strict Content Safety variant: no judge deployment, boolean label. Use it when you want a service-backed binary gate; use the GPT judge when you want a 1–5 rubric and a reason string.

Relevance does not take context. It cannot tell you the answer was copied from the wrong document. That is why you run both.

Coherence, fluency, and the English-language caveat

Coherence is whether a reader can follow the argument: order, transitions, no self-contradiction. It needs query and response so the judge can see whether the answer actually addresses the question in a logical sequence. Set evaluation_level to conversation when you care about flow across turns; all evaluators in that run must support conversation level — you cannot mix a turn-only metric into the same run.

Fluency is surface language quality. It needs only response. A fluent hallucination still scores well. Microsoft notes reliability drops on very short responses (under about 20 tokens) and that both general-purpose evaluators currently support English. Do not certify a Spanish agent with fluency 5 on English judges.

For the best cost/quality balance on LLM judges, Microsoft currently calls out gpt-5-mini. Older local-SDK docs mention gpt-4o-mini. The exam skill is a chat-capable deployment in the project, not a specific SKU you memorize as eternal. No deployment means the quality evaluator cannot run (except Groundedness Pro and the algorithmic overlap metrics).

When ground truth exists: similarity and F1

If adjusters already wrote the right answer, add textual similarity evaluators:

• Similarity — LLM semantic match of response to ground truth with respect to the query. Paraphrases can still pass.
• F1 — shared tokens; precision is overlap versus the generation, recall is overlap versus the truth.
• BLEU / GLEU / ROUGE / METEOR — n-gram overlap family; ROUGE returns precision, recall, and F1. Useful for tight wording, harsh on valid paraphrases.

These metrics require ground_truth. They do not require context. A response can F1-match a stale expert answer that is no longer in the index. Use them as a second opinion next to groundedness, not as a replacement.

Output fields you will see on a judge metric: score, label (pass/fail), reason, threshold, passed. Default quality threshold 3 on the 1–5 scale means 3, 4, and 5 pass. Raise it for high-risk answers; document the change in the evaluation definition so CI does not silently use 3.

Exam scenario

A prompt variant for Contoso’s member chat raises average fluency from 3.4 to 4.6 and coherence from 3.1 to 4.2 on the golden set. Groundedness falls from 4.5 to 2.1 because the new system message says "be helpful even if unsure" and the model fills gaps. Relevance stays at 4.0. You reject the variant. You restore the "answer only from context / I don't know" rule, rerun, and accept only when groundedness is back at or above the threshold and relevance has not collapsed. Fluency was never the ship criterion.

Common trap

Treating relevance as proof of groundedness (or the reverse). A second trap is running groundedness without mapping context, then wondering why scores are noise. A third is using F1 as the only RAG metric because it is "objective" and free of a judge — it needs ground truth and ignores retrieval. A fourth is omitting deployment_name for coherence and assuming safety-style hosted models will judge writing quality. A fifth is mixing conversation-level and turn-only evaluators in one run.