6.5 Monitoring, Testing and Evaluating Generative AI in Production
Key Takeaways
- Generative systems have no ground-truth label stream, so production quality is tracked through continuous automated evaluation on sampled traffic rather than through accuracy against outcomes.
- The production signal set is groundedness, safety-filter and Model Armor trigger rates, refusal rate, latency, token cost per request, and user feedback.
- Managed base models are updated by the provider, so a versioned regression suite must be re-run against any new version before it is adopted.
- Retrieval health is monitored separately from generation quality, because a fall in answer quality is usually a retrieval regression.
- Sampled human review remains necessary to keep an automated judge calibrated and to catch failure modes no rubric anticipated.
6.5 Monitoring, Testing and Evaluating Generative AI in Production
Blueprint reference: Section 6.2, "Monitoring, testing, and evaluating gen AI solutions."
Section 6.3 covered monitoring for a predictive model: a baseline distribution, statistical distance metrics, skew and drift alerts. Generative systems break most of those assumptions, and the exam expects you to know why and what replaces them.
Why Classical Monitoring Does Not Transfer
| Predictive model | Generative system |
|---|---|
| Fixed input schema | Free-form text, images, documents |
| Bounded output space | Unbounded output space |
| Ground-truth labels arrive eventually | Usually no label ever arrives |
| Feature distribution comparison is meaningful | "Distribution of the output text" is not a usable statistic |
| Accuracy computable from outcomes | Quality must be judged, not computed |
The consequence: production quality is established by continuously evaluating a sample of live traffic with the same rubric-based methods used offline, rather than by waiting for labels that never come. Offline evaluation methodology — judge design, rubric anchoring, calibration, RAG's two halves — is covered in Section 2.9; this section is about running it against production.
What to Monitor
| Signal | Why it matters | Typical alert |
|---|---|---|
| Groundedness / faithfulness on sampled traffic | The primary quality signal for RAG and summarization | Rolling score falls below threshold |
| Safety filter and Model Armor trigger rates | A spike means an attack campaign or a policy regression | Rate exceeds baseline band |
| Refusal rate | Rising refusals mean the model has become over-cautious or prompts have changed | Sustained increase |
| Latency p50/p95 | User experience and capacity | Breach of budget |
| Token cost per request | Prompt growth and verbosity creep are silent budget killers | Rolling cost per request rises |
| Retrieval hit rate / recall proxy | The most common root cause of quality loss | Fewer relevant documents retrieved |
| User feedback (thumbs, escalations, edits) | The only direct signal of perceived quality | Ratio deteriorates |
| Empty or malformed structured output | Schema violations break downstream systems | Any sustained rate |
Cost per request deserves specific attention. Prompts accumulate: an extra few-shot example here, a longer system instruction there, a larger retrieved context. None of it fails a test, and the bill grows steadily. Tracking tokens per request as a monitored metric catches it.
Continuous Evaluation on Sampled Traffic
The mechanics:
- Sample production requests and responses at a rate that is statistically useful and affordable. Sample deliberately across segments rather than taking the first N.
- Score the sample with the automated judge and rubric, and with any deterministic checks available — did the structured output parse, did every citation resolve to a retrieved document, did the response respect the required format.
- Log the scores as a time series so trends are visible and alertable.
- Route low scores to human review, both to confirm the finding and to build the next round of regression cases.
Deterministic checks are underrated because they are cheap and unambiguous. Citation resolution in particular — does every cited source actually appear in the retrieved set — catches a large fraction of grounding failures without any judge at all.
Regression Testing Against Model Updates
This is the generative-specific concern that has no analogue in classical monitoring. Managed base models are updated by the provider. A version change can alter tone, formatting, refusal behaviour, or reasoning in ways that break a downstream application built around the previous behaviour.
The discipline:
- Keep a versioned regression suite — the eval set from model selection, plus every incident that has been fixed, retained as a test case.
- Pin the model version in production where the platform allows it, so changes are adopted deliberately.
- Re-run the suite against any new version before adopting it, comparing per-dimension scores rather than a single aggregate.
- Re-run it after any prompt change too. A prompt edit is a code change to the system's behaviour and deserves the same gate.
Monitor Retrieval Separately
For a retrieval-augmented system, answer quality falling is usually a retrieval regression: the index grew stale, an embedding model changed, a document set was re-chunked, or a filter is now excluding relevant material. Monitoring only end-to-end quality tells you something broke without telling you which half.
Track on the retrieval side: index freshness and document count, retrieved-document relevance on a labelled probe set run periodically, the rate of queries returning nothing, and latency of the retrieval step.
Human Review and Feedback Loops
Automated judges drift out of calibration as traffic changes and cannot recognize failure modes their rubric never anticipated.
- Sample for human review continuously, not only during incidents.
- Re-measure judge-human agreement periodically; falling agreement means the judge needs recalibration or the rubric needs extending.
- Feed confirmed failures back into the regression suite so the same defect cannot silently return.
- Treat explicit user feedback as a monitored signal, remembering it is biased toward extremes — most satisfied users say nothing.
Exam Traps
- Applying feature-drift monitoring to free-form text. The statistic is not meaningful.
- Waiting for ground-truth labels. They usually never arrive.
- Adopting a new base model version without re-running the regression suite.
- Monitoring only end-to-end quality in a RAG system; measure retrieval separately.
- Trusting an automated judge indefinitely without re-checking human agreement.
- Ignoring token cost per request as prompts quietly grow.
A team wants to monitor a production RAG assistant using the same approach as their tabular models: comparing the distribution of inputs against a training baseline with statistical distance metrics. Why is this inadequate, and what should they do instead?
A production assistant's answer quality degrades noticeably over two weeks. The prompt and the model version are unchanged. What should be investigated first?
A managed base model is updated by the provider. The application's prompts and retrieval are unchanged. What should the team do before adopting the new version?
Over three months, a generative feature's monthly bill grows 40% while request volume is flat and the model tier is unchanged. What monitored signal would have surfaced this earliest?
You've completed this section
Continue exploring other exams