19.3 Monitoring, Testing & Evaluating Gen AI Solutions in Production
Key Takeaways
- The Agent Platform model observability dashboard reports metrics for fully managed models, such as requests per second, token throughput, first token latency, and API error rates.
- Request-response logging (preview) saves samples of Gemini and supported partner model requests and responses to BigQuery for analysis and evaluation.
- Online monitors for agents continuously evaluate production traces and export evaluation scores to Cloud Monitoring, where quality alerts can notify teams through Slack, email, or Pub/Sub.
- Model Armor logs and insights show prompt injection attempts, sensitive data detections, and harmful content blocks for monitored AI applications.
- Gen AI regression testing reruns a fixed evaluation set with Gen AI evals whenever prompts, models, tools, or retrieval settings change.
The last consideration in the exam guide is monitoring, testing, and evaluating gen AI solutions. A gen AI app needs the same operational monitoring as any service, plus quality, safety, and cost monitoring specific to LLMs.
What to Monitor
| Dimension | Signals | Google Cloud tools |
|---|---|---|
| Operational health | Requests per second, error rates (429, 5xx), time to first token, end-to-end latency, token throughput | Model observability dashboard (fully managed Gemini and partner models), Cloud Monitoring |
| Cost | Input, output, and cached tokens per request, requests per user or feature | Token usage metrics, billing exports, usageMetadata in responses |
| Quality | Rubric pass rates, groundedness, instruction following, task success, hallucination rate | Gen AI evals on sampled logs, online monitors for agents, human review |
| Safety and security | Safety filter blocks, prompt injection detections, sensitive data detections, malicious URLs | Model Armor logs and insights, Gemini safety ratings |
| User outcomes | Thumbs up or down, escalations to humans, task completion | Application analytics, Feedback service (preview) for agents |
| Behavior drift | Changes in topics, languages, prompt lengths, tool call patterns | Logged request analysis in BigQuery, agent traces |
Operational Monitoring
- The model observability dashboard covers fully managed models (Gemini and partner MaaS). It shows model requests per second, token throughput, first token latencies, and API error rates. Self-hosted model metrics aren't included, so use endpoint metrics for those (Chapter 14). Console usage such as Agent Studio isn't counted.
- Alert on 429 rates (capacity), latency percentiles, and token spikes, such as a prompt change that doubles context size.
Logging Requests and Responses
Request-response logging (preview) saves samples of requests and responses for Gemini models (generateContent, streamGenerateContent) and supported partner models such as Anthropic Claude to a BigQuery table, for base and tuned models. Uses:
- Build evaluation datasets from real traffic.
- Investigate complaints and incidents.
- Analyze topic and usage drift.
Treat these logs as sensitive data: restrict access, set retention, and de-identify where possible (Chapter 5).
Quality Monitoring and Evaluation in Production
- Sample production interactions from logs or traces.
- Evaluate them with Gen AI evals: adaptive rubrics such as GENERAL_QUALITY with your guidelines, GROUNDING for RAG answers, SAFETY, and custom functions such as JSON validity or citation presence (Chapter 7).
- Track scores over time in dashboards, sliced by feature, language, and user segment.
- Alert when scores fall below thresholds.
Online monitors for agents
For agents deployed on Agent Platform, online monitors continuously evaluate production traces, which requires Cloud Trace and OpenTelemetry gen AI signals such as agent name, conversation ID, input and output messages, system instructions, and tool definitions. They detect quality drift, meaning performance that declines over time even when the model is unchanged, because user behavior or data shifts. Evaluation scores are exported to Cloud Monitoring, where quality alerts notify teams through Slack, email, or Pub/Sub and link back to the traces that caused them.
Safety and Security Monitoring
- Model Armor logs every detection to Cloud Logging. In Inspect only mode this measures violation rates without blocking, and Model Armor insights summarize prompt injection attempts, sensitive data leakage, and harmful content across AI agents.
- Track Gemini safety ratings and
finishReasonvalues (SAFETY,RECITATION,SPII,PROHIBITED_CONTENT). A spike can signal an attack or a prompt regression. - Watch for false positives too: legitimate requests being blocked hurt users and suggest thresholds need tuning.
Testing Gen AI Before Release
| Test type | Purpose |
|---|---|
| Regression evaluation | Rerun a fixed evaluation set on every change to prompts, model versions, tools, retrieval settings, or safety thresholds |
| Pairwise comparison | Candidate vs. current version with an LLM judge and response flipping |
| Adversarial and red-team tests | Prompt injection, jailbreaks, data extraction attempts, off-topic abuse |
| Grounding tests | Questions whose answers must come only from provided documents |
| Load and cost tests | Latency and token usage at expected traffic |
| Canary release | A small share of users, with quality and safety monitoring before full rollout (Chapter 13) |
Common Gen AI Production Issues
| Symptom | Likely cause | Response |
|---|---|---|
| Groundedness scores fall after a document update | Retrieval index stale or chunking changed | Re-index, check retrieval quality, add retrieval evaluation |
| Token cost per request doubles | Prompt template change added context, or caching stopped hitting | Review prompt diffs and cachedContentTokenCount |
| Quality drops after a model version upgrade | Behavior change in the new version | Roll back the version, re-tune prompts, rerun evaluations |
| Spike in prompt injection detections | Active attack | Switch Model Armor to Inspect and block, add rate limits, investigate sources |
| Rising 429 errors at peak | Capacity limits | Provisioned Throughput, global endpoint (if residency allows), backoff (Chapter 4) |
| User thumbs-down on one topic | Missing knowledge or prompt gap | Add grounding content, update instructions, expand the evaluation set |
Worked Scenario
A health insurer's member-support assistant (Gemini with RAG) launches with:
- Dashboards: first token latency p95, 429 rate, tokens per conversation, and cost per day.
- Daily evaluation: 500 sampled conversations from request-response logs scored with GENERAL_QUALITY (guidelines: no medical advice, cite plan documents), GROUNDING, and a custom citation-presence check. Alerts fire if the grounding pass rate drops below 90%.
- Security: Model Armor input and output templates with Cloud Logging, with a weekly review of injection and sensitive-data detections.
- Change control: any prompt, model, or retrieval change must pass the regression evaluation set and a 5% canary before full rollout.
A team's Gemini-based assistant shows normal latency and error rates, but users report more fabricated answers since the knowledge base was reorganized. Which monitoring approach detects this issue directly?
Which tool gives a prebuilt view of requests per second, token throughput, first token latency, and API error rates for Gemini and partner models used through fully managed APIs?
Before upgrading an assistant from one Gemini model version to a newer one, what testing approach best reduces the risk of quality regressions?
You've completed this section
Continue exploring other exams