6.2 Continuous evaluation, synthetic test data & cost monitoring

Continuous evaluation, synthetic test data & cost monitoring

A GenAI system that passed every pre-release gate can still degrade in production: real user inputs differ from your test set, retrieval sources change, and the hosted model may be updated without notice. Continuous evaluation (production monitoring) is the discipline of measuring quality on live traffic, not only before release. The aim is to detect drift, quality regressions, and rising unsafe output while the problems are still small.

Signals to monitor

Canary, A/B, and feedback loops

You rarely change everything at once. A canary release routes a small slice of traffic to the new prompt or model and compares its live metrics to the current version before full rollout. A/B testing runs two versions in parallel to see which performs better on quality and business metrics. Both reuse the same evaluation harness built for pre-release testing, now applied online. User feedback loops — explicit ratings plus implicit signals such as regeneration or abandonment — supply a stream of real failure cases; the most valuable are triaged and promoted into the golden dataset, so tomorrow's regression tests cover today's real problems.

Detecting drift without a fixed answer key

In production you rarely have a labeled correct answer for live traffic, so continuous evaluation leans on reference-free techniques: an LLM-as-judge scoring responses against a rubric, groundedness checks that verify each claim against retrieved sources, and statistical alarms on shifts in input distribution or output length. Because judging every request is expensive, teams sample — evaluating a representative slice of live traffic on a rolling window and alerting when the rolling score crosses a control limit, rather than reacting to any single output.

Synthetic test data

Labeled real data is scarce, so teams use LLMs to generate synthetic test data — extra prompts, edge cases, paraphrases, and adversarial inputs — to expand coverage cheaply. It is especially useful for rare scenarios, under-represented user groups, and situations that are hard to collect in the wild.

Synthetic data carries risks the tester must actively manage:

• Bias amplification: a model tends to produce cases resembling its own training distribution, so a synthetic set can reinforce existing blind spots instead of filling them.
• Unvalidated ground truth: if one model both writes a test and its expected answer, an error becomes a wrong oracle — you check the system against a mistake.
• Lack of realism: synthetic inputs may miss the messy, unexpected phrasing of real users (distribution shift).
• Duplication / leakage: near-duplicate items inflate scores without adding real coverage.

The rule: synthetic data must be validated — reviewed by humans or checked against trusted references — and used to complement, never replace, real data. Good practice treats synthetic generation as a coverage tool guided by real gaps: identify under-tested topics or user segments from production data, generate candidates for exactly those gaps, then confirm each expected answer with a human or a trusted reference before it enters the golden set.

Cost monitoring and budget-aware testing

GenAI evaluation is not free: every eval item is one or more model calls priced by input and output tokens. A large golden set, run on every commit across several candidate configurations, can cost more than the feature it guards. Testers therefore treat cost as a first-class test dimension.

Practical practices:

• Track token usage and cost per eval run, and cost per request in production.
• Set budgets and alerts so a runaway prompt (for example an accidental huge context) is caught early.
• Use a tiered strategy: a small, fast smoke eval on every commit; the full golden-set regression nightly or before release.
• Prefer cheaper or smaller models and sampling for routine monitoring, reserving expensive full evaluations for release gates.
• Watch cost drift itself — a change that lengthens outputs raises spend even when quality is unchanged.

Balancing thoroughness against spend

Budget-aware testing is a risk decision, not a corner cut. High-impact or safety-critical paths justify a large, frequently run evaluation; low-risk cosmetic changes do not. Teams often cache results for unchanged prompt-and-model pairs so identical items are not re-scored, deduplicate the golden set to remove near-identical items, and cap output length in eval calls to control output-token cost. Reporting cost alongside quality lets stakeholders see the true price of a given confidence level and decide, deliberately, how much assurance they are buying.

Continuous evaluation, synthetic data, and cost monitoring form one loop: monitor live signals, mine real failures and add validated synthetic cases to grow the golden set, all within a budget that keeps evaluation sustainable. This is how testers keep a GenAI product responsible and reliable long after launch day.