14.1 Continuous Monitoring in Microsoft Foundry

Continuous Monitoring in Microsoft Foundry

Quick Answer: Continuous monitoring in Microsoft Foundry samples production agent traffic, runs quality and safety evaluators as responses complete or on a schedule, and charts the scores next to latency, tokens, and errors on the Agent Monitoring Dashboard. Telemetry lives in the project's Azure Monitor Application Insights resource. This is not an Azure Machine Learning tabular data-drift monitor.

Domain 4 asks you to examine continuous monitoring in Foundry. Chapter 13 covered pre-production evaluation: golden datasets, groundedness, relevance, coherence, fluency, and risk/safety evaluators. This chapter is what you attach after the agent is live. Microsoft's observability concept page (updated 2026-07-31) names three pillars that work together: evaluation, monitoring, and tracing. You already built evaluators. Monitoring is how those evaluators, plus operational metrics, stay attached to real users. (Microsoft Foundry is the current brand for what many materials still call Azure AI Studio or Azure AI Foundry.)

Why Foundry monitoring is not an Azure Machine Learning model monitor

Chapter 9 taught Azure Machine Learning production monitors: a scheduled Spark job that compares tabular production model inputs (and optionally predictions) against a reference distribution. Signals are data drift, prediction drift, and data quality. A threshold breach can email you or start a retraining pipeline.

Generative AI rarely fails that way first. A retrieval-augmented generation (RAG) agent can keep the same embedding space and still start hallucinating, citing stale chunks, or emitting unsafe completions because a prompt version changed, a tool started returning empty JSON, or a model deployment rolled forward. Foundry continuous monitoring therefore scores outputs (and traces) for quality and safety over time, not feature histograms. If an exam item mentions population stability or feature distributions, that is Chapter 9. If it mentions sampled production traffic, groundedness on live traces, or the Agent Monitoring Dashboard, that is this section.

The post-production loop

Microsoft's post-production list is the exam checklist. After deployment you:

• Measure operational metrics (token usage, latency, run success rate, errors).
• Run continuous evaluation: quality and safety on production traffic at a sampled rate.
• Run scheduled evaluation: the same evaluators against a test dataset on a cron so you detect system drift even if today's users ask easy questions.
• Run scheduled red teaming: adversarial scans for leakage and prohibited actions.
• Fire Azure Monitor alerts when outputs miss quality thresholds or produce harmful content.

Those five are complementary. Continuous evaluation tells you what users are experiencing now. Scheduled evaluation on a held-out golden set tells you whether the system (prompt, index, tools, model) drifted even if the traffic mix changed. Red teaming probes safety that ordinary users might never trigger. Operational metrics catch timeouts and 429s that evaluators never see because the run never completed.

Continuous evaluation versus scheduled evaluation

The Monitor tab on a Foundry agent (New Foundry portal: Build → agent → Monitor) is the operator view. The gear icon opens Monitor settings. Two recurring evaluation types matter:

• Continuous evaluation samples live traffic as it occurs. In the SDK this is an evaluation rule with event type RESPONSE_COMPLETED, an EvaluationRuleFilter on agent_name, and a ContinuousEvaluationRuleAction that points at an eval definition. max_hourly_runs caps how many sampled runs are judged per hour. Microsoft documents a default of 100 and skipped runs when the cap is hit.
• Scheduled evaluation uses a recurrence trigger (example: daily at 09:00) and a trace or dataset data source. Use it for benchmarks and drift against a fixed set.

The portal wizard also lets you pick Live traffic versus a Dataset, a turn level, and Random or Intelligent sampling, plus a maximum number of traces. Classic Foundry continuous evaluation exposed samplingPercent (0–100) and maxRequestRate with a system limit of 1,000 requests per hour. Do not memorize a single percent as "the official sampling rate." The exam cares that you sample rather than score every production call, and that you cap hourly judge traffic.

To create continuous rules, assign the project managed identity the Foundry User role (formerly named Azure AI User; role IDs are unchanged). Without that assignment, rule creation can look fine in your user context and then silently skip.

Sampling rate versus the cost of judge models

AI-assisted evaluators (groundedness, relevance, violence, protected material, and most custom large language model judges) call another model. Observability features such as risk/safety evaluations and agents playground evaluations are consumption-billed on the Foundry Observability meter. Playground evaluations are on by default; turn them off in the playground metrics panel if you are iterating and do not want to pay for every try.

Sampling is how you stay solvent:

• 100 percent sampling on a busy agent plus five LLM judges can cost more than the production model.
• Too low a rate hides a bad prompt that only hits a small slice of traffic.
• Prefer higher sampling on safety evaluators than on expensive quality judges if budget is tight.
• Intelligent sampling (portal) tries to spend the hourly budget on interesting traces rather than duplicates.
• When runs are skipped, raise max_hourly_runs or lower sampling. Raising the cap without watching the judge deployment's tokens-per-minute limit just moves the 429s to the evaluator.

Dashboard metrics you must be able to read

Microsoft's Agent Monitoring Dashboard definitions (docs dated 2026-08-08):

• Token usage: counts for the selected window. Spikes often mean verbose prompts or long completions.
• Latency: run response time. Values above about 10 seconds warrant a look at throttling, complex tool calls, or network.
• Run success rate: completed runs. Below about 95 percent is an investigation trigger, not a product pass/fail baked into every SKU.
• Evaluation metrics: scores from sampled outputs; interpretation is evaluator-specific (Chapter 13).
• Red teaming results: failed scans are security work, not "the model is slow."

Empty charts usually mean no recent traffic, a time range that excludes data, or ingestion delay — generate traffic and wait a few minutes before assuming the connection is broken. Authorization errors mean missing Log Analytics Reader (and Privileged Monitoring Data Reader if tables are protected).

You can also monitor custom agents that do not run in Foundry: register them in the control plane, emit OpenTelemetry generative-AI semantic conventions, and send traces to the same Application Insights instance. Continuous evaluation then has a data plane to sample.

Scenario

A benefits chatbot goes live Monday. Groundedness on the Chapter 13 golden set was 0.86. By Thursday the Monitor chart shows groundedness on sampled live traffic at 0.61 while the nightly scheduled golden-set eval is still 0.85. The agent is not "drifting" in the tabular sense. Users started asking about a policy PDF that is not in the index. Continuous evaluation caught the production mix; scheduled evaluation proved the prompt and model still pass the old tests. The fix is retrieval coverage, not a Spark data-drift job and not an immediate model swap.

Common trap

Do not turn on 100 percent continuous evaluation with a large judge ensemble "for safety" and then blame the production deployment for 429s and latency. Judge tokens share quota and cost. Sample. Cap hourly runs. Keep scheduled golden-set evals for regression. Do not use an Azure Machine Learning data collector plus model monitor as a substitute for Foundry continuous evaluation — it will never score groundedness.

Operator checklist

• Connect Application Insights before you expect charts or traces.
• Use continuous eval for live mix; scheduled eval for golden-set drift; red team on a schedule for adversarial coverage.
• Sample; set max_hourly_runs; watch judge spend.
• Grant Foundry User to the project managed identity.
• Alert on eval score, latency, tokens, and red-team findings — not only on HTTP 500s.