9.2 Monitor Production Metrics and Trigger Retraining or Alerts

Monitor Production Metrics and Trigger Retraining or Alerts

Quick Answer: A model monitor is a scheduled Spark job that evaluates signals against thresholds you set. Out-of-box monitors email the creator when a threshold is breached. Retraining does not start by itself. Subscribe to Event Grid Run status changed, filter on data.RunTags.azureml_modelmonitor_threshold_breached, and have a Logic App, Azure Function, or Azure Monitor action group submit a training pipeline job. Do not use the legacy Dataset drift detected event. Do not answer these items with Microsoft Foundry generative evaluation.

After you can detect drift, Domain 2 asks you to monitor production performance metrics and configure retraining or alert triggers. Metrics here are the monitor signals plus, when labels exist, true model performance — not token cost or groundedness from Domain 4.

What production metrics means on this exam

Split two families so exam stems cannot mix them.

Statistical and quality signals (no labels required):

• Data drift and prediction drift distances and tests (Jensen-Shannon, PSI, Wasserstein, Kolmogorov-Smirnov, chi-squared).
• Data quality rates (null, type error, out-of-bounds).
• Feature attribution drift NDCG (preview; needs a training reference and both inputs and outputs).

Objective model performance (labels required, preview):

• Classification: accuracy, precision, recall.
• Regression: mean absolute error (MAE), mean squared error (MSE), root mean squared error (RMSE).

You join production predictions to ground truth on a unique ID. The data collector's correlationid can serve if each row is unique; because the collector batches close requests into one JSON object, Microsoft recommends logging your own ID as a column for performance monitoring. If a batch of three rows shares correlation ID test, downstream joins use test_0, test_1, and test_2 unless you logged a dedicated ID column. Ground truth arrives later from the business process (did the loan default? was the transaction fraud?). You own that collection and you register it as a data asset with data_context: ground_truth.

If you do not have actuals, you cannot compute accuracy. Drift and quality still work. That is why best practice is to start monitoring immediately after deploy with drift and quality, then add model performance when labels land.

Out-of-box versus advanced monitors

Create the schedule with SDK/CLI v2 or studio Manage → Monitoring → Add.

Out-of-box (online endpoint plus data collection) does the following:

• Binds the deployment with monitoring_target.endpoint_deployment_id: azureml:<endpoint>:<deployment> and ml_task: classification or regression.
• Turns on data drift, prediction drift, and data quality.
• Uses recent production as the comparison reference and smart default thresholds.
• Emails addresses in alert_notification.emails when any enabled signal exceeds its threshold.

ml_task also lists question_answering for a generative preview path. That path is not the Domain 2 tabular operations loop; keep it out of credit, fraud, and similar classifier answers.

Advanced monitors add:

• Training or validation mltable as reference_data.
• top_n_feature_importance or an explicit feature list.
• Per-signal metric_thresholds and alert_enabled.
• Feature attribution drift and model performance signals.
• Custom signals: a registered component that outputs signal_metrics with columns group, metric_name, metric_value, and threshold_value.

az ml schedule create -f monitor.yaml is the CLI entry. The YAML schema is a schedule document (trigger plus create_monitor), not an online-deployment document. Spark compute.instance_type is required. Recurrence uses frequency (minute, hour, day, week, month) and interval; cron uses a five-field expression. Studio can pick recurrence or cron on the Basic settings page.

Thresholds are a product decision

Microsoft's own guidance: work with the data scientists who own the model. Set thresholds high enough to catch real change and low enough to avoid alert fatigue. A Jensen-Shannon threshold of 0.01 in a sample YAML is an example, not a certified cutoff. PSI "0.1 / 0.25" folklore from credit-risk blogs is not an Azure Machine Learning default. The exam wants you to say thresholds are user-specified per metric and column type, then wired to alerts.

Turn alerts off per signal with alert_enabled: false while you learn the noise floor, then enable them. Include multiple signals so you get both a broad view (did anything move?) and a granular view (which feature? was it nulls or a true histogram shift?).

Alert paths

Event Grid workspace event types include Microsoft.MachineLearningServices.RunCompleted, RunStatusChanged, ModelRegistered, and ModelDeployed. For model monitoring, Microsoft documents a specific recipe:

1. Register the Microsoft.EventGrid resource provider if needed, and create an Event Grid system topic for the workspace if you do not have one.
2. Create an event subscription on the workspace (portal Events, or az eventgrid event-subscription create). Contributor or Owner on the workspace is required.
3. Include only Run status changed. That is Microsoft.MachineLearningServices.RunStatusChanged.
4. Do not select Dataset drift detected. That event belongs to data drift v1, not the v2 model monitor. Selecting it is the classic exam distractor.
5. Advanced filter: key data.RunTags.azureml_modelmonitor_threshold_breached, operator String contains, value has failed due to one or more features violating metric thresholds.
6. Optionally add a second filter on the same tag that contains the monitoring signal name (<monitor-name>_<signal-description>), so one noisy monitor does not wake every handler.
7. Point the handler at Event Hubs, Azure Functions, Logic Apps, Azure Data Factory, or another supported endpoint.

Failed deployments do not raise ModelDeployed. Failed or canceled operations similarly may not raise the happy-path event — design for missing events and still inspect studio. Handlers should check topic and eventType rather than assuming every message is from the expected workspace and type.

Logic Apps can also be started from the workspace Events → Logic apps blade: topic type Machine Learning, event such as RunCompleted, then an email action. That path is excellent for human notification. For retraining, a Function or Logic App HTTP action that submits the training pipeline is the piece that actually changes the model.

Retraining is a pipeline you wire

The monitor never retrains by itself. The intended MLOps loop is:

1. The data collector keeps writing production inputs and outputs, and you append ground truth when it exists.
2. The scheduled monitor fails a threshold and tags the run with azureml_modelmonitor_threshold_breached.
3. Event Grid delivers RunStatusChanged to a Function, Logic App, or an Azure Monitor action group webhook.
4. That handler submits an Azure Machine Learning pipeline job (the training DAG from Chapter 6), often with a new data-asset version that includes recent labeled rows.
5. On success, register the model. ModelRegistered can start a second handler if you want promotion to be event-driven too.
6. Deploy with a progressive rollout and keep a rollback path (Chapter 8). Do not hot-swap 100 percent of traffic from an event handler without a safe deployment policy.

You can also retrain on a time schedule (az ml schedule create on the training pipeline) independent of drift, and keep drift alerts as a human signal. Many regulated shops require a person in the loop. The exam still expects you to know the Event Grid filter that can kick the job.

Not every breach should retrain. A null-rate spike is often a schema break in the request payload. Retraining on broken inputs makes the next model worse. Triage data quality first, then drift, then labeled performance.

Frequency, Spark size, and operational hygiene

• Size production lookback to how fast rows accumulate. Heavy daily traffic → daily monitor (frequency: day, interval: 1). Sparse batch scores → weekly or monthly so each window has enough rows for a distance metric to mean something.
• Spark SKUs: Standard_E4s_v3, Standard_E8s_v3, Standard_E16s_v3, Standard_E32s_v3, Standard_E64s_v3. Monitoring is a Spark workload; undersizing a wide feature table makes the job fail or stall, which is not the same as a drift alert.
• Start monitors the day you deploy, not after the first incident.
• Combine signals: data drift plus feature attribution drift is an early-warning pair; add model performance when labels exist for an objective view.
• Authenticate the job to the datastore with credentials or a user-assigned managed identity attached to the workspace (systemDatastoresAuthMode: identity).
• Managed virtual network setting AllowOnlyApprovedOutbound is not supported for model monitoring.

Interpreting a run in studio

Under Manage → Monitoring, open the monitor. The overview lists the model, endpoint, deployment, and configured signals. Notifications lists features that breached their metric. Opening data_drift shows per-feature metric values and a trend if you have multiple runs. Opening a feature shows production versus reference histograms. data_quality shows null, type-error, and out-of-bounds rates. Use those charts to decide retrain, fix the logging pipeline, or ignore a known campaign spike.

Custom signals exist when built-ins are not enough: register a component, implement any metric (for example a standard-deviation check with a std_deviation_threshold input), emit signal_metrics, and still use the same schedule and Event Grid path. Studio and the Python SDK do not currently author custom signals — CLI YAML does.

Exam scenario

A fraud classifier on a managed online endpoint has data collection on model_inputs and model_outputs. Weekly, analysts attach fraud/not-fraud labels keyed by transaction ID. Accuracy in the last lookback window fell from 0.97 to 0.88, and the data_drift signal on PAY_0 exceeded a Jensen-Shannon threshold the team set at 0.05. Leadership wants "automatic retraining."

What you build: a model_performance signal that joins predictions to the ground-truth data asset on the transaction ID column, with classification thresholds (for example accuracy 0.95). Keep the data-drift signal. Create an Event Grid subscription on Run status changed with the azureml_modelmonitor_threshold_breached filter. The Function submits the existing training pipeline job that reads the latest labeled data asset, then the promotion path from Chapters 7 and 8 registers and rolls out. You do not enable Dataset drift detected. You do not point this at a Foundry evaluation run. You do not assume alert_notification.emails retrained anything.

Common trap

The trap is believing creating a monitor equals continuous retraining. The schedule only computes metrics. Email is the default action. Retrain requires Event Grid (or a separate pipeline schedule) plus a job you already authored. A related trap is selecting Dataset drift detected because the words contain "drift" — that is v1 and will not fire for v2 monitors. Another trap is answering Domain 2 items with Foundry continuous evaluation (groundedness, coherence, token cost). Those are Domain 4. A fourth trap is firing retrain on every data-quality blip without checking whether production logging broke.