9.1 Detect and Analyze Data Drift

Detect and Analyze Data Drift

Quick Answer: Data drift compares the distribution of model inputs in production to a reference set (training data or a recent production window). Prediction drift does the same for model outputs. Azure Machine Learning (SDK/CLI v2) computes these as a scheduled Spark job, not inside the scoring request. Enable the data collector on a managed or Kubernetes online endpoint so production rows land in Blob Storage as model_inputs / model_outputs. Built-in tabular signals are data drift, prediction drift, data quality, feature attribution drift (preview), and model performance (preview). Concept drift is not a named signal.

Exam AI-300 Domain 2 asks you to detect and analyze data drift. Chapter 8 covered deploying and operating online and batch endpoints. This section is what you do after traffic is live: compare what the model now sees with what it was trained on, before business metrics have already collapsed.

Why production models go stale

Unlike a rules engine, a trained model encodes statistical relationships from a snapshot of data. After you deploy, any of the following can change that snapshot:

• Customer mix, seasonality, or a new product line shifts feature distributions (data drift).
• The model's score or class mix shifts even if you have not retrained (prediction drift).
• The relationship between features and the label changes — for example, a new fraud tactic that looks like yesterday's legitimate traffic (concept drift).
• Broken logging, schema changes, or null floods destroy integrity (data quality).

Azure Machine Learning model monitoring is built for tabular classification and regression. It is not the Microsoft Foundry generative AI continuous-evaluation stack (groundedness, fluency, risk and safety). Those appear in Domain 4. If a question mentions an online endpoint, model_inputs, Jensen-Shannon, or a monitor schedule, stay in Azure Machine Learning.

How a monitor actually runs

A monitor is not a live check inside score.py. Scoring stays fast. Collection is a side path:

1. The data collector logs pandas DataFrames from the online endpoint (or you land your own production files for batch or external models).
2. Files land in the workspace Blob datastore, by default under azureml://datastores/workspaceblobstore/paths/modelDataCollector/{endpoint}/{deployment}/.
3. A schedule (az ml schedule create with a create_monitor body, or SDK MonitorSchedule) fires on a recurrence or cron trigger.
4. The job runs on a serverless Spark pool (standard_e4s_v3 through standard_e64s_v3).
5. Spark compares the production lookback window to reference data, evaluates metric thresholds, and can email or raise Event Grid events.

If you skip data collection, the out-of-box online-endpoint monitor has nothing to compute. For batch endpoints or models outside Azure Machine Learning, you must register production inference data as a data asset, keep it updated, and usually supply a preprocessing component that turns a uri_folder into an mltable. That component's contract is data_window_start, data_window_end, and input_data in, preprocessed_data out.

Built-in signals versus the three drift concepts

Keep three concept names separate, then map them onto product signals.

• Data drift — the input distribution moved. Azure Machine Learning has a first-class data_drift signal. Production context is model_inputs.
• Prediction drift — the output (score or class) distribution moved. First-class prediction_drift signal. Production context is model_outputs.
• Concept drift — the mapping from features to the label moved. Azure Machine Learning does not ship a signal named concept drift. You infer it when prediction drift or feature attribution drift appears, especially once ground truth lets you compute model performance.

Out-of-box setup for an online endpoint with data collection enabled turns on data drift, prediction drift, and data quality only. The default reference is recent past production, not training. That is convenient but weaker: you can miss a slow walk away from the original training distribution because each week is compared to last week. Advanced monitors should set reference_data.data_context: training for data drift and data quality, and validation data for prediction drift.

Metrics: pick by column type, not by folklore

Azure Machine Learning does not ask you to implement the formulas. It does expect you to know which metric is legal for numeric versus categorical columns and that you choose thresholds. There is no official universal PSI cutoff on the exam, and sample YAML values such as 0.01 are examples, not certified cutoffs.

• Jensen-Shannon distance — a symmetric distance between two distributions. Allowed for numerical and categorical features on both data drift and prediction drift. Use it when you want one bounded distance that works across column types.
• Population stability index (PSI) — a binned-distribution shift score long used in credit risk. Allowed for numerical and categorical. Useful when the business already speaks PSI; still set the threshold with the model's owners.
• Normalized Wasserstein distance (earth-mover's distance) — numerical only. It reflects how far probability mass moved, not only whether two histograms differ. A small location shift on a continuous feature shows up here even when bins look similar.
• Two-sample Kolmogorov-Smirnov (KS) test — numerical only. It tests whether two samples share a cumulative distribution. Treat it as a hypothesis test on continuous features, not as a category-frequency tool.
• Pearson's chi-squared test — categorical only. It compares category frequencies to the reference. Do not apply it to raw continuous columns.

YAML threshold keys match those names: jensen_shannon_distance, population_stability_index, normalized_wasserstein_distance, and two_sample_kolmogorov_smirnov_test under metric_thresholds.numerical; jensen_shannon_distance, population_stability_index, and chi_squared_test (Pearson's) under metric_thresholds.categorical.

Data quality is not drift

Before you declare that the world changed, prove the pipeline still writes the same schema.

• Null value rate — share of nulls per feature in the production window. Supported for all feature types. If 10 of 100 temperature rows are null, the rate is 10 percent.
• Data type error rate — Spark infers types from reference (PySpark IntegerType, DoubleType, StringType, TimestampType, BooleanType, and similar). Production values that do not match count as errors. Unsupported types skip this metric rather than fail the job.
• Out-of-bounds rate — numeric features use the min/max interval in reference (for example [37, 77]); categorical features use the set of seen values (for example [red, yellow, green]). A new city code or a temperature of 120 raises this rate.

Azure Machine Learning documents precision for these rates down to 0.00001. That is a product fact, not a recommended alert threshold.

Reference windows, lookback size, and offset

Every run compares two time windows. Values are ISO 8601 durations such as P7D (seven days) and P0D (zero days).

• Lookback window size is how wide the slice is.
• Lookback window offset is how far the end of the slice sits before the monitoring run time.

Defaults that Microsoft documents:

• Production size defaults to the monitoring frequency. Offset defaults to P0D (the window ends when the job starts).
• Reference rolling offset defaults to twice the production lookback size so the baseline is large enough to be statistically meaningful.
• You can instead pin reference with window_start_date and window_end_date (a fixed training snapshot).

Do not overlap production and reference windows. If production is the last seven days with zero offset, a reference offset of P7D or more keeps the windows adjacent, not mixed. Overlap makes distance scores look artificially small.

Example: the monitor fires 31 January at 15:15 UTC. Production P7D plus P0D is 24 January 15:15 through 31 January 15:15. Reference offset P7D and size P24D is 1 January 15:15 through 24 January 15:15. A weekly job that must ignore weekends can use production size P5D and offset P2D so the slice is Monday through Friday.

Collecting production inputs

For managed online endpoints and Kubernetes online endpoints, enable collection on the deployment (data_collector.collections). Custom logging uses the azureml-ai-monitoring package:

• Instantiate Collector(name='model_inputs') and Collector(name='model_outputs') in init(). Those two names are auto-registered as data assets the monitor can bind.
• In run(), call collect() on pandas DataFrames only. The context object from the inputs call is passed into the outputs call so rows can be joined later (correlationid).
• Optional: BasicCorrelationContext to supply your own unique ID. If you skip it, Azure Machine Learning generates correlationid. The collector batches near-simultaneous requests into one JSON object, so put a business ID in its own DataFrame column if you will join ground truth later.

Payload logging (request / response collections) captures raw HTTP bodies without changing score.py. Those payloads are not guaranteed tabular. Model monitoring then needs a custom preprocessing component. Prefer DataFrame collectors for a seamless monitor.

MLflow deployments can turn collection on with a studio toggle; Azure Machine Learning instruments scoring for you and creates {endpoint}-{deployment}-model_inputs and ...-model_outputs assets.

For top N feature drift, the reference must be training data and you must name target_column. The monitor then ranks features and can limit drift or quality checks to top_n_feature_importance: 10 (or a named list such as SEX, EDUCATION, AGE). High-dimensional models should not monitor every column: Spark cost and alert noise both explode.

Feature attribution drift as an early warning

Feature attribution drift (preview) compares production feature importance to training using normalized discounted cumulative gain. It needs both collected inputs and outputs, and training data with a target column. When you use the data collector, the join key is correlationid unless you already logged a joined table. Use it with data drift: inputs can look stable while the drivers of the score rotate.

Exam scenario

A bank deploys a credit-default classifier to a managed online endpoint. Two months later a marketing campaign onboarded younger cardholders. Data scientists still have the original training mltable. Which monitor tells you the input mix moved, and which baseline should you pick?

Configure a data_drift signal on model_inputs, set reference_data to the training asset with data_context: training, and monitor the top N features (AGE, LIMIT_BAL, PAY_0). Jensen-Shannon or PSI on AGE will rise if the age histogram shifted. Prediction drift on the default flag may also rise, but that is an output symptom. Concept drift is not proven until you join later default labels and watch accuracy or precision on a model_performance signal.

Common trap

The trap is treating the monitor like an in-request validator, or conflating the three drift words. Data drift is not "the model got worse." Data drift is inputs. Prediction drift is outputs. Concept drift is the relationship, inferred, not a YAML type. A second trap is enabling only request/response payload logging and expecting out-of-box drift metrics — those files are not tabular. A third trap is overlapping lookback windows, or leaving the out-of-box recent production baseline in place forever, which hides a slow walk away from training.

Limitations to remember: monitoring uses Spark, so prefer simple tables over exotic MLTable transforms; AllowOnlyApprovedOutbound managed virtual networks are not supported; datastore access can be credential-based or a workspace user-assigned managed identity with systemDatastoresAuthMode set to identity.