PracticeVideosBlogFlashcardsEspañol

Free exam prep

AWS Certified Machine Learning Engineer — Associate (MLA-C01)

Practice QuestionsStudy Guide

11.1 Data Quality & Model Quality Monitoring

Key Takeaways

  • SageMaker Endpoint Data Capture logs real-time request payloads and response predictions to Amazon S3 in partitioned JSONLines format (/YYYY/MM/DD/HH/) with configurable sampling percentages and custom headers.
  • Data Quality Monitor evaluates captured live inference data against baseline constraints.json and statistics.json (generated using Amazon Deequ) to detect schema violations, missing values, and statistical feature drift.
  • Model Quality Monitor detects real-time model accuracy degradation (Precision, Recall, F1, ROC-AUC, RMSE) by merging captured inference predictions with delayed ground truth labels ingested into S3 using a common inferenceId.
  • Monitoring schedules execute periodic SageMaker Processing jobs (hourly/daily) that output constraint_violations.json and updated statistics.json to S3, publishing CloudWatch metrics and triggering automated alarms on quality violations.
Last updated: August 2026

Data Quality & Model Quality Monitoring

Deploying a machine learning model to a production endpoint is only the beginning of the operational lifecycle. Over time, real-world data distributions inevitably diverge from training data due to shifting consumer behaviors, seasonal trends, macroeconomic shifts, and upstream data pipeline modifications. This phenomenon—broadly known as model decay or concept drift—leads to silent degradation in model predictive accuracy.

Amazon SageMaker provides SageMaker Model Monitor, a fully managed continuous monitoring service that automatically detects data drift, schema violations, concept drift, and performance degradation on live production endpoints without requiring custom monitoring infrastructure.

For the AWS Certified Machine Learning Engineer — Associate (MLA-C01) exam, you must master the end-to-end monitoring lifecycle: enabling Data Capture, generating baseline statistical constraints (statistics.json and constraints.json), configuring Data Quality and Model Quality monitoring schedules, managing delayed ground truth label ingestion, and analyzing constraint_violations.json reports.

1. Endpoint Data Capture Architecture

Before SageMaker Model Monitor can analyze live traffic, the hosting endpoint must be configured to log incoming inference requests and outgoing model predictions. This is achieved via SageMaker Endpoint Data Capture.

+--------------------------------------------------------------------------------------------------+
|                         SAGEMAKER ENDPOINT DATA CAPTURE ARCHITECTURE                             |
|                                                                                                  |
|   [Client Application]                                                                           |
|            |                                                                                     |
|            | 1. HTTP POST InvokeEndpoint (Inference Payload + X-Amzn-SageMaker-Custom-Attributes)|
|            v                                                                                     |
|   +------------------------------------------------------------------------------------------+   |
|   | SageMaker Real-Time Endpoint                                                             |   |
|   |                                                                                          |   |
|   |   +----------------------------------------------------------------------------------+   |   |
|   |   | DataCaptureConfig Active: SamplingPercentage = 100%                              |   |   |
|   |   |                                                                                  |   |   |
|   |   |   [Model Container (predict_fn)]                                                 |   |   |
|   |   |         |                                                                        |   |   |
|   |   |         +---> Prediction Returned to Client (Synchronous <50ms)                  |   |   |
|   |   |         |                                                                        |   |   |
|   |   |         +---> Asynchronous Background Capture Engine                             |   |   |
|   +---|----------------------------------------------------------------------------------+---|   |
|   +---|--------------------------------------------------------------------------------------|   |
|       |                                                                                      |   |
|       v 2. Writes Partitioned JSONLines Records                                              v   |
|   +------------------------------------------------------------------------------------------+   |
|   | Amazon S3 Destination Bucket (s3://datacapture-bucket/endpoint-name/AllTraffic/)          |   |
|   |                                                                                          |   |
|   |   /YYYY/MM/DD/HH/                                                                        |   |
|   |     |-- 2026-08-16-01-00-00-abc123.jsonl                                                 |   |
|   |     |-- 2026-08-16-01-05-00-def456.jsonl                                                 |   |
|   +------------------------------------------------------------------------------------------+   |
+--------------------------------------------------------------------------------------------------+

1.1 Configuring DataCaptureConfig

Data capture is enabled at the EndpointConfig level. When defining or updating an endpoint configuration, you specify DataCaptureConfig with the following core parameters:

  • EnableCapture (Boolean): Set to True to activate payload logging.
  • SamplingPercentage (Integer): An integer from 1 to 100 defining the percentage of live requests to capture. For high-throughput endpoints (e.g., 5,000 RPS), capturing 10% to 20% reduces S3 storage costs while maintaining statistical power. For lower-volume critical endpoints, set to 100%.
  • DestinationS3Uri (String): The Amazon S3 location where captured JSONLines files are written.
  • CaptureOptions (List): Specifies what portion of the payload to log. You can capture Input (features submitted to the endpoint), Output (model predictions returned), or both.
  • CaptureContentTypeHeader (Dict): Specifies which MIME types to capture (e.g., text/csv, application/json).
  • KmsKeyId (Optional): AWS KMS customer managed key ARN used for server-side encryption of captured data in S3.

1.2 Python SDK Implementation: Enabling Data Capture

import sagemaker
from sagemaker.model_monitor import DataCaptureConfig

sagemaker_session = sagemaker.Session()
role = sagemaker.get_execution_role()

# Configure Data Capture for 100% of traffic, logging both inputs and outputs
data_capture_config = DataCaptureConfig(
    enable_capture=True,
    sampling_percentage=100,
    destination_s3_uri="s3://production-model-monitoring-bucket/fraud-endpoint/datacapture",
    capture_options=["REQUEST", "RESPONSE"],  # Captures both input features and model predictions
    csv_content_types=["text/csv"],
    json_content_types=["application/json"]
)

# Deploy PyTorch/XGBoost model with DataCaptureConfig attached
predictor = model.deploy(
    initial_instance_count=2,
    instance_type="ml.m5.xlarge",
    endpoint_name="fraud-detection-realtime",
    data_capture_config=data_capture_config
)

1.3 Captured S3 Data Structure & JSONLines Schema

Captured records are stored in S3 partitioned hierarchically by date and hour: s3://<bucket>/<prefix>/<endpoint-name>/<variant-name>/YYYY/MM/DD/HH/<timestamp>_<uuid>.jsonl

Each line in the file is a valid JSON object containing input data, output data, and execution metadata:

{
  "captureData": {
    "endpointInput": {
      "observedContentType": "text/csv",
      "mode": "INPUT",
      "data": "42.0,1500.50,3,1,0.85",
      "encoding": "CSV"
    },
    "endpointOutput": {
      "observedContentType": "text/csv",
      "mode": "OUTPUT",
      "data": "0.0341",
      "encoding": "CSV"
    }
  },
  "eventMetadata": {
    "eventId": "a1b2c3d4-e5f6-7a8b-9c0d-1e2f3a4b5c6d",
    "inferenceTime": "2026-08-16T01:14:22Z",
    "customHeaders": "X-Amzn-SageMaker-Custom-Attributes=CustomerID-98421"
  }
}

[!IMPORTANT] Inference ID Correlation: To correlate live inference records with delayed ground truth labels in Model Quality Monitoring, client applications should pass a unique identifier (such as transaction ID or customer ID) via the InferenceId parameter in the InvokeEndpoint API call or custom HTTP headers (X-Amzn-SageMaker-Custom-Attributes). SageMaker embeds this eventId/inferenceId in the captured JSONLines metadata.

2. SageMaker Data Quality Monitor

Data Quality Monitor evaluates the structural integrity and statistical distribution of incoming feature payloads against a baseline dataset (typically the clean training or validation dataset). It detects anomalies such as missing features, unexpected null values, data type mismatches, and numerical/categorical distribution drift.

+--------------------------------------------------------------------------------------------------+
|                              DATA QUALITY MONITORING WORKFLOW                                    |
|                                                                                                  |
|   [1. BASELINING PHASE]                                                                          |
|   Baseline Dataset (S3) ---> [DefaultModelMonitor.suggest_baseline()]                            |
|                                            |                                                     |
|                                            v (Managed Deequ Processing Job)                      |
|                       +--------------------+--------------------+                                |
|                       |                                         |                                |
|                       v                                         v                                |
|             [statistics.json]                           [constraints.json]                       |
|             (Mean, Min, Max, Quantiles)                 (Non-null, Data Types, Bounds)           |
|                                                                                                  |
|   [2. MONITORING PHASE]                                                                          |
|   Captured Inferences (S3) + constraints.json ---> [Scheduled Model Monitoring Job (Hourly)]     |
|                                                                   |                              |
|                                                                   v                              |
|                                                   [constraint_violations.json]                   |
|                                                                   |                              |
|                                      +----------------------------+----------------------------+ |
|                                      |                                                         | |
|                                      v                                                         v |
|                        [CloudWatch Alarms Triggered]                             [Amazon SNS Alert] |
+--------------------------------------------------------------------------------------------------+

2.1 Generating the Data Quality Baseline

To establish "normal" data behavior, SageMaker Model Monitor executes a managed baseline calculation job powered by Amazon Deequ (an open-source data quality verification library built on Apache Spark):

  1. Input: Baseline dataset stored in S3 (e.g., s3://bucket/data/train_baseline.csv without header lines, or with headers matching baseline config).
  2. Execution: DefaultModelMonitor.suggest_baseline() provisions an ephemeral processing instance.
  3. Generated Baseline Artifacts:
    • statistics.json: Comprehensive descriptive statistics for each feature (mean, standard deviation, min, max, median, quantiles, distinct count, null count).
    • constraints.json: Prescriptive rules defining valid data ranges, data types, completeness thresholds (e.g., completeness == 1.0 for non-nullable fields), and baseline drift distance metrics.
from sagemaker.model_monitor import DefaultModelMonitor
from sagemaker.model_monitor.dataset_format import DatasetFormat

my_monitor = DefaultModelMonitor(
    role=role,
    instance_count=1,
    instance_type='ml.m5.xlarge',
    volume_size_in_gb=20,
    max_runtime_in_seconds=3600
)

# Run baseline job against training data
my_monitor.suggest_baseline(
    baseline_dataset='s3://production-model-monitoring-bucket/baselines/train_data.csv',
    dataset_format=DatasetFormat.csv(header=True),
    output_s3_uri='s3://production-model-monitoring-bucket/baselines/data_quality_output',
    wait=True
)

2.2 Deep Dive: Baseline Artifacts

constraints.json Structure:

{
  "version": 0,
  "schema": {
    "columns": [
      {"name": "account_age", "type": "Integral"},
      {"name": "transaction_amount", "type": "Fractional"},
      {"name": "device_type", "type": "String"}
    ]
  },
  "monitoring_constraints": {
    "columns": {
      "account_age": {
        "completeness": 1.0,
        "data_type": "Integral",
        "distribution": {
          "min": 0,
          "max": 120
        }
      },
      "transaction_amount": {
        "completeness": 1.0,
        "data_type": "Fractional",
        "distribution": {
          "min": 0.50,
          "max": 50000.00
        }
      }
    }
  }
}

2.3 Creating the Monitoring Schedule

Once baselines are calculated, you schedule periodic evaluation jobs using create_monitoring_schedule(). The schedule runs at a specified cadence (e.g., hourly via cron(0 * ? * * *) or CronExpressionGenerator.hourly()):

from sagemaker.model_monitor import CronExpressionGenerator

# Schedule hourly data quality monitoring
my_monitor.create_monitoring_schedule(
    monitor_schedule_name="fraud-endpoint-data-quality-schedule",
    endpoint_input="fraud-detection-realtime",
    output_s3_uri="s3://production-model-monitoring-bucket/reports/data_quality",
    statistics=my_monitor.baseline_statistics(),
    constraints=my_monitor.suggested_constraints(),
    schedule_cron_expression=CronExpressionGenerator.hourly(),
    enable_cloudwatch_metrics=True
)

2.4 Violation Reports: constraint_violations.json

When live inference data deviates from constraints.json, Model Monitor generates a constraint_violations.json file in S3 and emits violation metrics to Amazon CloudWatch:

{
  "version": "0.0",
  "violations": [
    {
      "feature_name": "transaction_amount",
      "constraint_check_type": "baseline_drift_check",
      "description": "Baseline drift distance: 0.28 exceeded threshold: 0.10"
    },
    {
      "feature_name": "account_age",
      "constraint_check_type": "completeness_check",
      "description": "Feature completeness 0.84 failed constraint: completeness >= 1.0"
    },
    {
      "feature_name": "device_ip",
      "constraint_check_type": "extra_column_check",
      "description": "Unexpected column device_ip present in live inference"
    }
  ]
}
Violation Check TypeCause of ViolationProduction Example
completeness_checkProportion of non-null values fell below baseline threshold.Mobile app update started sending null for zip_code.
data_type_checkData type of live feature does not match baseline schema.Upstream API sent string "100" instead of numeric integer 100.
baseline_drift_checkStatistical distance (L-Infinity or Kolmogorov-Smirnov) exceeded threshold.Sudden spike in high-value transactions during Black Friday sales.
extra_column_checkClient sent columns not present in constraints.json.Upstream service added a new metadata field to request body.
missing_column_checkA required baseline column was missing from live payload.Microservice bug dropped the credit_score feature before calling endpoint.

3. SageMaker Model Quality Monitor

While Data Quality Monitor inspects inputs, Model Quality Monitor evaluates the predictive performance of the model (accuracy, precision, recall, F1-score, ROC-AUC, RMSE, MAE) on live traffic.

+--------------------------------------------------------------------------------------------------+
|                             MODEL QUALITY MONITORING WORKFLOW                                    |
|                                                                                                  |
|   [Endpoint Invocations]                            [Asynchronous Ground Truth Collection]        |
|             |                                                         |                          |
|             v                                                         v                          |
|   [Data Capture S3 Bucket]                          [Ground Truth S3 Bucket]                     |
|   - Records: {eventId: 101, prediction: 1}          - Records: {eventId: 101, groundTruth: 0}   |
|             |                                                         |                          |
|             +----------------------------+----------------------------+                          |
|                                          |                                                       |
|                                          v                                                       |
|                      [Scheduled ModelQualityMonitor Processing Job]                              |
|                      - Merges records by matching eventId / inferenceId                          |
|                      - Computes live confusion matrix & evaluation metrics                       |
|                      - Compares against baseline quality thresholds                              |
|                                          |                                                       |
|                                          v                                                       |
|                      [Quality Report & constraint_violations.json]                               |
|                      (e.g., Live F1-Score: 0.71 < Baseline Constraint: 0.85)                     |
|                                          |                                                       |
|                                          v                                                       |
|                      [CloudWatch Alarm & Automated Retraining Trigger]                           |
+--------------------------------------------------------------------------------------------------+

3.1 The Ground Truth Ingestion Challenge

In online production systems, the actual outcome (ground truth label) is rarely available at the exact moment of inference. For example:

  • In fraud detection, a transaction may only be confirmed as fraudulent days later after a chargeback.
  • In loan default prediction, default status is determined months or years after approval.
  • In e-commerce recommendations, click-through or purchase confirmation arrives seconds to minutes later.

To solve this, SageMaker Model Quality Monitor ingests ground truth labels asynchronously from S3 and merges them with captured predictions using a matching eventId or inferenceId.

3.2 Ground Truth S3 File Format

Ground truth labels must be uploaded to S3 as JSONLines formatted files:

{"groundTruthData": {"data": "1", "encoding": "CSV"}, "eventMetadata": {"eventId": "a1b2c3d4-e5f6-7a8b-9c0d-1e2f3a4b5c6d", "inferenceTime": "2026-08-16T01:14:22Z"}}
{"groundTruthData": {"data": "0", "encoding": "CSV"}, "eventMetadata": {"eventId": "b2c3d4e5-f6a7-8b9c-0d1e-2f3a4b5c6d7e", "inferenceTime": "2026-08-16T01:15:10Z"}}

3.3 Configuring ModelQualityMonitor in Python SDK

from sagemaker.model_monitor import ModelQualityMonitor
from sagemaker.model_monitor import EndpointInput

model_quality_monitor = ModelQualityMonitor(
    role=role,
    instance_count=1,
    instance_type='ml.m5.xlarge',
    volume_size_in_gb=20,
    max_runtime_in_seconds=3600
)

# 1. Generate Model Quality Baseline using validation dataset
model_quality_monitor.suggest_baseline(
    baseline_dataset='s3://production-model-monitoring-bucket/baselines/validation_predictions.csv',
    dataset_format=DatasetFormat.csv(header=True),
    output_s3_uri='s3://production-model-monitoring-bucket/baselines/model_quality_output',
    problem_type='BinaryClassification',
    ground_truth_attribute='label',
    inference_attribute='prediction_prob',
    probability_threshold_attribute=0.5
)

# 2. Schedule Model Quality Monitoring with Ground Truth Merging
model_quality_monitor.create_monitoring_schedule(
    monitor_schedule_name="fraud-endpoint-model-quality-schedule",
    endpoint_input=EndpointInput(
        endpoint_name="fraud-detection-realtime",
        destination="/opt/ml/processing/input/endpoint",
        start_time_offset="-PT6H",  # Looks back 6 hours for captured inferences
        end_time_offset="-PT1H"
    ),
    ground_truth_input="s3://production-model-monitoring-bucket/ground-truth/",
    output_s3_uri="s3://production-model-monitoring-bucket/reports/model_quality",
    problem_type='BinaryClassification',
    constraints=model_quality_monitor.suggested_constraints(),
    schedule_cron_expression=CronExpressionGenerator.hourly(),
    enable_cloudwatch_metrics=True
)

3.4 Model Quality Metrics by Problem Type

ML Problem TypeComputed MetricsMonitored Drift / Constraints
Binary ClassificationAccuracy, Precision, Recall, F1-Score, F0.5, F2, ROC-AUC, PR-AUC, False Positive Rate (FPR), Confusion Matrix.Drop in F1-score below baseline threshold (e.g., F1 < 0.85); ROC-AUC degradation.
Multiclass ClassificationMicro/Macro Precision, Micro/Macro Recall, Micro/Macro F1-Score, Accuracy, Multiclass Confusion Matrix.Macro F1-score drop across minority classes.
RegressionRoot Mean Square Error (RMSE), Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), R-squared score.Increase in RMSE above baseline threshold (e.g., RMSE > 15.2).

4. Comparing Data Quality vs. Model Quality Monitoring

Architectural DimensionData Quality MonitorModel Quality Monitor
Primary ObjectiveDetects input schema violations, missing data, and feature distribution drift.Detects degradation in model accuracy and predictive power.
Required Ingestion DataCaptured endpoint request/response payloads in S3.Captured endpoint predictions in S3 plus delayed ground truth labels.
Execution EngineAmazon Deequ (Spark data quality validation).SageMaker Model Quality Evaluation Container.
Key Baseline Artifactsstatistics.json, constraints.json (feature bounds).statistics.json, constraints.json (metric thresholds like min F1, max RMSE).
Ground Truth Needed?No (Operates entirely unsupervised on input features).Yes (Requires matching inferenceId with actual labels).
Typical Alert Triggersbaseline_drift_check, completeness_check, extra_column_check.f1_score < baseline_f1, rmse > baseline_rmse.
Loading diagram...
SageMaker Model Monitor End-to-End Operational Lifecycle
Test Your Knowledge

An ML engineer is deploying a fraud detection model to a SageMaker real-time endpoint that handles 4,000 transactions per second. The engineer needs to monitor the live endpoint for input feature schema anomalies and statistical drift while keeping S3 storage costs optimized. The downstream monitoring process must not introduce any latency overhead to the synchronous inference calls. Which configuration meets these requirements?

A
B
C
D
Test Your Knowledge

A financial services company notices that a production credit risk model's live F1-score has degraded from 0.88 to 0.72 over three months. Ground truth loan default outcomes arrive with a 30-day delay. The ML team wants to automatically compute live model evaluation metrics and receive alerts whenever the F1-score falls below 0.80. Which solution should the ML engineer implement?

A
B
C
D
Test Your Knowledge

An ML engineer runs suggest_baseline() for a Data Quality Monitor on a customer churn prediction model. Which two files are generated by this baseline job in Amazon S3?

A
B
C
D
Test Your Knowledge

During an hourly Data Quality monitoring schedule execution, SageMaker Model Monitor emits a violation in constraint_violations.json stating: Feature completeness 0.78 failed constraint: completeness >= 1.0. What is the root cause of this violation?

A
B
C
D
Up Next

11.2 Model Bias & Model Explainability Drift

Continue learning