4.1 Data Validation, Schema Drift, and Quality Rules

Data Validation, Schema Drift, and Quality Rules

In enterprise machine learning systems, data quality directly dictates model performance. While software engineering pipelines fail loudly when encountering syntax or network errors, machine learning systems often fail silently: corrupted, missing, or drifted data will still flow into training jobs and inference endpoints, producing degraded models and unreliable predictions without throwing runtime exceptions. This failure mode is captured by the foundational ML principle: Garbage In, Garbage Out.

For the AWS Certified Machine Learning Engineer — Associate (MLA-C01) examination, you must master automated data quality validation, declarative constraint authoring using AWS Glue Data Quality and Data Quality Definition Language (DQDL), Spark-based data unit testing with PyDeequ, automated quarantine routing, and metric alerting via Amazon CloudWatch.

+---------------------------------------------------------------------------------------------------+
|                         AWS DATA QUALITY & VALIDATION ARCHITECTURE                                |
|                                                                                                   |
|   [Raw S3 Data] ---> [AWS Glue ETL / PySpark]                                                     |
|                             |                                                                     |
|                             +---> [DQDL / PyDeequ Evaluation Engine]                              |
|                                          |                                                        |
|                     +--------------------+--------------------+                                   |
|                     | Score >= Threshold                      | Score < Threshold                 |
|                     v                                         v                                   |
|             [Curated S3 Bucket]                     [Quarantine S3 Bucket]                        |
|                     |                                         |                                   |
|                     v                                         v                                   |
|         [SageMaker Training Job]                    [CloudWatch Metric & Alarm]                   |
|                                                               |                                   |
|                                                               v                                   |
|                                                     [EventBridge Alert / Halt]                    |
+---------------------------------------------------------------------------------------------------+

---

1. The Critical Role of Data Quality in Production ML

Data quality degradation manifests in multiple forms across the ML lifecycle:

1. Silent Data Degradation: Upstream source schema updates (e.g., changing a currency format or renaming a column) do not crash ingestion jobs if null values are injected by default, but cause models to make erratic inferences.
2. Schema Drift: Unannounced modifications to incoming data contracts, including newly introduced categorical values, altered column types, or dropped attributes.
3. Distribution Drift: Statistical properties of features change over time (e.g., mean income shifts due to inflation or geographic expansion), eroding the validity of previously trained parameters.
4. Target & Feature Leakage: Inadvertently including post-event indicators or corrupted ground truth labels in feature sets.

To prevent silent corruption, ML engineers implement automated quality gates that evaluate datasets against deterministic constraints prior to feature store ingestion or model training.

---

2. AWS Glue Data Quality & DQDL

AWS Glue Data Quality is a fully managed, serverless capability that measures and monitors data quality for data stored in Amazon S3, AWS Glue Data Catalog tables, and within active AWS Glue ETL PySpark pipelines.

Glue Data Quality uses Data Quality Definition Language (DQDL), a declarative domain-specific language used to express data quality rules in a human-readable format.

+---------------------------------------------------------------------------------------------------+
|                                 DQDL RULE TYPES & SYNTAX                                          |
|                                                                                                   |
|   Completeness:     IsComplete 'customer_id'                                                      |
|   Uniqueness:       IsUnique 'transaction_uuid'                                                   |
|   Range / Bounds:   ColumnValues 'customer_age' between 18 and 120                                |
|   String Length:    ColumnLength 'country_iso_code' == 2                                          |
|   Set Membership:   ColumnValues 'subscription_tier' in ['FREE', 'BASIC', 'PREMIUM']              |
|   Cardinality:      DistinctValuesCount 'device_type' <= 10                                       |
|   Data Type:        ColumnDataType 'order_timestamp' = 'TIMESTAMP'                                |
|   Volume Gate:      RowCount > 10000                                                              |
|   Custom SQL:       CustomSql 'SELECT count(*) FROM primary WHERE transaction_amt < 0' = 0       |
+---------------------------------------------------------------------------------------------------+

Core DQDL Rule Categories

Automatic Recommendation Engine

AWS Glue Data Quality includes a built-in recommendation engine. By analyzing historical dataset profiles, Glue automatically generates a starter DQDL ruleset based on observed distributions, null rates, and column types. ML engineers can review, customize, and version-control these generated DQDL rulesets.

Evaluating DQDL within AWS Glue ETL Jobs

In PySpark Glue jobs, DQDL rules are evaluated dynamically using the EvaluateDataQuality transform:

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglueml.transforms import EvaluateDataQuality

sc = SparkContext()
glueContext = GlueContext(sc)
job = Job(glueContext)

# 1. Load source dynamic frame
source_dyf = glueContext.create_dynamic_frame.from_catalog(
    database='ml_lake',
    table_name='raw_sensor_telemetry'
)

# 2. Define DQDL ruleset
dqdl_rules = '''
Rules = [
    RowCount > 1000,
    IsComplete 'sensor_id',
    IsComplete 'timestamp',
    ColumnValues 'temperature_celsius' between -50.0 and 150.0,
    ColumnValues 'vibration_hz' >= 0.0,
    CustomSql 'SELECT count(*) FROM primary WHERE sensor_id IS NULL OR timestamp IS NULL' = 0
]
'''

# 3. Evaluate Data Quality with publishing enabled
eval_results = EvaluateDataQuality.apply(
    frame=source_dyf,
    ruleset=dqdl_rules,
    publishing_options={
        'dataQualityEvaluationContext': 'SensorETLJob',
        'enableDataQualityCloudWatchMetrics': True,
        'enableDataQualityResultsS3Publishing': True,
        'dataQualityResultsS3Path': 's3://ml-lake-governance/data-quality-results/'
    }
)

job.commit()

[!NOTE] Data Quality Score (0–100%): AWS Glue Data Quality aggregates rule evaluations into an overall quality score. If a ruleset contains 10 rules and 9 pass, the resulting Data Quality Score is 90%. You can enforce threshold gates (e.g., fail the job if Score < 95%).

---

3. PyDeequ & DeeQU on AWS

Deequ is an open-source library built on top of Apache Spark by AWS to define 'unit tests for data'. PyDeequ provides a native Python API for PySpark environments, making it a standard tool for ML engineers running data validation on Amazon EMR, AWS Glue, or Amazon SageMaker Processing jobs.

+---------------------------------------------------------------------------------------------------+
|                                 PYDEEQU CORE MODULE ARCHITECTURE                                  |
|                                                                                                   |
|   [PyDeequ Core Modules]                                                                          |
|   1. Suggestion:   Suggests constraints automatically based on statistical profiling.             |
|   2. Verification: Runs test suites against Spark DataFrames; returns PASS/FAIL states.          |
|   3. Metrics Repo: Persists computed metrics over time in S3 / DynamoDB to detect drift.          |
|   4. Anomaly:      Applies statistical models to metrics history to flag anomalous shifts.        |
+---------------------------------------------------------------------------------------------------+

Core PyDeequ Capabilities

1. Verification Suite

The VerificationSuite executes declarative assertions against Spark DataFrames and returns a structured verification report:

from pydeequ.checks import Check, CheckLevel, ConstrainableDataTypes
from pydeequ.verification import VerificationSuite, VerificationResult

# Construct declarative verification check
check = Check(spark, CheckLevel.Error, 'ML Training Dataset Verification')

check_suite = (
    check.hasSize(lambda sz: sz >= 5000)
    .isComplete('customer_id')
    .isUnique('customer_id')
    .isContainedIn('subscription_status', ['active', 'churned', 'suspended'])
    .hasMin('credit_score', lambda val: val >= 300)
    .hasMax('credit_score', lambda val: val <= 850)
    .hasCompleteness('annual_income', lambda c: c >= 0.98) # Allow up to 2% missing
    .hasCorrelation('monthly_spend', 'credit_limit', lambda corr: corr > 0.3)
)

# Run suite
verification_result = VerificationSuite(spark) \
    .onData(df) \
    .addCheck(check_suite) \
    .run()

# Inspect verification results
result_df = VerificationResult.checkResultsAsDataFrame(spark, verification_result)
result_df.show(truncate=False)

2. Constraint Suggestion

When onboarding a new dataset without predefined constraints, PyDeequ analyzes the DataFrame and suggests suitable rules:

from pydeequ.suggestions import ConstraintSuggestionRunner, DEFAULT

suggestion_result = ConstraintSuggestionRunner(spark) \
    .onData(raw_spark_df) \
    .addConstraintRule(DEFAULT()) \
    .run()

for suggestion in suggestion_result['constraint_suggestions']:
    print(f"{suggestion['column_name']}: {suggestion['description']}")

3. Metrics Repository & Anomaly Detection

PyDeequ can store computed metrics in a Metrics Repository (persisted to an S3 bucket in JSON format). Successive pipeline runs compare current metrics against historical distributions to detect anomalies (e.g., sudden jumps in missing value proportions or feature variance changes):

from pydeequ.repository import FileSystemMetricsRepository, ResultKey

metrics_file = 's3://ml-lake-governance/deequ-metrics-repository.json'
repository = FileSystemMetricsRepository(spark, metrics_file)
result_key = ResultKey(spark, ResultKey.current_milli_time(), {'run_id': 'daily_etl_20260816'})

VerificationSuite(spark) \
    .onData(df) \
    .addCheck(check_suite) \
    .useRepository(repository) \
    .saveOrAppendResult(result_key) \
    .run()

---

4. Automated Quality Gates, CI/CD & Quarantine Routing

Production ML architectures require automated mechanisms to handle records that violate quality constraints without crashing the entire business pipeline.

+---------------------------------------------------------------------------------------------------+
|                              DATA ROUTING & QUARANTINE PATTERN                                    |
|                                                                                                   |
|   [Raw Input Batch] ---> [Data Quality Validation Rule Split]                                     |
|                                  |                                                                |
|                 +----------------+----------------+                                               |
|                 | Passed Records                  | Failed Records                                |
|                 v                                 v                                               |
|      [Curated Data (S3)]                 [Quarantine Buffer (S3)]                                 |
|                 |                                 |                                               |
|                 v                                 v                                               |
|      [SageMaker Pipeline Step]           [Glue Error Table & SNS Alert]                           |
+---------------------------------------------------------------------------------------------------+

The Quarantine Pattern

1. Record-Level Splitting: Incoming dynamic frames are split based on rule outcomes. Records passing all validation checks flow to the Curated S3 Prefix (s3://lake/curated/year=YYYY/month=MM/).
2. Quarantine Storage: Records failing any check are routed to an Isolated Quarantine Prefix (s3://lake/quarantine/year=YYYY/month=MM/) tagged with the failed rule identifier, timestamp, and source job ID.
3. Conditional Job Execution:
   • If the percentage of quarantined records exceeds an operational tolerance (e.g., > 3%), the ETL job raises an exception and triggers an alert.
   • Downstream SageMaker Pipelines utilize ConditionStep to verify the Data Quality status before launching model training.

# PySpark dynamic routing pattern
clean_df = df.filter('customer_id IS NOT NULL AND customer_age >= 18 AND customer_age <= 120')
quarantine_df = df.filter('customer_id IS NULL OR customer_age < 18 OR customer_age > 120')

# Write clean data for ML ingestion
clean_df.write.mode('append').parquet('s3://ml-feature-store-lake/curated/customers/')

# Route corrupted data for inspection
quarantine_df.write.mode('append').json('s3://ml-feature-store-lake/quarantine/customers/')

---

5. CloudWatch Observability & Metric Alarms

When AWS Glue Data Quality executes with CloudWatch publishing enabled, it emits a pair of metrics under the custom CloudWatch namespace titled Glue Data Quality:

+---------------------------------------------------------------------------------------------------+
|                         CLOUDWATCH METRIC OBSERVABILITY MATRIX                                    |
|                                                                                                   |
|   Metric Name                        Namespace           Significance                          |
|   --------------------------------   -----------------   -------------------------------------   |
|   glue.data.quality.rules.passed     Glue Data Quality   Count of rules passed per evaluation   |
|   glue.data.quality.rules.failed     Glue Data Quality   Count of rules failed per evaluation   |
+---------------------------------------------------------------------------------------------------+

Incident Response Architecture

• CloudWatch Alarm: Configured to trigger when glue.data.quality.rules.failed > 0 over a single evaluation period (any failed rule). The per-run Data Quality Score is available in the evaluation results emitted to S3/CloudWatch for threshold-based dashboards.
• Amazon EventBridge: Captures the Glue Data Quality Rule Evaluation Failed state change event.
• Remediation Actions: EventBridge invokes an AWS Lambda function that disables the downstream SageMaker Training pipeline, sends a high-priority Amazon SNS notification to the MLOps on-call team, and creates an incident ticket.

---

6. Service Comparison: Data Quality on AWS