9.3 Implementing Data Quality Frameworks & Automated Validation Checks

Implementing Data Quality Frameworks & Automated Validation Checks

Maintaining high data fidelity across lakehouse environments requires embedding continuous data quality checks directly into automated data pipelines. Silent data corruption, schema mutations, and null-value spikes degrade downstream BI dashboards and machine learning models. This section details native and open-source data quality frameworks, quarantine architecture patterns, and automated pipeline alerting mechanisms on AWS.

---

AWS Glue Data Quality & Data Quality Definition Language (DQDL)

AWS Glue Data Quality provides managed, serverless automated data validation embedded into AWS Glue ETL jobs and the AWS Glue Data Catalog. It evaluates datasets against rules defined in Data Quality Definition Language (DQDL).

1. DQDL Syntax & Core Rule Categories

DQDL rulesets are declarative strings defining validation assertions over columns, row counts, and statistical metrics:

Rules = [
    Completeness "customer_id" = 1.0,
    Uniqueness "customer_id" = 1.0,
    Completeness "email" >= 0.98,
    ColumnValues "account_status" in ["ACTIVE", "SUSPENDED", "CLOSED"],
    ColumnLength "zip_code" = 5,
    RowCount > 10000,
    Mean "transaction_amount" between 10.0 and 500.0
]

• Completeness: Measures the proportion of non-null values in a column.
• Uniqueness: Evaluates duplicate ratios across one or more columns.
• ColumnValues: Validates column values against permitted domain enumerations or regex expressions (ColumnValues "phone" matches "^\\d{3}-\\d{3}-\\d{4}$").
• Mean / StandardDeviation / Minimum / Maximum: Assert aggregate properties of a numeric column to detect drift or implausible ranges.

2. AWS Glue Job Integration (EvaluateDataQuality)

In PySpark Glue jobs, the EvaluateDataQuality transform executes rulesets directly against DynamicFrames:

from awsgluedq.transforms import EvaluateDataQuality

# Define DQDL Ruleset
dq_ruleset = """
Rules = [
    Completeness "customer_id" = 1.0,
    Uniqueness "customer_id" = 1.0
]
"""

# Execute Data Quality Evaluation
dq_results = EvaluateDataQuality().process_rows(
    frame=input_dynamic_frame,
    ruleset=dq_ruleset,
    publishing_options={
        "dataQualityEvaluationContext": "CustomerIngestionJob",
        "enableDataQualityCloudWatchMetrics": True,
        "enableDataQualityResultsPublishing": True,
        "resultsS3Prefix": "s3://my-data-lake-dq-results/"
    },
    additional_options={"performanceTuning.caching": "CACHE_NOTHING"}
)

• Outcome Actions: Publishing options can send metrics to CloudWatch and detailed results to S3. To enforce a gate, configure the Glue Studio Data Quality outcome action or inspect ruleOutcomes in the script and explicitly stop or branch the workflow; fail_job_if_rules_fail is not an EvaluateDataQuality parameter.

---

Open-Source Frameworks: Apache Deequ & Great Expectations

For complex data quality requirements on Amazon EMR or AWS MWAA (Airflow), open-source quality frameworks offer advanced statistical checks and custom metric repositories.

1. Apache Deequ (Spark-Native Validation)

Apache Deequ is an open-source library built on Apache Spark by AWS to measure data quality at scale.

• Key Features:
  • Unit Tests for Data: Define checks programmatically in Scala or Python (PyDeequ):

    from pydeequ.checks import Check, CheckLevel
    from pydeequ.verification import VerificationSuite
    
    check = Check(spark, CheckLevel.Error, "Integrity Check")
    checkResult = VerificationSuite(spark) \
        .onData(df) \
        .addCheck(
            check.hasCompleteness("customer_id", lambda c: c == 1.0) \
                 .isUnique("customer_id") \
                 .hasPattern("email", r"^[\w-\.]+@([\w-]+\.)+[\w-]{2,4}$")
        ) \
        .run()
    

  • Metrics Repository: Deequ persists historical metrics (e.g., daily row counts, mean transaction values) to Amazon DynamoDB or Amazon S3, enabling automated anomaly detection based on statistical data drift.

2. Great Expectations

Great Expectations is a Python-based data quality library widely deployed on EMR or managed by AWS MWAA.

• Key Concepts:
  • Expectation Suites: JSON declarative rules defining expected data states (expect_column_values_to_be_unique).
  • Data Docs: Automatically renders human-readable HTML data quality documentation and validation reports, hosted directly on static Amazon S3 website buckets.

---

Data Quarantine & Conditional Routing Pattern

Failing an entire batch ETL job due to a small percentage of malformed records can violate business SLAs. The Data Quarantine Design Pattern routes valid records downstream while segregating bad records for auditing.

# Split DataFrame into Valid and Quarantined datasets
from pyspark.sql.functions import coalesce, col, lit

valid_condition = (
    col("customer_id").isNotNull()
    & coalesce(col("account_status").isin(["ACTIVE", "SUSPENDED", "CLOSED"]), lit(False))
)
valid_df = input_df.filter(valid_condition)
quarantine_df = input_df.filter(~valid_condition)

# Write Valid Records to Lakehouse Gold Layer
valid_df.write.format("parquet").mode("append").save("s3://lakehouse-gold/customers/")

# Enriched Quarantined Records with Failure Metadata
from pyspark.sql.functions import current_timestamp

quarantine_enriched = quarantine_df \
    .withColumn("quarantine_timestamp", current_timestamp()) \
    .withColumn("failure_reason", lit("NULL_CUSTOMER_ID_OR_INVALID_STATUS"))

# Write Quarantined Records to S3 Quarantine Storage
quarantine_enriched.write.format("parquet").mode("append").save("s3://lakehouse-quarantine/customers/")

• Dead Letter Queues (DLQ): For streaming workloads (Kinesis / MSK), bad records are routed to an Amazon SQS DLQ or an S3 quarantine prefix, triggering an Amazon SNS alert for data stewardship review.

---

Pipeline Governance, Orchestration & Alerting

To enforce enterprise data governance, data quality checks must be integrated into orchestration workflows using AWS Step Functions and Amazon EventBridge.

1. Step Functions Circuit Breaker Pattern

1. Execute Validation Task: Step Functions invokes a Glue Data Quality task.
2. Evaluate Outcome (Choice State):
   • If QualityScore >= 95%: Transition to the Load to Redshift state.
   • If QualityScore < 95%: Transition to the Circuit Breaker state.
3. Circuit Breaker State: Halts pipeline execution, executes a rollback transaction, and dispatches a critical SNS alert to Slack or PagerDuty.

2. CloudWatch & EventBridge Governance

When CloudWatch publishing is enabled, AWS Glue Data Quality publishes evaluation metrics under the configured data-quality evaluation context. Build alarms from the actual emitted dimensions and metrics for that run, and route alarm state changes through EventBridge or SNS for incident response.

---

Data Quality Framework Comparison Matrix

Reconciliation and data contracts

Quality is not only a row-level rule. Reconcile source and target counts, control totals, and business aggregates for every load. Persist the ruleset version, run identifier, source watermark, result, and quarantine location so an operator can reproduce a failure. A producer data contract should define field meaning, type, nullability, allowed evolution, and ownership; a schema-valid record can still violate a business invariant. For streaming pipelines, emit quality metrics by event time and isolate malformed records without silently advancing a checkpoint that makes recovery impossible.