4.1 Data Validation & Quality Frameworks

Data validation and quality frameworks are crucial for ensuring that downstream analytical applications, machine learning models, and executive dashboards consume reliable and accurate information. In the context of Databricks and the Databricks Certified Data Engineer Professional exam, data quality enforcement is heavily tested around declarative pipelines, programmatic data routing, and external validation frameworks. Building robust data engineering pipelines requires moving beyond simple error handling into comprehensive data quality architectures. As data flows through the Medallion Architecture, ensuring its integrity becomes paramount, as the cost of fixing corrupted data increases exponentially the further downstream it reaches.

Delta Live Tables (DLT) Expectations

Delta Live Tables (DLT) makes managing data quality natively easy by introducing 'expectations.' Expectations are constraints that you define for your datasets. When records flow through a DLT pipeline, they are evaluated against these expectations. Depending on the syntax used, DLT handles records failing these expectations in different ways. This declarative approach allows developers to encode business rules directly into the pipeline logic, rather than writing custom filtering transformations that can become difficult to maintain over time.

By integrating data quality directly into the pipeline graph, DLT ensures that data quality is not an afterthought but a first-class citizen of the data engineering lifecycle. The event log automatically captures the results of these expectations, providing built-in observability without requiring external monitoring tools.

The Three Expectation Types

1. @dlt.expect(name, constraint): This is the most lenient expectation. When a record violates the constraint, it is still processed and written to the target table. However, DLT records the failure in the event log metrics. This is extremely useful for monitoring data drift, soft validation rules, or non-critical anomalies without losing data. For instance, you might expect a discount_applied field to be less than 50%. If it exceeds 50%, you want to know about it, but you don't necessarily want to drop the transaction.

2. @dlt.expect_or_drop(name, constraint): When a record fails this expectation, DLT drops the row entirely and does not write it to the target table. The pipeline continues to run normally, processing the valid records. This is frequently used for strict data cleansing where invalid records are completely unusable and can be safely ignored. Examples include dropping records with a NULL primary key or removing sensor readings that fall completely outside physical possibilities (e.g., a temperature of absolute zero in a standard engine).

3. @dlt.expect_or_fail(name, constraint): This is the strictest expectation. If even a single record violates the condition, the entire pipeline execution fails and halts. No further data is processed. This is essential when data integrity is so critical that processing flawed data would corrupt the entire dataset or violate strict regulatory compliance. For instance, if a batch of financial transactions contains negative transfer amounts that should never occur, halting the pipeline prevents downstream financial dashboards from displaying incorrect revenue figures.

Code Example: DLT Expectations in Python

import dlt
from pyspark.sql.functions import expr

@dlt.table(name="valid_transactions")
@dlt.expect("valid_timestamp", "timestamp IS NOT NULL")
@dlt.expect_or_drop("positive_amount", "amount > 0")
@dlt.expect_or_fail("valid_account", "account_id IS NOT NULL AND length(account_id) = 10")
def get_valid_transactions():
    return dlt.read_stream("raw_transactions")

In this example, transactions with a null timestamp are kept but flagged, transactions with an amount less than or equal to zero are dropped, and if any transaction has an invalid account ID, the pipeline fails immediately.

The Quarantine Table Pattern

While @dlt.expect_or_drop is useful for discarding data, sometimes dropped records must be investigated, audited, or reprocessed. Dropping data silently can lead to compliance issues or lost business value. The Quarantine table pattern (sometimes referred to as a Dead Letter Queue or DLQ pattern in other contexts) addresses this requirement by splitting a single input stream into two distinct output streams: one for valid records (the target table) and one for invalid records (the quarantine table).

Instead of implicitly dropping data, the pipeline uses explicit transformations to route data. A common implementation involves creating a view or table that evaluates the quality rules and appends a boolean flag (e.g., is_valid) or an array of error messages. Downstream tables then filter based on these flags.

Implementation Example

@dlt.table(name="transactions_with_quality_flags")
def evaluate_transactions():
    df = dlt.read_stream("raw_transactions")
    # Add flags for various quality checks
    return df.withColumn("is_valid", expr("amount > 0 AND account_id IS NOT NULL")) \
             .withColumn("error_reason", expr(
                 "CASE WHEN amount <= 0 THEN 'Negative Amount' " +
                 "WHEN account_id IS NULL THEN 'Missing Account' ELSE NULL END"
             ))

@dlt.table(name="valid_transactions")
def get_valid():
    # Only forward valid records to the clean table
    return dlt.read_stream("transactions_with_quality_flags").filter("is_valid = true")

@dlt.table(name="quarantined_transactions")
def get_quarantined():
    # Route invalid records to the quarantine table for investigation
    return dlt.read_stream("transactions_with_quality_flags").filter("is_valid = false")

This pattern guarantees zero data loss while preventing bad data from entering the trusted "Silver" or "Gold" layers of the lakehouse. Data stewards can periodically query the quarantine table to identify upstream system bugs, correct the data, and reinject it into the pipeline.

Great Expectations on Databricks

For teams requiring complex data quality rules that span multiple tables, historical trend analysis, or statistical profiling, third-party frameworks like Great Expectations (GE) are often integrated into Databricks workflows. Great Expectations is an open-source Python library that allows you to define programmatic assertions about your data.

While DLT expectations are evaluated on a per-record basis as data streams through the pipeline, Great Expectations is typically executed in batch mode against an entire DataFrame or an existing Delta table. GE creates "Expectation Suites" which can include complex statistical bounds (e.g., expect_column_mean_to_be_between, expect_column_kl_divergence_to_be_less_than).

On Databricks, GE runs natively against PySpark DataFrames, leveraging the distributed compute of the Apache Spark cluster. The results of a GE validation run can be saved as JSON reports, visually rendered in HTML "Data Docs," or used to conditionally fail a Databricks Job task if the overall data quality score drops below a specific threshold. This makes GE incredibly powerful for post-load validation steps or certification of Gold layer datasets before they are published to business users.

Custom Data Quality Verification Rules

Beyond DLT and Great Expectations, data engineers often build custom data quality verification rules using standard PySpark DataFrame API methods. This might involve running aggregate queries to ensure row counts match between source and destination, or verifying that cross-table foreign key relationships remain intact after a large batch update.

When writing custom verification logic, engineers typically use assertions within a Databricks notebook. If an assertion fails, the notebook throws an exception, causing the encompassing Databricks Workflow task to fail. This is a common pattern for post-load validation steps in complex orchestration pipelines.

For instance, an engineer might write a custom script to calculate the total sum of account balances in a Silver table and compare it against a control file delivered by the upstream source system. If the values differ by more than a defined tolerance, the pipeline halts and sends an alert.

Querying DLT Event Logs for Quality Metrics

A crucial aspect of managing data validation in Databricks is observing the results over time. DLT captures all expectation metrics in its event log, which is itself stored as a Delta table. Data engineers can query this event log using Databricks SQL to create data quality dashboards.

By querying the flow_progress events in the DLT event log, engineers can extract the number of records that passed, failed, or were dropped for every expectation defined in the pipeline. This allows organizations to track data quality trends over time, identify specific days when data quality degrades, and calculate Data Quality Service Level Indicators (SLIs) for the enterprise.

Mastering these quality frameworks is absolutely essential for the Data Engineer Professional exam, as you must demonstrate the ability to architect resilient pipelines that handle imperfect data gracefully, ensure compliance, route data efficiently, and alert stakeholders to severe anomalies without writing overly complex manual logic.