6.4 Data Quality & Pipeline Alerting

6.4 Data Quality & Pipeline Alerting

In modern data engineering, delivering data on time is only half the battle; the data must also be accurate, trustworthy, and compliant with business rules. Databricks Delta Live Tables (DLT) provides built-in declarative data quality through Expectations. However, defining expectations is just the first step. To maintain a healthy lakehouse, you must actively monitor these data quality metrics, build automated dashboards, and set up alerts for when data drifts or quality thresholds are breached. This section covers how to unlock and utilize the DLT Event Log for advanced pipeline observability, enabling a robust DataOps culture.

The DLT Event Log

Every Delta Live Tables pipeline automatically generates an event log. This log is not just a dump of error messages; it records every action, state change, and data quality metric generated during the pipeline's execution. Crucially, the event log itself is stored as a standard Delta table, managed entirely by the Databricks platform.

Because it is a Delta table, you can query it using standard SQL in Databricks SQL, a notebook, or even through external BI tools. The event log is the foundational dataset for all DLT observability, acting as a unified catalog of the operational state and data fidelity of your pipelines.

Querying the Event Log

To query the event log, you use the built-in event_log table-valued function, passing the ID of your pipeline as a string argument. You can easily find the Pipeline ID in the DLT UI settings pane.

-- Retrieve the most recent 100 events from a specific pipeline
SELECT
  timestamp,
  event_type,
  level,
  message
FROM event_log('a1b2c3d4-e5f6-7890-abcd-1234567890ab')
ORDER BY timestamp DESC
LIMIT 100;

The event_type column is critical for filtering. Common event types include flow_progress (which contains data quality metrics and throughput statistics), user_action (such as starting or stopping the pipeline), and plan_update (recording changes to the DAG structure). Filtering by event type allows you to isolate exactly the telemetry you need.

Extracting Expectation Metrics

When you define an expectation in DLT (e.g., EXPECT (age > 0) OR DROP), DLT evaluates every record against this rule during the data processing phase. The results of these evaluations are logged in the event log under the flow_progress event type. This provides a historical audit trail of data quality.

The detailed metrics are stored as a complex JSON string within the details column. To analyze data quality effectively, you must parse this JSON payload and extract quantitative fields like passed_records and failed_records.

SQL Query for Data Quality Metrics

Here is how you parse the JSON to extract the pass/fail rates for specific datasets and expectations. Databricks SQL provides robust JSON parsing functions that make this straightforward:

-- Extract data quality metrics from the event log
SELECT
  timestamp,
  details:flow_progress:data_quality:dataset_name AS dataset,
  explode(from_json(details:flow_progress:data_quality:expectations,
    'array<struct<name: string, dataset: string, passed_records: int, failed_records: int>>'
  )) AS expectation_metrics
FROM event_log('your-pipeline-id')
WHERE event_type = 'flow_progress'
  AND details:flow_progress:data_quality IS NOT NULL;

By expanding the array of expectations using the explode function, you get a distinct row for every rule evaluated. This row contains the exact count of records that passed and failed during that specific micro-batch or pipeline run, providing granular insights into exactly which rules are failing and how often.

Handling Failed Records

Recall how DLT handles expectation failures based on your declaration. Understanding this behavior is critical for interpreting the metrics:

• EXPECT (Warn): The record fails the rule but is still written to the target table. Metrics are logged. This is useful for observing data drift without impacting downstream consumers immediately.
• EXPECT OR DROP: The record fails and is dropped (excluded) from the target table. The pipeline successfully continues processing the remaining good records. This ensures that downstream dashboards never see corrupted data.
• EXPECT OR FAIL: The entire pipeline immediately halts execution and throws an exception if a single record fails. This is a severe safeguard used for mission-critical assertions where bad data is worse than no data.

By querying the event log, you can track exactly how many records are being silently dropped by an EXPECT OR DROP rule over time. If a rule typically drops 5 records a day but suddenly drops 5,000, that is a critical incident that requires investigation.

Building Automated Data Quality Dashboards

Once you have the SQL queries extracting passed_records and failed_records, you can build automated dashboards directly within Databricks SQL. These dashboards serve as the command center for data quality.

By calculating the failure rate using the formula (failed_records / (passed_records + failed_records)) * 100, you can create time-series line charts tracking data quality over time. If the failure rate for an expectation checking user_email_format suddenly spikes from 0.1% to 15%, the dashboard will immediately visualize this data drift. This visual indication often points to an upstream schema change by a vendor or a new bug introduced in the source system software.

Triggering Alert Notifications

Visual dashboards are excellent for reporting, but they require human eyes to be actively watching. For proactive, enterprise-grade observability, you need automated alerting mechanisms.

Databricks SQL Alerts allow you to run these analytical queries on a schedule and trigger notifications (via Email, Slack, PagerDuty, or custom Webhooks) if a certain conditional threshold is met.

Setting up a Quality Drop Alert

To build a robust alerting system for data quality, follow these steps:

1. Write the Query: Create a query in Databricks SQL that calculates the failure rate for the last 24 hours directly from the DLT event log.
2. Set the Threshold: Configure the Databricks SQL Alert to evaluate this query periodically and trigger if the failure rate column evaluates to > 5.0 (i.e., a failure rate greater than 5%).
3. Configure Notifications: Route the alert to the Data Engineering Slack channel or an incident management tool.

With this architecture, if a vendor subtly changes the format of the incoming data, causing records to fail DLT expectations, the engineering team receives a Slack alert within minutes of the pipeline run completing. They can then check the Databricks SQL dashboard, identify the exact expectation that failed, investigate the dropped records, and remediate the issue proactively before business users consume corrupted data or make flawed business decisions based on it. This represents the pinnacle of modern data pipeline observability.