11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling

11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling

DP-750 Exam Focus: Master operational observability, notification integrations, and failure remediation in Lakeflow Jobs. Understand system notification events (on_start, on_success, on_failure, on_duration_warning_threshold_exceeded), notification destinations (Email, Slack, Teams, Webhooks), the exact mechanics of "Repair and Rerun" for failed DAG tasks, and enterprise identity governance using Run as Service Principal.

---

1. System Notification Architecture & Lifecycle Events

Real-time visibility into production workflow status is vital for meeting enterprise Service Level Agreements (SLAs). Lakeflow Jobs provides a decoupled notification framework that emits alerts on specific job lifecycle transitions.

+---------------------------------------------------------------------------------------------------------+
|                                 LAKEFLOW JOBS NOTIFICATION ENGINE                                       |
+---------------------------------------------------------------------------------------------------------+
|                                                                                                         |
|   JOB LIFECYCLE EVENTS                                      DESTINATION INTEGRATIONS                    |
|   +---------------------------------------+                 +---------------------------------------+   |
|   | 1. on_start                           |                 | 1. Email (Users & Distribution Lists) |   |
|   |    - Run initializes & allocates VMs  |                 |                                       |   |
|   | 2. on_success                         |                 | 2. Slack Notification Channels        |   |
|   |    - All DAG tasks complete cleanly   |   =========>    |                                       |   |
|   | 3. on_failure                         |                 | 3. Microsoft Teams Webhook Cards      |   |
|   |    - A task fails & halts the DAG     |                 |                                       |   |
|   | 4. on_duration_warning_threshold      |                 | 4. PagerDuty Incident Manager         |   |
|   |    - Job exceeds expected runtime SLA |                 | 5. Custom HTTP Webhooks (JSON POST)   |   |
|   +---------------------------------------+                 +---------------------------------------+   |
+---------------------------------------------------------------------------------------------------------+

Lifecycle Event Types

1. on_start: Fired immediately when the job run transitions from PENDING to RUNNING. Useful for tracking pipeline start times in external audit logs.
2. on_success: Fired when all tasks in the DAG complete successfully with state TERMINATED / SUCCESS.
3. on_failure: Fired when one or more tasks fail and the overall job run transitions to FAILED.
4. on_duration_warning_threshold_exceeded (Soft SLA Alert): Fired when the active execution duration of a job run exceeds a specified threshold in seconds (e.g., 3600 seconds for a 1-hour SLA). Crucially, this alert does NOT terminate the job; it proactively notifies operations teams of cluster degradation or data skew before a hard timeout occurs.

// Example: Job-Level Notification Configuration
{
  "email_notifications": {
    "on_start": ["dataops-audit@company.com"],
    "on_failure": ["dataops-oncall@company.com"],
    "on_duration_warning_threshold_exceeded": ["lead-engineer@company.com"],
    "no_alert_for_skipped_runs": true
  },
  "health": {
    "rules": [
      {
        "metric": "RUN_DURATION_SECONDS",
        "op": "GREATER_THAN",
        "value": 3600
      }
    ]
  }
}

---

2. Notification Destinations (Slack, Teams, Webhooks)

Instead of managing static lists of individual user email addresses across dozens of jobs, Azure Databricks enables Notification Destinations—centralized, reusable webhook integrations managed at the workspace or account level.

Supported Destination Types

• Slack: Posts formatted alerts directly to dedicated DataOps Slack channels via incoming webhooks.
• Microsoft Teams: Dispatches actionable MessageCards to Microsoft 365 / Teams channels.
• PagerDuty: Triggers and resolves incidents automatically for 24/7 on-call engineers.
• Custom HTTP Webhooks: Sends HTTP POST requests with custom JSON payloads to internal monitoring tools, Azure Logic Apps, or ServiceNow.

// Example: Custom Webhook Payload with Dynamic Templating
{
  "job_id": "{{job.id}}",
  "run_id": "{{job.run_id}}",
  "job_name": "{{job.name}}",
  "event": "{{event_type}}",
  "run_url": "{{run_page_url}}",
  "timestamp": "{{start_time.iso_date}}"
}

---

3. Failure Handling & The "Repair and Rerun" (Matrix Rerun) Engine

In complex multi-task DAGs, a failure in a late-stage task (e.g., Task 4 of 5) often leaves the upstream data ingestion and cleansing tasks (Tasks 1, 2, and 3) completed successfully. Re-running the entire job from scratch would waste hours of compute time, double cloud infrastructure costs, and risk duplicate processing on non-idempotent sinks.

Repair and Rerun (also known as Matrix Rerun) allows engineers to fix the underlying issue (e.g., updating a query syntax error or granting table permissions) and re-execute only the failed tasks and their downstream dependents.

+---------------------------------------------------------------------------------------------------------+
|                                 REPAIR AND RERUN (MATRIX RERUN) WORKFLOW                                |
+---------------------------------------------------------------------------------------------------------+
|                                                                                                         |
|  ORIGINAL FAILED RUN (#101):                                                                            |
|  [ Task 1: Ingest ] ===> [ Task 2: Cleanse ] ===> [ Task 3: Aggregate ] ===> [ Task 4: Publish ]       |
|     ( SUCCESS )             ( SUCCESS )                ( FAILED! )                 ( SKIPPED )          |
|                                                             |                                           |
|                                     (Engineer fixes bug in Task 3)                                      |
|                                                             v                                           |
|  REPAIR AND RERUN (#101 - Attempt 2):                                                                   |
|  [ Task 1: Ingest ]      [ Task 2: Cleanse ]      [ Task 3: Aggregate ] ===> [ Task 4: Publish ]       |
|  ( PRESERVED STATE )     ( PRESERVED STATE )         ( RE-EXECUTED )             ( EXECUTED )           |
|  - Uses cached output    - Uses cached TaskValues    - Runs fixed logic          - Downstream completes |
+---------------------------------------------------------------------------------------------------------+

Operational Mechanics of Repair and Rerun

1. Preserved Upstream State: Succeeded tasks are not re-executed. Their outputs, logs, and Task Values (dbutils.jobs.taskValues) remain cached and accessible to the downstream tasks.
2. Selective Execution: Databricks traverses the DAG from the failed task(s), re-running the failed node and any tasks that depend on it.
3. Single Run History: The repair run is recorded as a new attempt attached to the original run_id, preserving audit continuity in Unity Catalog lineage and system tables.
4. API Integration: Can be triggered via the UI ("Repair run") or programmatically via POST /api/2.1/jobs/runs/repair.

# Trigger a Repair and Rerun via Databricks CLI / REST API
databricks jobs repair-run --run-id 451289632 --tasks "aggregate_gold_sales"

---

4. Production Identity Governance: "Run As" Service Principal

In early development, jobs are frequently created by individual data engineers and run under their personal user identities. In an enterprise production environment, this is a critical anti-pattern.

+---------------------------------------------------------------------------------------------------------+
|                                 PRODUCTION IDENTITY GOVERNANCE MATRIX                                   |
+---------------------------------------------------------------------------------------------------------+
|                                                                                                         |
|  ANTI-PATTERN: RUN AS INDIVIDUAL USER                                                                   |
|  - User leaves company -> Microsoft Entra ID account disabled -> ALL PRODUCTION JOBS FAIL!             |
|  - Personal tokens expire -> Pipelines break unexpectedly.                                              |
|  - Auditing is blurred between human interactive queries and automated background ETL.                 |
|                                                                                                         |
|  ENTERPRISE BEST PRACTICE: RUN AS SERVICE PRINCIPAL                                                     |
|  - Dedicated Microsoft Entra ID (Azure AD) App Registration / Service Principal.                        |
|  - Permanent identity decoupled from employee lifecycles.                                               |
|  - Granted explicit, least-privilege Unity Catalog grants (`USE CATALOG`, `SELECT`, `MODIFY`).          |
|  - Secret tokens secured via Azure Key Vault-backed Databricks Secret Scopes.                           |
+---------------------------------------------------------------------------------------------------------+

Steps to Configure "Run As" Service Principal

1. Create Service Principal: Register an Application in Microsoft Entra ID and add it as a Service Principal in the Databricks Account Console.
2. Grant Unity Catalog Privileges: Grant the Service Principal explicit access to required catalogs, schemas, volumes, and SQL warehouses:

   GRANT USE CATALOG, USE SCHEMA ON CATALOG production TO `application-sp-client-id`;
   GRANT SELECT, MODIFY ON SCHEMA production.silver TO `application-sp-client-id`;
   GRANT USE WAREHOUSE ON SQL WAREHOUSE `analytics_serverless_wh` TO `application-sp-client-id`;
   

3. Set Job "Run As" Identity: In the Lakeflow Job settings, navigate to Job details > Run as and select the Service Principal.
4. Secure Credentials: Use Azure Key Vault-backed secret scopes (dbutils.secrets.get(scope="kv-prod", key="sp-secret")) for any required external API credentials.

---

5. Observability with Unity Catalog Lakeflow System Tables

Lakeflow Jobs telemetry is queryable directly within Unity Catalog through dedicated System Tables in the system.lakeflow schema:

-- Example: Querying Average Task Duration & Failure Rates across Production Jobs
SELECT 
    t.task_key,
    t.task_type,
    COUNT(*) AS total_executions,
    SUM(CASE WHEN t.outcome = 'FAILED' THEN 1 ELSE 0 END) AS failure_count,
    AVG(t.execution_duration_seconds) AS avg_duration_sec
FROM system.lakeflow.job_tasks t
JOIN system.lakeflow.job_runs r
    ON t.job_run_id = r.job_run_id
WHERE r.start_time >= CURRENT_DATE() - INTERVAL 30 DAYS
GROUP BY t.task_key, t.task_type
ORDER BY failure_count DESC;