9.3 Scheduled Refresh, Email Subscriptions & Alerts

Maintaining automated data freshness and configuring proactive notifications are critical components of operational reporting in Databricks. Business decision-makers rely on up-to-date visual metrics and timely alerts when key business indicators breach acceptable boundaries. Databricks provides two distinct mechanisms to handle automated updates and notifications: Scheduled Refresh with Email Subscriptions for AI/BI Dashboards, and Databricks SQL Alerts for standalone query monitoring.

Scheduled Dashboard Refreshes & Engine Architecture

A published AI/BI Dashboard can be configured with an automated Scheduled Refresh. When a schedule triggers, Databricks automatically wakes or utilizes the assigned Databricks SQL Warehouse to re-execute all dataset queries powering the published dashboard canvas, populating the dashboard cache with fresh data.

Scheduled Execution Mechanics

1. Trigger Phase: The schedule fires based on a predefined UI interval (e.g., Every 2 hours) or a custom 5-field CRON expression (e.g., 0 0 8 * * ? for daily at 8:00 AM).
2. Compute Allocation: Databricks routes query execution to the SQL Warehouse designated in the dashboard settings. Using Serverless SQL Warehouses for scheduled refreshes is strongly recommended, as serverless warehouses start in seconds and automatically scale down immediately after queries complete.
3. Cache Updating: The resulting query dataset outputs overwrite cached result sets stored for the published dashboard. Subsequent user visits to the published dashboard load instantly from cache without waiting for query re-execution.
4. Parameter Context: Scheduled refreshes execute using default parameter values saved in the published dashboard state unless specific parameter overrides are configured in the schedule definition.

# Example 5-field CRON schedule for weekdays at 6:00 AM UTC
0 6 * * MON-FRI

Email Subscriptions & Snapshot Distribution

Alongside scheduled refreshes, authors can configure Email Subscriptions to automatically broadcast updated dashboard visual assets to stakeholders.

Subscription Delivery Formats

Subscribers can receive dashboard updates in three complementary formats:

• Inline Image (PNG): A high-resolution rendered visual snapshot of the primary dashboard canvas embedded directly into the body of the email.
• PDF Attachment: A multi-page PDF document rendering all dashboard canvas tabs and visual widgets, formatted for printing or executive presentation.
• CSV Data Export: Downloadable CSV files attached to the email containing raw result sets for specified underlying SQL datasets.

Distribution Governance & Admin Restrictions

Subscriptions can be delivered to individual workspace users, Databricks user groups, or external email addresses (such as external clients or vendors). However, sending email subscriptions to non-workspace email domains requires workspace admin authorization. In the Databricks Admin Console, administrators can enable or disable external email subscriptions workspace-wide to prevent sensitive corporate metrics from leaving the corporate security perimeter.

Databricks SQL Alerts & Threshold Monitoring

While Dashboard Email Subscriptions send periodic reports regardless of data changes, Databricks SQL Alerts provide proactive, condition-based monitoring. Alerts periodically evaluate the output of a single Databricks SQL query and trigger notifications only when specified data thresholds are breached.

Alert Architecture & Evaluation Workflow

Databricks SQL Alerts operate independently of visual dashboard canvases. They are attached directly to scheduled SQL queries written in the Databricks SQL Editor.

-- Query designed for a Databricks SQL Alert monitoring high-latency jobs
SELECT 
    COUNT(*) AS high_latency_count,
    MAX(duration_seconds) AS max_duration
FROM main.system_logs.job_execution_history
WHERE execution_date = CURRENT_DATE()
  AND duration_seconds > 3600;

Alert Condition Configuration

When setting up an alert, the author defines a logical condition based on a column in the query result set:

• Value Comparison: Triggers when a specified column value is Greater Than, Less Than, Greater Than or Equal To, Less Than or Equal To, or Equal To a threshold number (e.g., high_latency_count > 0).
• Null Checking: Triggers when a column Is Null or Is Not Null (useful for detecting pipeline data gaps).
• Change Threshold: Triggers when a value changes by a specified absolute amount or percentage compared to the previous execution.

Alert States & Re-notification Policies

Databricks SQL Alerts transition between three operational states:

Authors can configure one of two re-notification policies when an alert enters the TRIGGERED state:

• Just Once: Sends a single notification when the state transitions from OK to TRIGGERED. No further notifications are sent on subsequent scheduled query runs as long as the alert remains in TRIGGERED state.
• Every Time: Sends a notification on every scheduled execution for as long as the alert remains in TRIGGERED state (ideal for urgent operational outages).

Notification Destinations & Webhook Integration

Both Dashboard Subscriptions and SQL Alerts integrate with Notification Destinations configured in Databricks SQL. Rather than relying solely on email, alerts can broadcast payload notifications to enterprise communication platforms and incident response tools.

Supported Destinations

• Email: Direct delivery to specified email addresses.
• Slack: Webhook integration posting structured alert messages into designated Slack channels.
• Microsoft Teams: Webhook delivery into Teams channel channels.
• PagerDuty: Incident creation for critical production threshold breaches.
• Generic Webhooks: Standard HTTP POST requests delivering JSON payloads to custom API endpoints, AWS Lambda functions, or external workflow engines.

// Example payload delivered by a Databricks SQL Alert Generic Webhook
{
  "event_type": "ALERT_TRIGGERED",
  "alert_id": "alert-884920",
  "alert_name": "Job Latency SLA Breach",
  "state": "TRIGGERED",
  "query_id": "query-55123",
  "value": 14,
  "threshold": 0
}

Comparing Dashboard Refreshes vs. SQL Alerts