11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts

11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts

DP-750 Exam Focus: Master the configuration of automated trigger mechanisms in Lakeflow Jobs: Scheduled (Cron with timezones), File Arrival (event-driven triggers on ADLS Gen2 External Locations and Unity Catalog Volumes), and Continuous execution. Understand concurrency control with max_concurrent_runs, task retry policies with exponential backoff, and hard/soft timeout boundaries for FinOps and SLA protection.

---

1. Trigger Types in Lakeflow Jobs

Automating pipeline execution requires matching the trigger mechanism to the incoming data frequency, latency requirements, and system architecture.

                               LAKEFLOW JOBS TRIGGER ARCHITECTURE

  +------------------------+  +------------------------+  +------------------------+
  | 1. SCHEDULED TRIGGER   |  | 2. FILE ARRIVAL TRIGGER|  | 3. CONTINUOUS TRIGGER  |
  | (Time-Driven)          |  | (Event-Driven)         |  | (Stream-Driven)        |
  | - Quartz Cron syntax   |  | - Monitors ADLS / UC Vol| | - 24/7 Perpetual Daemon|
  | - Explicit Timezones   |  | - Debounce quiet window|  | - Auto-restart on exit |
  | - Periodic batch ETL   |  | - Ingest on file landing| | - Structured Streaming |
  +------------------------+  +------------------------+  +------------------------+

1. Scheduled Triggers (Cron)

• How It Works: Executes the job periodically based on a standard Quartz Cron expression.
• Timezone Awareness: Always specify an explicit timezone_id (e.g., "America/New_York", "UTC", "Europe/London"). Omitting timezones or relying on server defaults can cause pipelines to shift during Daylight Saving Time (DST) transitions.
• Pause State: Schedules can be paused ("pause_status": "PAUSED") during maintenance windows or environment freezes without deleting the schedule configuration.

// Example: Scheduled Trigger Configuration (Weekdays at 06:00 UTC)
{
  "schedule": {
    "quartz_cron_expression": "0 0 6 ? * MON-FRI",
    "timezone_id": "UTC",
    "pause_status": "UNPAUSED"
  }
}

2. File Arrival Triggers (Event-Driven)

• How It Works: Monitors a specific cloud storage location—such as a Unity Catalog Volume path (/Volumes/catalog/schema/volume/landing/) or an ADLS Gen2 External Location (abfss://container@account.dfs.core.windows.net/raw/).
• Automatic Execution: When a new file arrives or an existing file is updated, Lakeflow Jobs automatically initiates a new job run.
• Debounce Window (wait_after_last_change_seconds): When upstream systems upload large multi-part files (e.g., a 10 GB parquet batch written across several minutes), triggering immediately on the first byte would corrupt downstream ingestion. The quiet period specifies how long the storage path must experience zero write activity before Databricks considers the arrival complete and launches the job (default: 50 seconds).
• Minimum Time Between Triggers (min_time_between_triggers_seconds): Throttles execution frequency to prevent high-velocity file arrivals from triggering thousands of overlapping runs (e.g., minimum 300 seconds between runs).
• Privileges Required: The Job execution identity (Service Principal) must have READ FILES on the Unity Catalog Volume or External Location.

// Example: File Arrival Trigger on a Unity Catalog Volume
{
  "file_arrival": {
    "url": "/Volumes/raw_catalog/landing_zone/partner_feeds/",
    "min_time_between_triggers_seconds": 300,
    "wait_after_last_change_seconds": 60
  }
}

3. Continuous Execution Mode

• How It Works: The job runs perpetually as a 24/7 daemon. When a run completes, terminates, or encounters a recoverable error, Lakeflow Jobs immediately spins up a new run instance.
• Primary Use Case: Long-running Structured Streaming jobs and continuous real-time feature transformation pipelines.

---

2. Concurrency Control & max_concurrent_runs

When jobs are triggered frequently (via schedules, webhooks, or file arrivals), a new run may trigger while a previous run is still processing.

+---------------------------------------------------------------------------------------------------+
|                         CONCURRENCY CONTROL: MAX_CONCURRENT_RUNS = 1                              |
+---------------------------------------------------------------------------------------------------+
|                                                                                                   |
|  Time:    08:00                   08:30                   09:00                   09:30           |
|           |                       |                       |                       |               |
|  Run 1:   [==============================================] (Completes at 08:45)                   |
|                                   |                                                               |
|  Run 2:                           * Trigger Fires!                                                |
|                                   [ QUEUED / WAITING ] -> [==============================]          |
|                                                           (Starts at 08:45 once Run 1 finishes)   |
+---------------------------------------------------------------------------------------------------+

Operational Rules for max_concurrent_runs

• max_concurrent_runs: 1 (Default & Gold Standard for Delta Writes): Ensures that only a single instance of the job executes at any given moment. Subsequent trigger events are placed into a FIFO Queue (up to a workspace queue limit) and execute sequentially.
• Preventing Write Conflicts: Multiple concurrent runs appending or merging into the same target Delta table can cause optimistic concurrency control conflicts (ConcurrentAppendException or ConcurrentModificationException). Setting max_concurrent_runs: 1 serializes execution and completely eliminates race conditions.
• Scaling Concurrency (max_concurrent_runs > 1): Appropriate for stateless ETL tasks, parameter-driven backfills, or read-only analytical workloads where runs operate on completely disjoint partition slices.

---

3. Task-Level Retry Policies & Exponential Backoff

Distributed systems inevitably experience transient network blips, API rate-limiting (HTTP 429), or temporary deadlocks on external databases. Lakeflow Jobs allows configuring granular Retry Policies at the task level.

                              EXPONENTIAL BACKOFF RETRY MECHANICS

  [ Task Fails ] ===> Attempt 1 Failed (HTTP 429 Rate Limit)
         |
         +---> Wait: 10,000 ms (10s)  ===========> [ Attempt 2 Fails ]
                     |
                     +---> Wait: 20,000 ms (20s)  =======> [ Attempt 3 Fails ]
                                 |
                                 +---> Wait: 40,000 ms (40s)  ===> [ Attempt 4: SUCCESS! ]

Retry Policy Parameters

• max_retries: The maximum number of retry attempts after the initial execution fails (e.g., 3). A value of 0 disables retries.
• min_retry_interval_millis: The initial delay before the first retry attempt (e.g., 10000 for 10 seconds).
• retry_on_timeout: A boolean flag (true/false). If true, the task will retry even if the failure was caused by reaching the task's timeout limit.

Exponential Backoff Formula

When retries are triggered, Azure Databricks applies an exponential backoff formula with jitter to prevent retrying tasks from overwhelming downstream services (thundering herd problem): $\text{Delay}(n) = \text{min\_retry\_interval\_millis} \times 2^{(n-1)}$ Where $n$ represents the retry attempt index (1, 2, 3...).

// Example: Robust Task Retry Configuration
{
  "task_key": "fetch_rest_api_data",
  "max_retries": 3,
  "min_retry_interval_millis": 15000,
  "retry_on_timeout": false,
  "timeout_seconds": 600
}

---

4. Timeout Governance: Task-Level vs. Job-Level

Unchecked, hanging Spark jobs (e.g., infinite loops in custom Python code, unresolved network sockets, or Cartesian product joins) can run indefinitely on Azure VMs, consuming thousands of dollars in compute costs without making progress.

+---------------------------------------------------------------------------------------------------+
|                                TIMEOUT GOVERNANCE HIERARCHY                                       |
+---------------------------------------------------------------------------------------------------+
|                                                                                                   |
|  JOB-LEVEL TIMEOUT (`timeout_seconds = 7200` / 2 Hours)                                           |
|  +---------------------------------------------------------------------------------------------+  |
|  |                                                                                             |  |
|  |  Task 1: Ingest (Timeout: 1200s) ===> Completed in 300s                                     |  |
|  |                                                                                             |  |
|  |  Task 2: Transform (Timeout: 1800s) ===> HANGS! Terminated at 1800s by Task Timeout!        |  |
|  |                                                                                             |  |
|  |  Overall Job Hard Kill: If total DAG wall-clock time reaches 7200s, ENTIRE JOB TERMINATES.  |  |
|  +---------------------------------------------------------------------------------------------+  |
+---------------------------------------------------------------------------------------------------+

1. Task-Level Timeout (timeout_seconds)

• Scope: Applied to an individual task within the DAG.
• Behavior: If the task execution duration exceeds this threshold, Databricks automatically issues a cancel command to the cluster executor and fails the task.
• Best Practice: Set task-level timeouts based on historical P99 runtimes plus a reasonable safety buffer (e.g., 2x average duration).

2. Job-Level Timeout (timeout_seconds)

• Scope: Applied to the entire Job Run from start to finish.
• Behavior: Provides an absolute safety ceiling for the total wall-clock duration of all tasks, retries, and provisioning steps combined. If reached, all running tasks are immediately aborted and the job state transitions to TIMED_OUT.