11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables

11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables

DP-750 Exam Focus: Master dynamic control flow, state sharing, and parameter management in Lakeflow Jobs. Understand the exact operational mechanics of all six Run If conditional trigger rules, programmatic state sharing with dbutils.jobs.taskValues, dynamic variable templating ({{tasks.<task_name>.values.<key>}}, {{job.id}}, {{start_time.iso_date}}), and parameter precedence hierarchies.

---

1. Task Dependencies & The Run If Execution Engine

In a multi-task DAG, tasks are linked via the depends_on property, which specifies one or more upstream task keys. By default, a downstream task executes only when all direct upstream tasks succeed. However, real-world enterprise pipelines require sophisticated failure recovery, branch convergence, and cleanup routines.

Lakeflow Jobs provides Run If conditional triggers that dictate the exact operational condition under which a downstream task is triggered.

+---------------------------------------------------------------------------------------------------------+
|                                 LAKEFLOW JOBS "RUN IF" TRIGGER ENGINE                                   |
+---------------------------------------------------------------------------------------------------------+
|                                                                                                         |
|   +-----------------------+       +-----------------------+                                             |
|   |  Upstream Task A      |       |  Upstream Task B      |                                             |
|   |  (e.g., Ingest POS)   |       |  (e.g., Ingest Web)   |                                             |
|   +-----------------------+       +-----------------------+                                             |
|               \                       /                                                                 |
|                \                     /                                                                  |
|                 v                   v                                                                   |
|        +-------------------------------------+                                                          |
|        |        DOWNSTREAM TASK C            |                                                          |
|        |                                     |                                                          |
|        | Evaluates Selected `Run If` Trigger |                                                          |
|        +-------------------------------------+                                                          |
|                           |                                                                             |
|   +-----------------------+-----------------------+-----------------------+                             |
|   |                       |                       |                       |                             |
|   v                       v                       v                       v                             |
| [ ALL_SUCCESS ]     [ NONE_FAILED ]     [ AT_LEAST_ONE_FAILED ]     [ ALL_DONE ]                        |
| (Standard Flow)     (Tolerates Skips)   (Alerting & Incident)       (Resource Cleanup)                  |
+---------------------------------------------------------------------------------------------------------+

Comprehensive Run If Trigger Reference

| Run If Trigger Condition | Execution Rule & Upstream Dependency Evaluation | Typical Architectural Use Case | |:---|:---|:---|| | ALL_SUCCESS (Default) | Executes only when every direct upstream task completes with status SUCCESS. If any upstream task fails or is skipped, this task is skipped. | Standard linear or fan-in ETL where all data sources are required. | | AT_LEAST_ONE_SUCCESS | Executes if at least one direct upstream task completes with SUCCESS. Does not execute if all upstream tasks fail or are skipped. | High-availability fan-in where secondary data feeds provide redundancy. | | NONE_FAILED | Executes if no direct upstream task failed. Upstream tasks can be SUCCESS, SKIPPED, or excluded by upstream conditions. | Downstream tasks following conditional If/Else branches where one branch was intentionally skipped. | | ALL_FAILED | Executes only if every direct upstream task terminates with status FAILED. | Specialized disaster recovery, fallback data restoration, or primary/secondary pipeline failure triage. | | AT_LEAST_ONE_FAILED | Executes if one or more direct upstream tasks terminate with status FAILED. | Automated incident ticket creation (ServiceNow, Jira), Slack error alerting, or emergency resource teardown. | | ALL_DONE | Executes when all upstream tasks finish execution, regardless of whether they succeeded, failed, or were skipped. | Mandatory final cleanup tasks, audit logging, telemetry reporting, or releasing external distributed locks. |

Exam Tip: If a task follows an If/Else condition where one branch is skipped, setting the downstream convergence task to ALL_SUCCESS will cause it to be skipped! You must use NONE_FAILED to allow the convergence task to execute after conditional branching.

---

2. Programmatic State Sharing with Task Values (dbutils.jobs.taskValues)

In traditional architectures, passing small runtime variables (such as row counts, dynamic file paths, anomaly scores, or max timestamps) between tasks required writing temporary JSON or Delta files to ADLS Gen2 storage. This introduced storage I/O latency, required storage cleanup, and created concurrency conflicts.

Task Values provide a high-performance, in-memory, key-value state store managed directly by the Lakeflow Jobs Control Plane.

+---------------------------------------------------------------------------------------------------+
|                               TASK VALUES INTER-TASK COMMUNICATION                                |
+---------------------------------------------------------------------------------------------------+
|                                                                                                   |
|   TASK 1: "validate_orders" (PySpark)                                                             |
|   +--------------------------------------------------------------------------------------------+  |
|   | bad_rows = df.filter(col("is_valid") == False).count()                                     |  |
|   | dbutils.jobs.taskValues.set(key="quarantine_count", value=bad_rows)                        |  |
|   +--------------------------------------------------------------------------------------------+  |
|                                                |                                                  |
|                     (Persisted in Job Control Plane - Max 48 KB)                                  |
|                                                v                                                  |
|   TASK 2: "process_silver" (PySpark)                  TASK 3: "notify_sql" (Databricks SQL)       |
|   +---------------------------------------------+     +----------------------------------------+  |
|   | count = dbutils.jobs.taskValues.get(        |     | SELECT * FROM silver.orders            |  |
|   |   taskKey="validate_orders",                |     | WHERE quarantine_records >             |  |
|   |   key="quarantine_count",                   |     |   '{{tasks.validate_orders.values.     |  |
|   |   default=0                                 |     |     quarantine_count}}';               |  |
|   | )                                           |     +----------------------------------------+  |
|   +---------------------------------------------+                                                 |
+---------------------------------------------------------------------------------------------------+

Task Values API Syntax & Rules

Setting a Task Value (Upstream Task):

# PySpark Notebook or Python Task
row_count = df.count()
max_watermark = "2026-08-26T12:00:00Z"

# Set task values (Key-Value pairs)
dbutils.jobs.taskValues.set(key="ingested_rows", value=row_count)
dbutils.jobs.taskValues.set(key="watermark_timestamp", value=max_watermark)

Retrieving a Task Value Programmatically (Downstream Python/Scala Task):

# Downstream PySpark Notebook or Python Task
ingested_count = dbutils.jobs.taskValues.get(
    taskKey="ingest_task", 
    key="ingested_rows", 
    default=0,
    debugValue=100  # Used when running notebook interactively outside of a Job
)

print(f"Processing {ingested_count} records from upstream task.")

Task Values Constraints & DP-750 Gotchas

• Payload Size Limit: The total size of all task values set by a single task cannot exceed 48 KB (JSON-serialized). Task values are designed for metadata, metrics, and scalar values—never for passing DataFrames or binary payloads.
• Data Types: Must be JSON-serializable primitives (integers, floats, strings, booleans, lists, dicts).
• Scope: Scoped strictly to the specific job run instance. A task cannot access task values from a prior or concurrent run of the job.
• Interactive Execution: When testing a notebook interactively in a workspace, taskValues.get() will fail unless a debugValue parameter is provided.

---

3. Dynamic Value References & Built-in System Variables

Lakeflow Jobs supports Dynamic Value References using double curly-brace syntax ({{ ... }}). This allows injecting upstream task values, job metadata, and temporal parameters directly into SQL queries, Python CLI arguments, notebook base parameters, and webhook notification payloads.

1. Referencing Upstream Task Values in Configuration & SQL

Any task type—including SQL tasks, JAR tasks, and dbt tasks—can consume task values set by upstream Python/Scala tasks without writing custom retrieval code:

-- Databricks SQL Query Task consuming an upstream Task Value
INSERT INTO silver.metrics.pipeline_audit
VALUES (
    '{{job.id}}',
    '{{job.run_id}}',
    '{{tasks.ingest_raw_events.values.ingested_rows}}',
    CURRENT_TIMESTAMP()
);

2. Built-in Dynamic System Variables Reference

| Dynamic Variable Syntax | Description & Output Format | Example Value | |:---|:---|:---|| | {{job.id}} | Unique numerical identifier of the Job definition | 984210542389 | | {{job.run_id}} | Unique numerical identifier of the specific Job Run | 451289632 | | {{job.name}} | Name of the Job | "Enterprise_Medallion_ETL" | | {{task.name}} | Name of the currently executing task key | "curate_silver_customers" | | {{task.run_id}} | Unique run identifier of the specific task attempt | 874512963 | | {{task.execution_count}} | Number of times this task has executed in this run (1-based) | 1 (or 2 on retry) | | {{start_time.iso_date}} | Job start date formatted in ISO 8601 UTC date | 2026-08-26 | | {{start_time.timestamp}} | Job start timestamp formatted in seconds since epoch | 1787745600 | | {{parent_run_id}} | Run ID of the parent job (if triggered via a Run Job task) | 451289000 |

// Example: Passing dynamic variables to a Python Script Task in Job Definition
{
  "task_key": "process_daily_partition",
  "spark_python_task": {
    "python_file": "/Volumes/main/etl/scripts/daily_transform.py",
    "parameters": [
      "--date", "{{start_time.iso_date}}",
      "--run_id", "{{job.run_id}}",
      "--threshold", "{{tasks.calculate_threshold.values.dynamic_cutoff}}"
    ]
  }
}

---

4. Parameter Hierarchy & Precedence Rules

Enterprise pipelines often require setting default parameters while allowing ad-hoc backfills or environments to override values at runtime.

                             PARAMETER PRECEDENCE HIERARCHY

     HIGHEST PRECEDENCE  ===>  [ 1. RUNTIME TRIGGER PARAMETER OVERRIDES ]
                               (Passed via API `run-now` or UI "Run with parameters")
                                             |
                                             v
                               [ 2. TASK-LEVEL PARAMETERS ]
                               (Configured inside individual task definition)
                                             |
                                             v
     LOWEST PRECEDENCE   ===>  [ 3. JOB-LEVEL BASE PARAMETERS ]
                               (Configured at top-level job definition)

Parameter Resolution Rules

1. Job-Level Base Parameters: Defined at the top-level job configuration. These serve as global fallback variables inherited by all tasks in the DAG.
2. Task-Level Parameters: Defined inside specific task definitions. If a task parameter shares the same key as a job base parameter, the task-level parameter overrides the job-level value for that task only.
3. Runtime Trigger Overrides: Passed dynamically when triggering a job via the Databricks UI ("Run now with different parameters") or via the REST API (POST /api/2.1/jobs/run-now with job_parameters). Runtime overrides take highest precedence, overwriting both job-level and task-level parameter defaults across all tasks.

# In Databricks Notebook Task: Accessing Parameters via dbutils.widgets
dbutils.widgets.text("target_environment", "dev", "Target Deployment Env")
dbutils.widgets.text("batch_date", "2026-01-01", "Batch Processing Date")

env = dbutils.widgets.get("target_environment")
batch_date = dbutils.widgets.get("batch_date")
print(f"Executing ETL for Environment: {env}, Date: {batch_date}")