11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection

11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection

DP-750 Exam Focus: Master the design and implementation of multi-task Directed Acyclic Graphs (DAGs) in Lakeflow Jobs (Databricks Workflows). Understand the operational boundaries, compute bindings, and optimal use cases for every task type: Notebook, Python Script, Python Wheel, SQL (Query, Dashboard, Alert), Lakeflow Declarative Pipelines (DLT), dbt, If/Else Condition, and Run Job. Learn compute optimization strategies, specifically comparing Shared Job Clusters with Serverless Workflows and dedicated clusters.

---

1. Architectural Foundations of Lakeflow Jobs

In enterprise data engineering, orchestrating complex end-to-end data flows requires coordinating data ingestion, cleansing, transformation, quality validation, machine learning inference, and business intelligence reporting. Traditional monolithic scripts or single-notebook jobs create severe operational bottlenecks: a failure in an end-stage reporting query requires re-running the entire hours-long ingestion pipeline, and diverse workloads cannot leverage specialized compute engines.

Lakeflow Jobs (the evolution of Databricks Workflows) provides a cloud-native, fully managed orchestration engine natively integrated into Azure Databricks and Unity Catalog. Lakeflow Jobs allows data engineers to construct Directed Acyclic Graphs (DAGs) composed of discrete, loosely coupled tasks that execute sequentially, concurrently, or conditionally based on upstream dependencies.

+---------------------------------------------------------------------------------------------------------+
|                                    LAKEFLOW JOBS CONTROL PLANE & RUNTIME                                |
+---------------------------------------------------------------------------------------------------------+
|                                                                                                         |
|   +-------------------------------------------------------------------------------------------------+   |
|   |                                  LAKEFLOW JOBS DAG COORDINATOR                                  |   |
|   |  - Evaluates Task Dependencies (`depends_on`)          - Evaluates Run If Conditions            |   |
|   |  - Injects Parameters & Dynamic Variables              - Coordinates Matrix Repair & Rerun      |   |
|   +-------------------------------------------------------------------------------------------------+   |
|                                                    |                                                    |
|                                                    v                                                    |
|   +-------------------------------------------------------------------------------------------------+   |
|   |                                  HETEROGENEOUS TASK EXECUTION                                   |   |
|   |                                                                                                 |   |
|   |  [Task 1: DLT Pipeline]  -->  [Task 2: Python Wheel]  -->  [Task 3: If/Else]                    |   |
|   |       (Ingest Bronze)             (Silver Cleansing)          (Quality Check)                   |   |
|   |              |                            |                             |                       |   |
|   |              v                            v                             +--> [Task 4: Gold SQL] |   |
|   |   (Serverless DLT Engine)       (Shared Job Cluster)                    +--> [Task 5: Alert]    |   |
|   +-------------------------------------------------------------------------------------------------+   |
+---------------------------------------------------------------------------------------------------------+

Key Architectural Characteristics of Lakeflow Jobs

1. Native Unity Catalog Governance: Every task inherits the identity and governance boundaries of the job's execution context (such as an Azure Entra ID Service Principal), enforcing fine-grained table, volume, and compute permissions without passing embedded credentials.
2. Multi-Task Orchestration: A single job can contain up to 100 distinct tasks organized into complex dependency graphs with fan-out (parallel branching) and fan-in (convergence) topologies.
3. Cross-Engine Workload Support: Different tasks within the same job can execute on different compute engines (e.g., Serverless SQL Warehouses for SQL queries, automated multi-node GPU clusters for ML training, and Shared Job Clusters for PySpark transformations).
4. End-to-End Lineage & Observability: Execution metadata, task durations, data quality metrics, and task value handoffs are captured in Unity Catalog system tables (system.lakeflow.jobs and system.lakeflow.job_runs).

---

2. Comprehensive Task Type Selection Matrix

Selecting the correct task type is essential for pipeline maintainability, testability, security, and performance. Azure Databricks supports a wide array of specialized task types:

                                  LAKEFLOW JOBS TASK ECOSYSTEM

  +------------------------+  +------------------------+  +------------------------+
  | CODE-CENTRIC TASKS     |  | SQL & BI TASKS         |  | DECLARATIVE & SPECIAL  |
  | - Notebook             |  | - SQL Query            |  | - Lakeflow DLT Pipeline|
  | - Python Script        |  | - SQL Dashboard / View |  | - dbt Task             |
  | - Python Wheel         |  | - SQL Alert            |  | - If/Else Condition   |
  | - JAR / Spark Submit   |  |                        |  | - Run Job (Child DAG)  |
  +------------------------+  +------------------------+  +------------------------+

Deep Dive into Supported Task Types

1. Notebook Task

• Description: Executes an interactive Databricks notebook (.py, .sql, .scala, .r) stored in workspace folders or Databricks Git Folders.
• Compute: Runs on Serverless Workflows, Automated Job Clusters, or All-Purpose Clusters.
• Use Case: Rapid prototyping, exploratory pipeline development, and workflows requiring mixed language cells or rich inline visualizations.
• Limitations: Harder to unit test with standard CI/CD frameworks (pytest) compared to modular Python packages.

2. Python Script Task

• Description: Executes a standalone Python script (.py) hosted in a Unity Catalog Volume, workspace file, or Git repository.
• Compute: Serverless Workflows or Automated Job Clusters.
• Use Case: Production data engineering jobs that adhere to standard Python coding standards without requiring notebook UI overhead.

3. Python Wheel Task

• Description: Executes a packaged, version-controlled Python Wheel (.whl) distribution installed from a Unity Catalog Volume, cloud storage URI, or PyPI repository. Specifies an explicit entry_point function and named parameters.
• Compute: Serverless Workflows or Automated Job Clusters.
• Use Case: Enterprise Gold Standard for Production ETL. Enables strict software engineering rigor, including strict unit testing (pytest), linting (flake8, ruff), dependency locking, and semantic versioning.

{
  "task_key": "curate_silver_orders",
  "python_wheel_task": {
    "package_name": "enterprise_etl",
    "entry_point": "process_orders",
    "named_parameters": {
      "source_catalog": "bronze",
      "target_catalog": "silver",
      "batch_date": "{{start_time.iso_date}}"
    }
  },
  "libraries": [
    {
      "whl": "/Volumes/main/artifacts/wheels/enterprise_etl-2.4.1-py3-none-any.whl"
    }
  ]
}

4. Databricks SQL Tasks (Query, Dashboard, Alert)

• SQL Query Task: Executes a saved Databricks SQL query or inline SQL statements against a designated Serverless SQL Warehouse or Pro SQL Warehouse. Supports passing query parameters and exporting results.
• SQL Dashboard / Lakeview Dashboard Task: Refreshes an entire AI/BI dashboard or legacy SQL dashboard, updating underlying cached query results and distributing scheduled PDF/export reports to stakeholders.
• SQL Alert Task: Evaluates a SQL query condition (e.g., COUNT(*) > 0 on an anomaly table) and triggers alerts to notification destinations if the threshold is breached.

5. Lakeflow Declarative Pipelines / Delta Live Tables (DLT) Task

• Description: Triggers the execution of a declarative Delta Live Tables pipeline. Supports running in Triggered (Batch) mode or updating a Continuous streaming pipeline.
• Parameters: Allows passing full_refresh: true to rebuild all streaming tables and materialized views from source, or selective table refresh.
• Compute: Compute is managed directly by the DLT pipeline settings (Serverless DLT or classic DLT clusters).

6. dbt Task

• Description: Natively orchestrates dbt-core or dbt-databricks transformation projects hosted in Git repositories. Runs commands such as dbt deps, dbt seed, dbt run, and dbt test directly on Databricks compute or SQL Warehouses.
• Use Case: Enterprise analytics engineering teams using dbt for semantic modeling and SQL transformations inside the Lakehouse.

7. If/Else Condition Task

• Description: A control-flow evaluation node that evaluates a boolean expression comparing dynamic variables, job parameters, or upstream Task Values (e.g., {{tasks.validate_data.values.error_count}} == 0).
• Execution Paths: Directs downstream DAG execution to either the true branch or false branch. Does not require dedicated compute infrastructure.

8. Run Job Task (Cross-Job Orchestration)

• Description: Triggers another independent Databricks Job as a child task within the parent DAG. Can pass parameters dynamically to the child job and wait for its completion.
• Use Case: Modular, federated orchestration where different domain teams maintain their own sub-pipelines (e.g., Finance Job triggers Supply Chain Job).

---

Task Type Comparison Matrix for DP-750

---

3. Compute Optimization: Shared Job Clusters vs. Serverless Workflows

Compute selection in Lakeflow Jobs significantly influences execution latency, total cost of ownership (DBUs and Azure VM infrastructure), and operational reliability.

+---------------------------------------------------------------------------------------------------+
|                                 JOB COMPUTE ALLOCATION STRATEGIES                                 |
+---------------------------------------------------------------------------------------------------+
|                                                                                                   |
|  STRATEGY A: SHARED JOB COMPUTE (Cost & Provisioning Optimized)                                   |
|  +---------------------------------------------------------------------------------------------+  |
|  | Job Start ===> [ Spin up Single Job Cluster ]                                               |  |
|  |                     |                                                                       |  |
|  |                     +---> [ Task 1: Ingest ] (Uses Cluster)                                 |  |
|  |                     +---> [ Task 2: Cleanse ] (Reuses Same Cluster)                         |  |
|  |                     +---> [ Task 3: Aggregate ] (Reuses Same Cluster)                       |  |
|  |                     |                                                                       |  |
|  | Job Finish <== [ Terminate Cluster Automatically ]                                          |  |
|  +---------------------------------------------------------------------------------------------+  |
|                                                                                                   |
|  STRATEGY B: NEW CLUSTER PER TASK (Anti-Pattern: High Latency & Overhead)                         |
|  +---------------------------------------------------------------------------------------------+  |
|  | Task 1: Spin Up VM (4m) -> Run (2m) -> Terminate                                            |  |
|  | Task 2: Spin Up VM (4m) -> Run (3m) -> Terminate (Repeated 4m cold-start overhead!)         |  |
|  +---------------------------------------------------------------------------------------------+  |
|                                                                                                   |
|  STRATEGY C: SERVERLESS WORKFLOWS (Next-Gen: Sub-10s Startup, Zero VM Management)                 |
|  +---------------------------------------------------------------------------------------------+  |
|  | Task 1 (Instant 5s Start) -> Task 2 (Instant Start) -> Auto-scales and bills exact seconds!  |  |
|  +---------------------------------------------------------------------------------------------+  |
+---------------------------------------------------------------------------------------------------+

1. Shared Job Compute (Job Clusters)

• How It Works: When configuring a multi-task job, you define a single Job Cluster at the job level. Multiple tasks within the DAG are assigned to reuse this cluster.
• Lifecycle: The cluster is provisioned exactly once when the first task starts and is terminated automatically when the final task completes.
• Benefits:
  • Eliminates Cold Start Latency: Avoids waiting 3–6 minutes for new Azure VM provisioning between consecutive tasks.
  • Lowest DBU Pricing: Job compute is billed at the discounted Job DBU rate (significantly cheaper than All-Purpose interactive compute).
  • Resource Reuse: In-memory cached tables or libraries installed during initialization remain available across sequential tasks on the same cluster.

2. Serverless Compute for Workflows

• How It Works: Databricks manages the underlying compute pool completely. Tasks run on secure, pre-warmed container infrastructure managed by Microsoft and Databricks.
• Benefits:
  • Near-Zero Startup Latency: Starts executing within 5–10 seconds instead of minutes.
  • Granular Auto-Scaling: Automatically scales compute up or down per task based on workload demands.
  • Zero Infrastructure Maintenance: Eliminates the need to select VM families, configure subnet CIDRs, or manage cluster policies.
  • Isolated Security Boundaries: Each task runs in a separate secure micro-container, preventing memory contamination between jobs.

3. Dedicated Job Cluster per Task

• When to Use: Only use separate job clusters within the same DAG when individual tasks have radically conflicting compute requirements—such as a single-node memory-optimized task (Standard_E32ds_v5) followed by a multi-node GPU-accelerated ML training task (Standard_NC6s_v3).

Exam Trap: Never use All-Purpose Compute for scheduled production jobs. All-Purpose clusters are billed at roughly double the DBU rate of Job Compute, fail to isolate job run environments, and can be inadvertently terminated by interactive users.

---

4. Best Practices for Enterprise Multi-Task DAG Design

1. Enforce Modularity: Break monolithic notebooks into small, single-responsibility tasks (e.g., Ingestion, Validation, Transformation, Publishing, Alerting). This enables granular retries and parallel execution.
2. Leverage Shared Compute: Standardize tasks on a single Shared Job Cluster or Serverless compute to eliminate provisioning overhead.
3. Use Python Wheels for Production Logic: Package shared utility functions, schema definitions, and business rules into Python Wheels and reference them across tasks.
4. Apply Cluster Policies: Attach Cluster Policies to Job clusters to enforce maximum node sizes, approved VM families (e.g., preventing expensive GPU nodes for simple ETL), and mandatory FinOps billing tags.
5. Design for Idempotency: Ensure that every task in the DAG is idempotent (using Delta MERGE INTO or deterministic partition overwrites) so that retried tasks produce identical, non-corrupted state.