12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure

12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure

As enterprise lakehouses grow in complexity, managing pipelines, scheduled workflows, cluster configurations, and permissions through the web UI becomes error-prone and unscalable. Infrastructure drift between development, staging, and production environments frequently causes deployment failures.

Databricks Asset Bundles (DAB) is Databricks' official Infrastructure-as-Code (IaC) standard. DAB allows data engineers to define complete data engineering projects—including multi-task Lakeflow Jobs, Delta Live Tables (DLT) declarative pipelines, MLflow experiments, custom Python wheels, and notebooks—as declarative, version-controlled YAML and source files.

---

1. DAB Architecture & Directory Anatomy

A Databricks Asset Bundle project structures source code, resource configuration manifests, and test suites into a modular directory hierarchy:

my_lakehouse_project/
├── databricks.yml                 # Root bundle configuration manifest
├── resources/                     # Modular resource YAML definitions
│   ├── sales_ingest_job.yml       # Multi-task Lakeflow Job definition
│   └── telemetry_pipeline.yml     # Delta Live Tables pipeline definition
├── src/                           # Source code (Python, SQL, Wheels)
│   ├── __init__.py
│   ├── transforms.py
│   └── dlt_bronze_silver.py
├── tests/                         # Unit and integration test suites
│   ├── conftest.py
│   └── test_transforms.py
├── fixtures/                      # Sample datasets for unit testing
│   └── sample_events.json
└── pyproject.toml                 # Build configuration for Python wheels

+-------------------------------------------------------------------------+
|                     DAB COMPILATION & DEPLOYMENT FLOW                   |
+-------------------------------------------------------------------------+
|                                                                         |
|  1. LOCAL DEV               2. BUNDLE ENGINE         3. WORKSPACE       |
|  +--------------------+     +-------------------+    +----------------+ |
|  | databricks.yml     |     | CLI parses YAML   |    | Workspace File | |
|  | resources/*.yml    | --> | Interpolates vars | -> | /Users/..      | |
|  | src/*.py           |     | Builds wheels/pkg |    | Jobs / DLT API | |
|  +--------------------+     +-------------------+    +----------------+ |
|                                                                         |
+-------------------------------------------------------------------------+

---

2. The databricks.yml Root Manifest Schema

The databricks.yml file sits at the root of the project. It defines the bundle metadata, file inclusions, variable definitions, and target environments:

# Root bundle identifier
bundle:
  name: retail_sales_analytics

# Include modular YAML definitions from the resources/ folder
include:
  - resources/*.yml

# Global variable declarations with defaults
variables:
  catalog_name:
    description: "Target Unity Catalog catalog for Delta tables"
    default: "dev_catalog"
  schema_name:
    description: "Target Unity Catalog schema"
    default: "retail"
  worker_node_type:
    description: "Azure VM node type for compute clusters"
    default: "Standard_D4ds_v5"
  notification_email:
    description: "Email recipient for failure alerts"
    default: "dataops-alerts@corp.com"

# Target environment definitions
targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://adb-dev.7.azuredatabricks.net

  prod:
    mode: production
    workspace:
      host: https://adb-prod.7.azuredatabricks.net
      root_path: /Shared/.bundle/${bundle.name}/${bundle.target}
    variables:
      catalog_name: prod_catalog
      worker_node_type: Standard_D8ds_v5

Variable Interpolation Syntax

DAB supports dynamic string interpolation within YAML manifests using the ${...} syntax:

• Custom Variables: ${var.catalog_name}, ${var.schema_name}
• Bundle Context: ${bundle.name}, ${bundle.target}, ${bundle.environment}
• Workspace Context: ${workspace.current_user.userName}, ${workspace.current_user.short_name}, ${workspace.host}

---

3. Resource Definitions: Jobs & Delta Live Tables Pipelines

Resource configurations define the cloud assets provisioned in Azure Databricks.

Defining a Multi-Task Job (resources/sales_ingest_job.yml)

resources:
  jobs:
    sales_daily_etl_job:
      name: "[${bundle.target}] Daily Sales Processing DAG"
      
      schedule:
        quartz_cron_expression: "0 0 4 * * ?"
        timezone_id: "UTC"
        pause_status: UNPAUSED
      
      email_notifications:
        on_failure:
          - "${var.notification_email}"
      
      job_clusters:
        - job_cluster_key: etl_cluster
          new_cluster:
            spark_version: "15.4.x-photon-scala2.12"
            node_type_id: "${var.worker_node_type}"
            autoscale:
              min_workers: 2
              max_workers: 8
            spark_conf:
              spark.databricks.delta.preview.enabled: "true"
      
      tasks:
        # Task 1: Bronze Ingestion via Python Script
        - task_key: ingest_bronze
          job_cluster_key: etl_cluster
          spark_python_task:
            python_file: ../src/ingest_bronze.py
            parameters:
              - "--catalog"
              - "${var.catalog_name}"
              - "--schema"
              - "${var.schema_name}"
        
        # Task 2: Silver Cleansing (Depends on Bronze Ingestion)
        - task_key: transform_silver
          depends_on:
            - task_key: ingest_bronze
          job_cluster_key: etl_cluster
          notebook_task:
            notebook_path: ../src/notebooks/cleanse_silver.py
            base_parameters:
              catalog: "${var.catalog_name}"
              schema: "${var.schema_name}"

Defining a Delta Live Tables (DLT) Pipeline (resources/telemetry_pipeline.yml)

resources:
  pipelines:
    telemetry_dlt_pipeline:
      name: "[${bundle.target}] Clickstream Telemetry Pipeline"
      catalog: "${var.catalog_name}"
      target: "${var.schema_name}"
      continuous: false
      serverless: true
      libraries:
        - notebook:
            path: ../src/dlt_bronze_silver.py
      configuration:
        pipelines.autoOptimize.zOrderCols: "customer_id,event_timestamp"

---

4. DAB CLI Commands & Lifecycle Execution

The Databricks CLI provides a suite of bundle commands to manage every stage of the lifecycle:

+-------------------------------------------------------------------------+
|                    DAB CLI COMMAND EXECUTION LIFECYCLE                  |
+-------------------------------------------------------------------------+
|                                                                         |
|  1. VALIDATE          2. DEPLOY               3. RUN & MONITOR          |
|  $ databricks bundle  $ databricks bundle     $ databricks bundle run   |
|    validate             deploy -t dev           -t dev sales_daily_job  |
|        |                      |                          |              |
|        v                      v                          v              |
|  Checks syntax,       Uploads files,          Triggers remote run,      |
|  schemas & types      creates Jobs & DLT      streams live output logs  |
+-------------------------------------------------------------------------+

CLI Command Matrix

Validating and Deploying via CLI

# Step 1: Validate bundle syntax for the development target
databricks bundle validate -t dev

# Output confirms valid structure:
# Name: retail_sales_analytics
# Target: dev
# Workspace: https://adb-dev.7.azuredatabricks.net
# Valid: true

# Step 2: Deploy bundle to development workspace
databricks bundle deploy -t dev

# Step 3: Trigger a test run of the daily sales job
databricks bundle run -t dev sales_daily_etl_job

---

5. Deploying Bundle Resources Through the REST API

The DP-750 blueprint lists two bundle deployment bullets: "Deploy a bundle by using the Azure Databricks command-line interface (CLI)" and "Deploy a bundle by using REST APIs." The CLI above covers the first. The second matters whenever the deploying agent is not a shell - an Azure Function, a Logic App, an external orchestrator, or a governance service that must create resources through an authenticated HTTP call.

How the Two Paths Relate

databricks bundle deploy is itself an API client. It resolves the target, interpolates variables, uploads artifacts to the workspace, and then calls the workspace REST APIs to create or update each resource - Jobs, Pipelines, Model Serving endpoints, and so on. Anything the CLI does, an HTTP client can do directly.

Authenticating the Call

Production deployments authenticate as a service principal using an OAuth machine-to-machine token, never a personal access token belonging to an engineer. Section 12.3 covers the run_as identity pattern that pairs with this.

# 1. Exchange the service principal client credentials for an OAuth token
TOKEN=$(curl -s -X POST \
  https://adb-prod.7.azuredatabricks.net/oidc/v1/token \
  -u "$DATABRICKS_CLIENT_ID:$DATABRICKS_CLIENT_SECRET" \
  -d "grant_type=client_credentials&scope=all-apis" | jq -r .access_token)

# 2. Create or update the job the bundle defines
curl -s -X POST \
  https://adb-prod.7.azuredatabricks.net/api/2.2/jobs/create \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  --data @resources/daily_sales_job.json

# 3. Create the declarative pipeline the bundle defines
curl -s -X POST \
  https://adb-prod.7.azuredatabricks.net/api/2.0/pipelines \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  --data @resources/telemetry_pipeline.json

Why the CLI Is Still the Default Answer

Deploying resource-by-resource over HTTP gives up what the bundle abstraction provides: the CLI knows which resources the bundle previously created in that target, so a redeploy updates them and a databricks bundle destroy removes exactly that set. A hand-rolled REST deployment must track those identifiers itself, or it will create a duplicate job on every run.

Use the REST path when the environment genuinely cannot run the CLI. Otherwise, invoke the CLI and let it call the APIs for you.

---

6. Artifact Building & Custom Python Wheels

DAB natively manages Python package compilation. If your project contains custom libraries, DAB can automatically build a .whl package and attach it to job tasks:

# Declare artifact build in databricks.yml
artifacts:
  core_transforms:
    type: whl
    path: .
    build: "python -m build --wheel"

resources:
  jobs:
    wheel_processing_job:
      name: "[${bundle.target}] Wheel Pipeline"
      tasks:
        - task_key: execute_wheel
          job_cluster_key: etl_cluster
          python_wheel_task:
            package_name: "retail_core_lib"
            entry_point: "process_daily_sales"
            parameters:
              - "--batch-date=2026-08-26"
          libraries:
            - whl: ../dist/retail_core_lib-1.0.0-py3-none-any.whl