11.1 Databricks Asset Bundles (DABs) & Infrastructure as Code

Databricks Asset Bundles (DABs) fundamentally shift how data engineering teams manage their Databricks infrastructure, moving from manual UI-based configurations to a robust Infrastructure-as-Code (IaC) methodology. This section explores the structural, operational, and lifecycle aspects of DABs, which are a major focus area for the Databricks Certified Data Engineer Professional exam.

The Anatomy of Databricks Asset Bundles

At its core, a Databricks Asset Bundle is a collection of source files and a centralized configuration file, typically named databricks.yml (or occasionally bundle.yml), that describes how these files should be deployed and executed in a Databricks workspace. By treating Databricks resources (like Jobs, Delta Live Tables pipelines, and Model Serving endpoints) as code, DABs enable version control, automated testing, and CI/CD integration. This paradigm shift from manual click-ops to programmatic deployment ensures that environments can be easily recreated, audited, and rolled back if necessary. The bundle directory structure usually contains the databricks.yml at the root, a src directory for code (like Python scripts, notebooks, or SQL files), a tests directory for unit and integration testing, and a resources directory for modularized YAML definitions if the project is large.

The databricks.yml Structure

The databricks.yml file is the heart of a DAB. It defines the bundle's name, its target environments, the resources it manages, and any workspace variables. Here is a comprehensive example of a databricks.yml file:

bundle:
  name: my_etl_project

variables:
  catalog_name:
    description: "The catalog to use for deployment"
    default: "dev_catalog"

targets:
  dev:
    default: true
    workspace:
      host: https://adb-123456789.1.azuredatabricks.net
    variables:
      catalog_name: "dev_catalog"

  prod:
    workspace:
      host: https://adb-987654321.2.azuredatabricks.net
    variables:
      catalog_name: "prod_catalog"

resources:
  jobs:
    daily_etl:
      name: "[${bundle.target}] Daily ETL Job"
      tasks:
        - task_key: process_data
          notebook_task:
            notebook_path: ./src/process.py
          job_cluster_key: default_cluster
      job_clusters:
        - job_cluster_key: default_cluster
          new_cluster:
            spark_version: 14.3.x-scala2.12
            node_type_id: i3.xlarge
            num_workers: 2

In this example, the bundle dynamically adapts based on the target environment. The ${bundle.target} syntax is a bundle variable substitution that dynamically injects the target name (e.g., dev or prod) into the job name. Furthermore, Databricks bundles can integrate deeply with Unity Catalog by parameterizing the catalog and schema names, ensuring that development jobs read and write to development catalogs, completely isolated from production data.

Target Environment Definitions

A critical feature of DABs is the ability to define multiple targets. A target maps a logical environment (like dev, staging, or prod) to a specific Databricks workspace and configuration set. In larger enterprise architectures, each target typically corresponds to an entirely separate Databricks workspace to ensure the blast radius of any misconfiguration is kept strictly to non-production environments.

Environment Isolation

By switching targets, data engineers can deploy the exact same codebase to different workspaces without modifying the underlying source files, eliminating the "it works on my machine" problem. When the bundle is deployed to the prod target, the Databricks CLI natively uses the prod configuration, including specific service principal credentials, avoiding accidental cross-contamination.

Declaring Resources

DABs support the declaration of various Databricks resources. The most common resources declared in a databricks.yml file are:

1. Jobs: Scheduled or triggered workflows consisting of multiple tasks (notebooks, Python wheels, SQL scripts). These definitions include cluster sizing, retry policies, and timeout constraints.
2. Pipelines: Delta Live Tables (DLT) pipelines, specifying the edition (core, pro, advanced) and the path to the pipeline source code, as well as target schemas.
3. Artifacts: Compiled code (like Python .whl files) that need to be built and uploaded to the workspace before execution.
4. Model Serving Endpoints: Definitions for serving MLflow models as REST APIs.

Example: Declaring a DLT Pipeline

resources:
  pipelines:
    my_dlt_pipeline:
      name: "${bundle.target}_dlt_pipeline"
      development: true # Set to false in prod target
      clusters:
        - label: "default"
          num_workers: 1
      libraries:
        - notebook:
            path: ./src/dlt_pipeline.sql

When declaring DLT pipelines, the development flag is crucial. In a dev target, setting this to true ensures that clusters are reused across pipeline updates, significantly accelerating the development cycle. In a prod target, this must be overridden to false so the pipeline runs in production mode, spinning up fresh, isolated clusters for execution and terminating them upon completion.

The Bundle Lifecycle Commands

The Databricks CLI provides a suite of commands to manage the bundle lifecycle. Understanding these commands and their order of operations is heavily tested on the professional exam.

1. databricks bundle validate

Before deploying, you must ensure your configuration is syntactically correct. The databricks bundle validate command parses the databricks.yml file, resolves variable substitutions, and checks for schema violations. Crucially, this command does not communicate with the Databricks workspace to create or modify resources. It is purely a local validation step, making it ideal for the early stages of a CI/CD pipeline. By running validation on every commit, teams can catch syntax errors long before they attempt to deploy broken configurations.

2. databricks bundle deploy

The databricks bundle deploy command synchronizes your local project state with the target Databricks workspace. Under the hood, this command performs several critical actions:

• It builds any declared artifacts (e.g., compiling a Python wheel using standard setuptools or poetry).
• It uploads the local files (notebooks, SQL scripts, built artifacts) to the Databricks workspace (typically into the workspace files or DBFS depending on the configuration).
• It calls the Databricks REST API to create or update the declared resources (Jobs, DLT pipelines) to match the state defined in the bundle.
• It applies any tag or permission changes specified in the YAML configuration.

By default, if no target is specified via the -t flag, it deploys to the target marked with default: true.

3. databricks bundle run

Once a bundle is deployed, you can trigger its resources using databricks bundle run. This command requires you to specify the resource key you wish to execute.

# Runs the daily_etl job in the default (dev) target
databricks bundle run daily_etl

# Runs the daily_etl job in the prod target
databricks bundle run daily_etl -t prod

This command submits a run request to the Databricks Jobs or DLT API and streams the output/logs back to your local terminal, allowing you to monitor execution without switching to the Databricks UI. It is highly beneficial for executing integration tests as part of a CD pipeline.

Practical Scenario: DRY Configurations and Overrides

Imagine you have a single data processing pipeline that needs to run in development with a small cluster and in production with a massive cluster. Instead of writing two separate JSON definitions, DABs allow you to handle this gracefully using variables and target overrides.

variables:
  worker_count:
    default: 2

targets:
  prod:
    variables:
      worker_count: 10

resources:
  jobs:
    scalable_job:
      job_clusters:
        - job_cluster_key: main
          new_cluster:
            num_workers: ${var.worker_count}

When deployed to dev, the job is provisioned with 2 workers. When deployed to prod, the override takes effect, and the job is provisioned with 10 workers. This variable injection is a powerful feature that ensures parity between environments while accommodating necessary scaling differences. Another common pattern is using bundle variables to swap out instance profiles or IAM roles depending on the environment, allowing a dev job to read from a sandbox S3 bucket while the prod job reads from the secured, production S3 bucket.