12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides

12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides

Enterprise lakehouse architectures require rigorous isolation between Development, Staging (User Acceptance Testing / Pre-Production), and Production environments. Code executing in development must never write to production Delta tables, consume production compute budgets, or trigger active alerting webhooks.

Databricks Asset Bundles (DAB) solves this through its Target and Environment Model. By defining target blocks and applying target overrides in databricks.yml, data engineers maintain a single source of truth while ensuring environment-specific configurations (such as catalogs, cluster sizes, schedules, and permissions) are applied deterministically during deployment.

---

1. Development vs. Production Deployment Modes

The mode directive inside a DAB target fundamentally alters how Databricks builds, names, isolates, and executes deployed resources:

targets:
  dev:
    mode: development    # Developer isolation mode
    default: true
  
  prod:
    mode: production     # Strict enterprise production mode

+-------------------------------------------------------------------------+
|                 DEVELOPMENT MODE VS. PRODUCTION MODE                    |
+-------------------------------------------------------------------------+
|  BEHAVIOR / SETTING       | mode: development     | mode: production    |
|---------------------------|-----------------------|---------------------|
|  Resource Name Prefix     | [dev ${user.short}]   | None (Clean name)   |
|  Deployment Path          | /Users/${user}/...    | /Shared/.bundle/... |
|  Job Schedules & Triggers | PAUSED (Automatic)    | UNPAUSED (Active)   |
|  DLT Continuous Pipelines | PAUSED (Run on-demand)| CONTINUOUS (Active) |
|  Git Working Tree Check   | Allows uncommitted    | FAILS if dirty      |
|  Resource Ownership       | Deploying Developer   | Service Principal   |
+-------------------------------------------------------------------------+

1. mode: development Deep-Dive

When an engineer runs databricks bundle deploy -t dev:

• Resource Name Mangling: If a job is defined as name: Daily Ingestion, DAB renames it to [dev alice] Daily Ingestion. This prevents collision when multiple developers deploy the same bundle in a shared development workspace.
• Automatic Schedule Pausing: Scheduled cron triggers and continuous DLT pipelines are automatically set to PAUSED to avoid unexpected compute costs and rogue background executions during development.
• Path Isolation: Files are deployed to /Users/alice@corp.com/.bundle/retail_sales/dev/files/.
• Dirty Git Tolerance: Developers can deploy local edits and uncommitted branch code to iterate quickly.

2. mode: production Deep-Dive

When a release pipeline runs databricks bundle deploy -t prod:

• Clean Git State Enforcement: DAB verifies that the local Git working tree is clean and matches the remote commit. If uncommitted local changes exist, deployment aborts immediately to prevent unversioned artifacts in production.
• Standard Naming & Shared Paths: Resource names are preserved without user prefixes, and files deploy to /Shared/.bundle/${bundle.name}/prod/files/.
• Active Scheduling: Job schedules and continuous pipelines are deployed in their active, unpaused operational states.

---

2. Multi-Target Configuration & Override Mechanics

Target overrides allow teams to customize any property of a bundle (workspace URL, variables, cluster specifications, notification emails, and permissions) based on the target environment.

Comprehensive databricks.yml Multi-Target Manifest

bundle:
  name: lakehouse_sales_pipeline

include:
  - resources/*.yml

# Global variable definitions
variables:
  catalog:
    description: "Unity Catalog name"
    default: "dev_catalog"
  schema:
    description: "Target schema name"
    default: "sales_raw"
  cluster_node_type:
    description: "Worker VM size"
    default: "Standard_D4ds_v5"
  min_workers:
    description: "Minimum auto-scaling workers"
    default: 1
  max_workers:
    description: "Maximum auto-scaling workers"
    default: 2

targets:
  # 1. Local Developer Target (Default)
  dev:
    mode: development
    default: true
    workspace:
      host: https://adb-dev-workspace.7.azuredatabricks.net
    variables:
      catalog: dev_catalog
      min_workers: 1
      max_workers: 2

  # 2. Staging / UAT Environment
  staging:
    mode: production
    workspace:
      host: https://adb-staging-workspace.7.azuredatabricks.net
      root_path: /Shared/.bundle/${bundle.name}/${bundle.target}
    variables:
      catalog: staging_catalog
      cluster_node_type: Standard_D8ds_v5
      min_workers: 2
      max_workers: 4

  # 3. Production Environment
  prod:
    mode: production
    workspace:
      host: https://adb-prod-workspace.7.azuredatabricks.net
      root_path: /Shared/.bundle/${bundle.name}/${bundle.target}
    run_as:
      service_principal_name: 11111111-2222-3333-4444-555555555555
    variables:
      catalog: prod_catalog
      schema: sales_core
      cluster_node_type: Standard_E8ds_v5  # Memory-optimized for production Photon
      min_workers: 4
      max_workers: 16
    permissions:
      - level: CAN_MANAGE
        group_name: data_engineering_leads
      - level: CAN_VIEW
        group_name: bi_analysts

---

3. Service Principal Deployment & the run_as Identity Pattern

In production, data pipelines must never run under the identity of an individual developer. If an employee leaves the company or their Entra ID account is deactivated, all jobs owned by that user fail immediately.

The run_as Architecture

DAB supports the run_as block to specify the execution identity of deployed workflows:

targets:
  prod:
    mode: production
    run_as:
      service_principal_name: 11111111-2222-3333-4444-555555555555

+-------------------------------------------------------------------------+
|               SERVICE PRINCIPAL RUN_AS EXECUTION PATTERN                |
+-------------------------------------------------------------------------+
|                                                                         |
|  CI/CD Runner (Azure DevOps / GitHub Actions)                           |
|  ├── Authenticates via Entra Service Principal Secret / OIDC            |
|  └── Executes: databricks bundle deploy -t prod                         |
|                                                                         |
|  Azure Databricks Production Workspace                                  |
|  ├── Deploys multi-task Lakeflow Job with Owner: Service Principal      |
|  ├── Executes Job under Service Principal Unity Catalog Privileges      |
|  │   (SELECT / MODIFY on prod_catalog.sales_core.*)                     |
|  └── Result: Zero dependency on individual developer credentials       |
+-------------------------------------------------------------------------+

Requirements for run_as Service Principal Deployment:

1. The Service Principal must exist in Microsoft Entra ID and be synchronized into the Azure Databricks Account Console via SCIM.
2. The Service Principal must be granted workspace access and assigned the Workspace Access entitlement.
3. The CI/CD agent or deploying identity must possess the Service Principal User role to deploy assets on behalf of that Service Principal.
4. The Service Principal must hold appropriate Unity Catalog permissions (USE CATALOG, USE SCHEMA, CREATE TABLE, MODIFY, SELECT) on the target production catalog and schemas.

---

4. Fine-Grained Resource Property Overrides

In addition to global variables, DAB allows overriding specific attributes of individual resources per target directly inside the target block or using target-specific resource schemas:

# File: databricks.yml
resources:
  jobs:
    sales_job:
      name: "Sales Processing Pipeline"
      tasks:
        - task_key: process_data
          job_cluster_key: main_cluster
          notebook_task:
            notebook_path: ../src/process.py

targets:
  dev:
    mode: development
    resources:
      jobs:
        sales_job:
          schedule:
            pause_status: PAUSED
  
  prod:
    mode: production
    resources:
      jobs:
        sales_job:
          schedule:
            pause_status: UNPAUSED
            quartz_cron_expression: "0 0 2 * * ?"
          email_notifications:
            on_failure:
              - "pagerduty-prod@corp.com"

---

5. Multi-Workspace Isolation vs. Single-Workspace Namespace Isolation

Organizations adopt two primary architectural patterns for environment separation in Azure Databricks: