11.2 CI/CD Automation & Databricks CLI / REST API

Continuous Integration and Continuous Deployment (CI/CD) is mandatory for enterprise-grade Data Engineering on Databricks. Moving code from a git repository to a production workspace requires headless automation, secure authentication, and robust programmatic interfaces. This section explores how the Databricks CLI v0.200+, the REST API, and CI/CD tools interact to achieve this.

The Modern Databricks CLI (v0.200+)

Databricks completely overhauled its CLI with version 0.200. The legacy CLI was written in Python and often suffered from virtual environment conflicts and limited authentication mechanisms. The modern CLI is a compiled Go binary, making it extremely fast, easy to distribute, and deeply integrated with Databricks Asset Bundles (DABs). Because it is distributed as a single static binary, installing it on CI runners like GitHub Actions or Azure DevOps agents is a trivial process—often just a matter of executing a simple curl command to download the executable.

Authentication for Automation

In a CI/CD environment, human interaction is impossible. Therefore, the CLI must authenticate securely in a headless manner. The professional exam tests your understanding of authentication best practices.

1. Personal Access Tokens (PATs):

   • How it works: A token tied to a specific user identity (e.g., a data engineer's user account).
   • Drawback: If the user leaves the company, the token is revoked, breaking the production pipeline. They are also subject to short-term expiration policies in secure enterprises, requiring constant manual rotation. Furthermore, operations performed by the pipeline appear in audit logs as being performed by the human user, obscuring accountability.
   • Verdict: Suitable for local development, strongly discouraged for CI/CD.

2. OAuth Machine-to-Machine (M2M) with Service Principals:

   • How it works: A Service Principal is an identity created specifically for automated tools and applications. It is not tied to a human user and has its own isolated lifecycle.
   • Implementation: The CI/CD pipeline uses a Client ID and a Client Secret to securely request a temporary, short-lived OAuth token from Databricks.
   • Verdict: The industry standard and best practice for all production CI/CD pipelines.

Environment Variable Injection

To configure the Databricks CLI in a CI/CD runner (like GitHub Actions, GitLab CI, or Azure DevOps), you do not run databricks configure. Instead, you inject secure environment variables into the runner's context. The CLI automatically detects these variables and uses them for subsequent commands.

For OAuth M2M authentication, you must set:

• DATABRICKS_HOST: The URL of the target workspace.
• DATABRICKS_CLIENT_ID: The UUID of the Service Principal.
• DATABRICKS_CLIENT_SECRET: The secure secret for the Service Principal.

When these are present in the runner's environment, commands like databricks bundle deploy authenticate silently and securely.

Databricks REST API 2.1

While the CLI and DABs are powerful, they ultimately act as wrappers around the Databricks REST API 2.1. In scenarios where you need granular control, are integrating Databricks with a custom orchestrator (like Apache Airflow), or are building a custom application, interacting directly with the API is required.

Core API Endpoints

Data engineers must be familiar with the following critical API families:

Example: Triggering a Job via REST API

When a CI/CD pipeline finishes deploying code, it often triggers an integration test job. Here is how that looks using a raw curl command against the REST API, utilizing a Service Principal token:

curl -X POST https://adb-123456789.1.azuredatabricks.net/api/2.1/jobs/run-now \
  -H "Authorization: Bearer $OAUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "job_id": 112233,
    "job_parameters": {
      "environment": "staging",
      "commit_hash": "8f9a2b1"
    }
  }'

This API call triggers a run of job 112233 and dynamically passes parameters. The CI runner can then use the returned run_id to poll the Jobs API, waiting for the job to succeed or fail.

Parameter Substitution in CI/CD

The example above highlights a crucial pattern: parameter substitution. When a CI/CD pipeline triggers a Databricks Job, it frequently passes runtime context (like the Git commit hash, the branch name, or a pipeline run ID) as job parameters.

Inside Databricks, if this job is a Python wheel task, these parameters are passed as sys.argv. If it is a notebook task, they are accessed using Databricks Utilities:

# Inside a Databricks Notebook Task
env = dbutils.widgets.get("environment")
commit = dbutils.widgets.get("commit_hash")

print(f"Running integration tests for {env} on commit {commit}")

By passing the commit hash, data engineers can trace exactly which version of the codebase produced a specific output or model.

Designing a CI/CD Pipeline

A standard robust CI/CD pipeline for Databricks follows these discrete stages:

1. Continuous Integration (CI):
   • Trigger: A pull request is opened against the main branch.
   • Action: The CI runner checks out the code, runs static analysis (linting, black formatting), runs unit tests (using pytest locally on the runner), and executes databricks bundle validate to ensure IaC correctness. The runner might also check for hardcoded secrets or credentials.
2. Continuous Deployment (CD) - Staging:
   • Trigger: The pull request is merged into main.
   • Action: The runner authenticates via Service Principal, runs databricks bundle deploy -t staging, and triggers a staging integration job via the CLI (databricks bundle run). If the integration job fails, the pipeline fails, alerting the team before code reaches production.
3. Continuous Deployment (CD) - Production:
   • Trigger: A release tag (e.g., v1.2.0) is pushed, or a manual approval gate is cleared.
   • Action: The runner authenticates with a different production Service Principal, deploys artifacts using databricks bundle deploy -t prod, and updates the production job schedules.

By keeping execution environments separate and strictly enforcing Service Principal authentication, organizations ensure their Databricks platform remains secure, stable, and highly automated. The separation of concerns between code definition in Git and actual environment execution ensures that testing is rigorous and deployment is reliable.