12.1 Databricks Git Folders & Source Control Integration

12.1 Databricks Git Folders & Source Control Integration

In modern lakehouse engineering, data assets—including data pipelines, transformation notebooks, modular Python libraries, and workflow definitions—must adhere to strict software development lifecycle (SDLC) standards. Early cloud notebook environments suffered from code isolation, manual copy-pasting, lack of version history, and untracked changes in production.

Databricks Git Folders (formerly known as Databricks Repos) bridges this gap by embedding full Git version control directly into the Azure Databricks workspace. Git Folders allows data engineers to develop code collaboratively, maintain branch isolation, perform code reviews via pull requests (PRs), import modular helper packages, and trigger automated continuous integration and continuous deployment (CI/CD) pipelines.

---

1. Git Folders Architecture & Workspace File Integration

Databricks Git Folders is not merely a notebook viewer; it is a full local Git clone hosted inside the Databricks workspace control plane and synchronized with your remote Git repository.

+-------------------------------------------------------------------------+
|                    DATABRICKS GIT FOLDERS ARCHITECTURE                  |
+-------------------------------------------------------------------------+
|                                                                         |
|  REMOTE GIT PROVIDER               DATABRICKS WORKSPACE                 |
|  +--------------------+            +----------------------------------+ |
|  | Azure DevOps       |            | /Workspace/Users/alice@corp.com/ | |
|  | GitHub Enterprise  | <---HTTPS---|   └── feature-sales-pipeline/    | |
|  | GitLab / Bitbucket |    Sync    |       ├── config.yaml            | |
|  +--------------------+            |       ├── main_etl.py            | |
|                                    |       ├── transforms/            | |
|                                    |       │   └── cleanse.py         | |
|                                    |       └── tests/                 | |
|                                    |           └── test_cleanse.py    | |
|                                    +----------------------------------+ |
+-------------------------------------------------------------------------+

Workspace Files & Modular Python Imports

Historically, Databricks notebooks could only reference other notebooks via the legacy %run magic command. Git Folders introduces Arbitrary Workspace Files, allowing non-notebook files (.py, .sql, .yaml, .json, .csv, .sh, .txt) to reside alongside notebooks.

This enables standard Python software engineering patterns within notebooks and jobs:

# File: /Workspace/Users/alice@corp.com/feature-sales-pipeline/main_etl.py
import sys
import yaml
from transforms.cleanse import deduplicate_transactions, mask_pii_columns
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

# Load local YAML configuration file directly from Git Folder
with open("config.yaml", "r") as f:
    config = yaml.safe_load(f)

# Ingest raw Bronze Delta table
raw_df = spark.table(f"{config['catalog']}.bronze.raw_transactions")

# Apply modular business transformation functions
clean_df = deduplicate_transactions(raw_df, key_col="transaction_id")
silver_df = mask_pii_columns(clean_df, cols=["credit_card", "ssn"])

# Persist to Silver layer
silver_df.write.format("delta").mode("append").saveAsTable(f"{config['catalog']}.silver.transactions")

When a notebook or Python script executes within a Git Folder, Databricks automatically adds the root directory of the Git Folder to the Python sys.path. Consequently, custom packages and relative module imports function seamlessly without requiring manual sys.path.append() statements or wheel installations.

---

2. Git Provider Authentication Mechanisms

To synchronize code between Azure Databricks and a remote Git repository, each engineer or service principal must configure Git credentials.

+-------------------------------------------------------------------------+
|               AUTHENTICATION WORKFLOW: AZURE DEVOPS & ENTRA ID          |
+-------------------------------------------------------------------------+
|  1. Developer logs into Azure Databricks via Entra ID (Azure AD).       |
|  2. In User Settings > Git Integration, select 'Azure DevOps Services'. |
|  3. Select 'Azure DevOps OAuth' (or provide PAT with Code Read/Write).  |
|  4. Databricks securely requests an OAuth token from Entra ID.          |
|  5. Workspace clones/pushes to https://dev.azure.com/org/project/_git/  |
+-------------------------------------------------------------------------+

Exam Tip: For Azure DevOps repositories residing in the same Microsoft Entra ID tenant as Azure Databricks, Microsoft recommends using native Entra ID OAuth integration. This eliminates secret sprawl and avoids pipeline disruptions caused by expired Personal Access Tokens.

---

3. Branching Strategies & Lakehouse Lifecycle Management

Enterprise data teams implement disciplined branching workflows to prevent untested code from polluting staging and production lakehouses.

+-------------------------------------------------------------------------+
|                   TRUNK-BASED LAKEHOUSE BRANCHING MODEL                 |
+-------------------------------------------------------------------------+
|                                                                         |
|  [main branch] ====================================================>    |
|       \                                                   / (PR Merge)  |
|        \--> [feat/ingest-telemetry] (Alice's Git Folder) -/             |
|             * Local testing against dev catalog                         |
|             * Automated PR checks (pytest, linting)                     |
|                                                                         |
+-------------------------------------------------------------------------+

Comparison: Git Flow vs. Trunk-Based Development

Developer vs. Production Git Folder Topologies

1. Developer Workspace Path (/Workspace/Users/<email>/):
   • Each data engineer clones the repository into their personal workspace folder.
   • Engineers switch branches, modify files, run interactive notebook cells against development compute, commit changes, and push to the remote repository.
   • Isolation: Changes made by Engineer A never affect Engineer B.
2. Shared Production Workspace Path (/Workspace/Repos/Production/ or DAB-managed):
   • Production workspaces should never be manually edited by individual developers.
   • Production Git Folders are maintained on the main or release tag branch and updated exclusively via automated CI/CD service principals.

---

4. Programmatic Management with the Repos REST API

Azure Databricks provides the Repos REST API (/api/2.0/repos) to allow automated CI/CD pipelines, orchestrators, and release agents to manage Git Folders programmatically.

Core Repos REST API Endpoints

Automating Deployment via REST API

When a Pull Request is approved and merged into main, a CI/CD pipeline (such as Azure DevOps Pipelines or GitHub Actions) executes a PATCH request to pull the latest code into the staging/production Git Folder:

# Triggered by CI/CD Runner authenticated with Databricks Service Principal Token
curl -X PATCH \
  https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/repos/987654321098765 \
  -H "Authorization: Bearer ${DATABRICKS_SP_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "branch": "main"
  }'

JSON Response:

{
  "head_commit_id": "e4d9b2a1c0f8e7d6a5b4c3d2e1f0a9b8c7d6e5f4",
  "id": 987654321098765,
  "path": "/Repos/Production/lakehouse-etl",
  "url": "https://dev.azure.com/myorg/dataplat/_git/lakehouse-etl",
  "provider": "azureDevOpsServices",
  "branch": "main"
}

Checking Out Specific Release Tags

For production releases requiring strict immutability, the CI/CD pipeline can update the Git Folder to a specific immutable release tag rather than a floating branch:

curl -X PATCH \
  https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/repos/987654321098765 \
  -H "Authorization: Bearer ${DATABRICKS_SP_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "tag": "v2.4.0"
  }'

---

5. Storage Layer Comparison & Operational Constraints

Understanding where code and data reside in Azure Databricks is critical for architecture design and exam questions:

Git Folders Constraints & Limitations

• File Size Limits: Individual non-notebook files inside Git Folders cannot exceed 500 MB.
• Git Submodules: Databricks Git Folders does not support Git submodules.
• Git LFS (Large File Storage): Git LFS pointers are not resolved; store large binary files in Unity Catalog Volumes instead.
• Merge Conflicts: Merge conflicts cannot be resolved in the Databricks web UI; developers must resolve conflicts locally using their IDE/Git CLI and push resolved commits.