Databricks Repos, Secrets, and Development Tools

Quick Answer: Databricks provides Git folders for version control, a secrets utility for managing credentials, a CLI for automation, a REST API for programmatic access, and notebook widgets for parameterization. These tools support professional software engineering practices in data engineering.

Git Folders (Repos)

Git folders integrate Git version control directly into the Databricks workspace:

Supported Git Providers

• GitHub / GitHub Enterprise
• GitLab / GitLab Self-Managed
• Bitbucket Cloud / Bitbucket Server
• Azure DevOps

Key Capabilities

• Clone remote repositories into your workspace
• Branch, commit, push, pull from the Databricks UI
• Sync notebooks and code files bidirectionally
• File types: .py, .sql, .scala, .r, .ipynb, and Databricks notebooks
• Best practice: Use Git folders for all production code; use the workspace for ad-hoc exploration

Git Folder Workflow

1. Clone a remote repo into Databricks workspace
2. Create a feature branch
3. Develop and test in notebooks/files
4. Commit and push changes to remote
5. Create a pull request for code review
6. Merge to main branch

Databricks Secrets

Secrets provide secure storage for sensitive credentials:

Using dbutils.secrets

# List available secret scopes
dbutils.secrets.listScopes()

# List secrets in a scope
dbutils.secrets.list("my-scope")

# Get a secret value (displayed as [REDACTED] in notebook output)
password = dbutils.secrets.get(scope="my-scope", key="db-password")

Key Points About Secrets

• Secret values are redacted in notebook output — they display as [REDACTED]
• Secrets are stored in a secret scope (backed by Databricks or an external vault like Azure Key Vault)
• Access control can restrict which users/groups can read specific secrets
• Secrets are ideal for database connection strings, API keys, and storage credentials

On the Exam: Remember that dbutils.secrets.get() returns the actual secret value for use in code, but the value is automatically redacted when displayed in notebook cell output. This prevents accidental exposure.

Databricks CLI

The Databricks CLI provides command-line access to the workspace:

# Configure authentication
databricks configure --token

# List workspaces resources
databricks workspace list /

# Deploy a job
databricks jobs create --json @job-config.json

# Run a job
databricks jobs run-now --job-id 123

# Bundle commands (for Databricks Asset Bundles)
databricks bundle validate
databricks bundle deploy
databricks bundle run

Notebook Widgets

Widgets parameterize notebooks for reusable, interactive workflows:

# Create a text widget
dbutils.widgets.text("start_date", "2026-01-01", "Start Date")

# Create a dropdown widget
dbutils.widgets.dropdown("environment", "dev", ["dev", "staging", "prod"])

# Get widget value
start_date = dbutils.widgets.get("start_date")

# Remove a widget
dbutils.widgets.remove("start_date")

# Remove all widgets
dbutils.widgets.removeAll()

-- Using widget values in SQL
SELECT * FROM sales
WHERE order_date >= getArgument('start_date')

Widget Types

On the Exam: Widgets are important for creating parameterized notebooks that can be called from jobs with different input values. The %run command also accepts widget parameters.