1.1 MLOps Best Practices on Databricks
Key Takeaways
- The exam's MLOps objective is about process discipline, not tooling trivia: version everything, isolate environments, automate promotion, and keep a human approval gate on production models.
- A Databricks MLOps stack separates dev, staging, and production into distinct workspaces and Unity Catalog catalogs, with production write access reserved for service principals driven by CI/CD.
- Databricks Git Folders version notebooks and Python modules; Databricks Asset Bundles (DABs) version the jobs, pipelines, and endpoints that run them.
- Reproducibility requires four artifacts pinned together: the code commit, the environment (`requirements.txt` or the runtime version), the training data snapshot (Delta version), and the MLflow run.
- Automated retraining is only a best practice when it is paired with automated validation — a scheduled job that registers an unvalidated model version is a governance failure, not maturity.
1.1 MLOps Best Practices on Databricks
Modern machine learning systems require rigorous engineering practices to bridge the gap between ad-hoc experimentation and resilient, production-grade deployments. On the Databricks Lakehouse, Machine Learning Operations (MLOps) leverages unified storage (Delta Lake), centralized data and AI governance (Unity Catalog), integrated experiment tracking (MLflow), and optimized compute environments (Databricks Runtime for Machine Learning) to build reproducible, governed, and automated ML lifecycles.
+---------------------------------------------------------------------------------------------------+
| DATABRICKS LAKEHOUSE MLOps PLATFORM |
| |
| +---------------------+ +---------------------+ +---------------------+ +----------------+ |
| | Data Preparation | | Feature Engineering | | Experiment Tracking | | Model Registry | |
| | (Delta Lake / |-->| (Unity Catalog |-->| (MLflow |-->| (Unity Catalog | |
| | Structured Stream) | | Feature Tables) | | Runs & Metrics) | | Models/Aliases)| |
| +---------------------+ +---------------------+ +---------------------+ +----------------+ |
| | |
| v |
| +---------------------+ +---------------------+ +---------------------+ +----------------+ |
| | Model Monitoring |<--| Model Serving |<--| Automated CI/CD |<--| Model Validation | |
| | (Lakehouse Ingest) | | (Serverless Serving)| | (DABs / Git) | | & Staging | |
| +---------------------+ +---------------------+ +---------------------+ +----------------+ |
+---------------------------------------------------------------------------------------------------+
MLOps Lifecycle and Multi-Environment Architecture
A production-ready MLOps framework on Databricks isolates risk by segregating responsibilities across dedicated workspaces and storage containers. This separation ensures that experimental modifications, exploratory data queries, and unvalidated library installations cannot degrade critical production inference pipelines.
Multi-Workspace Isolation Pattern
Enterprise architectures typically establish three distinct tiers of workspaces, each bound to specific Unity Catalog catalogs and compute policies:
-
Development Workspace (
dev):- Purpose: Interactive data exploration, rapid prototyping, feature creation, and model architecture tuning.
- Access & Permissions: Data scientists maintain broad permissions (
CAN EDIT,USE SCHEMA) to create temporary schemas, test custom libraries, and execute ad-hoc notebooks. - Data Governance: Reads from anonymized or obfuscated development schemas (
dev_catalog.gold_features). Raw customer PII is masked or excluded.
-
Staging Workspace (
staging/test):- Purpose: Automated integration testing, candidate model evaluation, pre-production load testing, and shadow deployments.
- Access & Permissions: Heavily restricted interactive access. Workflows and tests are executed primarily by service principals through Continuous Integration (CI) runners (such as GitHub Actions, GitLab CI, or Azure DevOps).
- Validation: Candidate models trained in
devor automated staging pipelines run against pre-release integration tests, latency benchmarks, and schema validation suites.
-
Production Workspace (
prod):- Purpose: Mission-critical scheduled retraining pipelines, automated feature table updates, and live inference endpoints (batch, streaming, and real-time REST serving).
- Access & Permissions: Zero interactive write access for standard user accounts. All infrastructure, jobs, endpoints, and code deployments are provisioned via Infrastructure as Code (Terraform) and Databricks Asset Bundles (DABs) driven by Continuous Deployment (CD) service principals.
- Data Governance: Full access to production Delta tables (
prod_catalog.gold_features) under strict Unity Catalog access control audits.
+-----------------------+ Git PR Merge +-------------------------+ Automated Release +-----------------------+
| DEVELOPMENT WORKSPACE | --------------------> | STAGING WORKSPACE | ------------------------> | PRODUCTION WORKSPACE |
| - Interactive EDA | | - Automated CI Testing | | - Retraining Jobs |
| - Ad-hoc Notebooks | | - Performance Benchmarks| | - Serverless Serving |
| - `dev_catalog` | | - `staging_catalog` | | - `prod_catalog` |
+-----------------------+ +-------------------------+ +-----------------------+
Code Versioning with Databricks Git Folders (Repos)
Reproducibility in machine learning requires versioning not only the training data and model artifacts, but also the exact source code and environment definitions. Databricks Git Folders (formerly Databricks Repos) integrates the workspace directly with Git providers (GitHub, GitLab, Bitbucket, Azure DevOps):
- Branch Isolation: Data scientists work on isolated feature branches (
feature/xgboost-churn-v2), pulling changes and committing code directly from the Databricks notebook UI or via the Databricks CLI. - CI/CD Integration: Merging a Pull Request into the
mainbranch triggers automated CI/CD webhooks that synchronize the production Git Folder and execute deployment pipelines using the Databricks REST API. - Modular Codebases: Git Folders support full repository structures, enabling developers to import reusable Python modules (
from src.utils.preprocessing import clean_text), test suites withpytest, and environment dependency specifications (requirements.txt).
What "Best Practice" Means on This Exam
Exam items on MLOps strategy are almost always scenario questions: a team describes how they work today, and you pick the change that most improves reliability or governance. Four principles decide nearly all of them.
Version the whole recipe, not just the model
A model version is reproducible only when four things are recoverable together:
| Artifact | Where it lives on Databricks | Failure if unversioned |
|---|---|---|
| Source code | Databricks Git Folder commit SHA (auto-logged as an MLflow tag) | Cannot rebuild the training script that produced the weights |
| Environment | requirements.txt in the repo, or the pinned DBR ML version | Library drift silently changes predictions |
| Training data | Delta Lake table version / timestamp (VERSION AS OF) | Cannot reproduce metrics; audit fails |
| Run metadata | MLflow run (params, metrics, artifacts, model signature) | Cannot compare candidates or trace a production prediction |
Isolate environments, and let permissions do the enforcing
Segregation is enforced by Unity Catalog privileges and cluster policies, not by
convention. If data scientists can write to prod_catalog, the environments are not
actually isolated no matter how the folders are named.
Automate promotion, gate it on validation
The mature pattern is: CI runs unit and integration tests on every pull request; a
merge to main triggers a deployment bundle; a scheduled retraining job produces a
candidate model version; an automated validation job compares the candidate
against the current champion on a frozen holdout set and on fairness/latency checks;
only a passing candidate receives the production alias. Scheduling retraining without
that validation step is the most commonly tested anti-pattern.
Monitor after deployment, and close the loop
Inference tables capture request and response payloads; Lakehouse Monitoring compares serving feature distributions against the training baseline. Drift alerts should trigger the same automated retraining and validation path, not a manual notebook run.
Exam tip: when an option describes moving a model binary between workspaces by hand, or promoting a model that no automated job ever evaluated, it is the wrong answer. Section 1.3 covers the related choice of promoting code versus promoting models.
In an enterprise Databricks Lakehouse MLOps architecture, what is the standard practice for promoting code and models from the Staging workspace to the Production workspace?
A team retrains a churn model nightly with a Databricks Workflow. The job trains on the latest Delta snapshot, registers a new model version, and immediately moves the production alias to it. Predictions have quietly degraded twice this quarter. Which change most directly follows Databricks MLOps best practice?
Which set of artifacts must be recorded together for a Databricks training run to be genuinely reproducible?