1.3 Promoting Code vs. Promoting Models
Key Takeaways
- Deploy-code promotes the *training pipeline* through dev, staging, and production; the production model is trained in production on production data.
- Deploy-model promotes a *trained model artifact* built in a lower environment; production only scores with it.
- Databricks recommends deploy-code as the default because the model that serves traffic was trained on the production data it will actually see, and every environment runs identical code.
- Deploy-model is the right choice when production data cannot be re-read for training, when training is far too expensive to repeat, or when a strict change-control regime requires the exact evaluated binary to be the one deployed.
- Both patterns still register the model in Unity Catalog and use aliases for promotion — the difference is *where training runs*, not whether MLflow is involved.
1.3 Promoting Code vs. Promoting Models
One of the more conceptual objectives on this exam asks you to identify when promoting code beats promoting models and vice versa. The two patterns describe the same lifecycle — train, validate, register, serve — but they disagree about which environment does the training.
+---------------------------------------------------------------------------------+
| DEPLOY CODE (recommended default) |
| |
| dev repo ---PR---> staging repo ---release---> PRODUCTION repo |
| | |
| v |
| production training job runs on PROD data |
| | |
| v |
| model registered in prod catalog, aliased |
+---------------------------------------------------------------------------------+
| DEPLOY MODELS |
| |
| dev/staging training job runs on DEV data ---> model artifact |
| | |
| v |
| artifact promoted to prod registry, aliased |
| production only ever SCORES |
+---------------------------------------------------------------------------------+
Deploy Code: the Recommended Default
In the deploy-code pattern the versioned, reviewed, tested asset is the training pipeline — feature computation, training script, validation job, and the Databricks Asset Bundle that schedules them. Code is promoted by merging a pull request and releasing a bundle. The production workspace then executes that pipeline against production tables and registers the resulting model version.
Why Databricks recommends it:
- The model sees production data. Development catalogs are typically sampled, masked, or stale. A model trained there can be systematically miscalibrated on the real distribution.
- The whole pipeline is tested. Unit tests, integration tests, and data-quality expectations run against the same code that will produce the production model, so feature-engineering bugs are caught before they ship.
- Retraining is free. Because the pipeline already lives in production, scheduled retraining on fresh data needs no new promotion step at all.
- PII never leaves its boundary. Training happens where the sensitive data already resides, which is often a hard regulatory requirement.
Deploy Models: When the Artifact Must Move
Here the training job runs in dev or staging and the trained model is the object promoted into the production registry. Production compute never trains; it loads the registered version and scores.
Choose deploy-model when:
| Scenario | Why the artifact must move |
|---|---|
| Production data cannot be read by a training job (regulatory ring-fencing, or the production workspace has no training compute) | Training cannot physically happen in production |
| Training costs are extreme — multi-day GPU fine-tuning, or a model trained once on an external corpus | Repeating the run in production is wasteful or impossible |
| Change control requires that the exact binary reviewed by a risk committee be the one that serves traffic | Retraining in production would produce a different model than the one approved |
| The model comes from outside the pipeline entirely (a vendor model, a hub checkpoint, a hand-tuned notebook artifact) | There is no training code to promote |
What you give up: the production model was fitted on non-production data, so train/serve skew is a live risk; and each retraining cycle requires another manual promotion.
What Physically Moves
Naming the artifact that crosses the environment boundary is usually enough to settle the question:
| Deploy code | Deploy model | |
|---|---|---|
| Promoted artifact | A git commit / released Databricks Asset Bundle | A registered model version |
| Where training runs | Production workspace, on production data | Dev or staging workspace |
| Where the model version is first created | Production catalog | Dev/staging catalog, then copied across |
| What production compute does | Trains, validates, registers, and serves | Serves only |
| Routine retraining | Already scheduled in production; no promotion step | Needs a fresh promotion each cycle |
| Main risk carried | Production training cost and pipeline complexity | Train/serve skew from non-production data |
The Three-Environment Shape
Both patterns assume the same dev → staging → production progression, and Unity Catalog usually models it as three catalogs rather than three separate workspaces. Under deploy-code the pipeline definition is parameterised by target catalog, so the only thing that differs between environments is configuration. That is what makes "test the pipeline in staging, then run it for real in production" a meaningful claim: the code under test is identical to the code that produces the production model. Under deploy-model the pipeline is not what advances, so staging validates a model — its metrics on a frozen holdout — rather than the process that produced it.
The Hybrid Most Teams Actually Run
The two patterns are not mutually exclusive. A common arrangement promotes code for
the routine path and models for the exceptional one: the training pipeline is deployed
to production and retrains weekly, while an occasional expensively pretrained
embedding model is promoted as an artifact and consumed by that pipeline. Both paths
end at the same place — a version registered in Unity Catalog, evaluated by an
automated validation job, and pointed at by an alias such as @champion.
Exam framing: the question usually gives you one decisive constraint. "Production data cannot leave the production workspace" or "the pipeline must retrain nightly on fresh data" points to deploy code. "Fine-tuning takes three days on 8 GPUs" or "the auditor must approve the exact artifact that serves traffic" points to deploy model.
A regulated bank requires that model training run only inside the production workspace, because customer data may not be copied into development catalogs. Nightly retraining on fresh data is also mandatory. Which deployment pattern fits these constraints?
A computer-vision team fine-tunes a large image model over roughly 60 GPU-hours. A model risk committee reviews and signs off on the exact trained artifact before it may serve traffic. Which pattern is appropriate, and why?
Which statement correctly distinguishes the deploy-code and deploy-model patterns?