1.13 Information Available in the MLflow UI

1.13 Information Available in the MLflow UI

Machine learning is an inherently empirical discipline. Developing high-performing models requires testing dozens of algorithmic architectures, feature combinations, and hyperparameter permutations. MLflow Tracking is the central component of the open-source MLflow platform integrated into Databricks, providing a unified logging engine for parameters, metrics, code versions, environments, and model artifacts.

+---------------------------------------------------------------------------------------------------+
|                                    MLFLOW TRACKING ARCHITECTURE                                   |
+---------------------------------------------------------------------------------------------------+
| EXPERIMENT: Top-level organizational unit (e.g., `/Shared/Experiments/churn_prediction`)          |
|                                                                                                   |
|  +---------------------------------------------------------------------------------------------+  |
|  | RUN: Individual execution instance (Run ID: `a1b2c3d4e5f6...`)                              |  |
|  |                                                                                             |  |
|  |  [ Parameters ]    -->  `learning_rate`: 0.01, `n_estimators`: 200, `max_depth`: 6          |  |
|  |  [ Metrics ]       -->  `train_loss`: 0.14, `val_f1`: 0.89, `epoch_time`: 12.4s (by step)   |  |
|  |  [ Tags ]          -->  `git_commit`: `8f3a1b`, `environment`: `dev`, `author`: `alex`      |  |
|  |  [ Artifacts ]     -->  `model/` (MLmodel, weights), `plots/confusion_matrix.png`, `data/`  |  |
|  +---------------------------------------------------------------------------------------------+  |
+---------------------------------------------------------------------------------------------------+

---

Experiment Types: Workspace vs. Notebook Experiments

On Databricks, experiments exist in two distinct operational forms:

1. Notebook Experiments:

   • Lifecycle: Automatically created when an MLflow logging command (e.g., mlflow.start_run()) is executed within a Databricks notebook that does not explicitly set an active experiment.
   • Binding: Firmly bound to the specific notebook. Accessible via the Experiment Sidebar in the Databricks notebook UI.
   • Best Use Case: Ad-hoc, personal exploratory data science and rapid model prototyping.

2. Workspace Experiments:

   • Lifecycle: Explicitly created as standalone workspace objects via the Databricks UI (New Experiment) or programmatically via mlflow.create_experiment("/Shared/Experiments/fraud_v1").
   • Binding: Not tied to any single notebook. Multiple team members, automated Databricks Workflows, and CI/CD pipelines can log runs into the same shared workspace experiment.
   • Best Use Case: Collaborative team projects, centralized production retraining jobs, and formal benchmarking.

---

The Experiment Page

Opening an experiment shows the run table — one row per run — with a fixed set of built-in columns and a dynamic set drawn from what the runs logged:

The table supports sorting by any column, a search box using the same filter grammar as search_runs (metrics.val_f1 > 0.9), column show/hide, and grouping. Nested runs appear collapsed under their parent, which is how a Hyperopt sweep of 200 trials stays readable.

The Run Detail Page

Clicking a run opens a page divided into fixed sections:

1. Overview — run ID, status, timing, source, user, and the Git commit or notebook revision.
2. Parameters — the full parameter set, as logged.
3. Metrics — the final value of each metric. Clicking a metric that was logged with step opens an interactive line chart of the whole series.
4. Tags — including MLflow's own tags (mlflow.source.name, mlflow.user, mlflow.parentRunId) plus anything you set.
5. Artifacts — a file browser over everything logged to the run.
6. Models — the logged model, its flavour, and a link to any registered version created from it.

What the artifact viewer renders

• The MLmodel file, listing available flavours and the loader module.
• The model signature — declared input and output schema.
• Environment files (requirements.txt, conda.yaml, python_env.yaml) showing the exact pinned dependencies.
• Images (confusion matrices, SHAP plots) inline, plus tables, text, and JSON.

Chart View and Run Comparison

Two views turn a pile of runs into a decision:

• Chart view plots metrics across runs in the experiment. Parallel-coordinates plots are the standard way to read a hyperparameter sweep: each vertical axis is a parameter or metric, and each line is a run, so the parameter ranges that produce good scores become visually obvious.
• Compare (select several runs, then Compare) produces a side-by-side table that highlights differing parameters, a combined metric table, and overlaid metric curves for stepped metrics.

Databricks-Specific Additions

Exam framing: questions here are usually "which of these can you see in the MLflow UI". Parameters, metrics (including per-step curves), tags, artifacts, model signature, environment, run source, and run status are all visible. Things that are not in the MLflow UI: cluster billing details, Unity Catalog grants, and raw training data rows.