1.2 Advantages of Databricks ML Runtimes and Compute Selection

1.2 Advantages of Databricks ML Runtimes and Compute Selection

Every machine learning workload on Databricks starts with a compute decision, and the exam tests two halves of it: which runtime image to attach, and which cluster topology to allocate. Both are graded on the same criterion — does the configuration match the parallelism the training library actually has?

---

Databricks Runtime for Machine Learning (DBR ML)

The Databricks Runtime for Machine Learning (DBR ML) is a specialized operating environment built on top of the standard Databricks Runtime. It eliminates manual environment setup by pre-packaging and optimizing the industry's standard machine learning, deep learning, and distributed computing libraries.

Key Components Built into DBR ML

Exam Tip: When creating a cluster for machine learning workloads, always select a Databricks Runtime ML version (e.g., 14.3 LTS ML) rather than standard DBR. Standard DBR contains Apache Spark and core dependencies, but lacks scikit-learn, torch, xgboost, and MLflow out of the box.

---

Compute Architecture: CPU vs. GPU & Single-Node vs. Multi-Node

Selecting the correct cluster topology and hardware architecture directly impacts training velocity, memory utilization, and cloud compute cost.

   +-----------------------------------------------------------------------------------------+
   |                              COMPUTE TOPOLOGY SELECTION                                 |
   +-----------------------------------------------------------------------------------------+
                                               |
                       Is the training framework distributed?
                                               |
                        +----------------------+----------------------+
                        |                                             |
                       NO                                            YES
                        |                                             |
                        v                                             v
             [ Single-Node Cluster ]                        [ Multi-Node Cluster ]
             - Driver node only (no workers)               - 1 Driver + N Worker Nodes
             - Zero Spark network overhead                 - Distributed data partitions
             - Best for: pandas, scikit-learn,             - Best for: Spark MLlib, PyTorch FSDP,
               single-GPU deep learning                      Hyperopt with `SparkTrials`

Single-Node vs. Multi-Node Clusters

1. Single-Node Clusters:

   • Architecture: Allocates only a driver node with 0 worker nodes (spark.databricks.cluster.profile: singleNode). Spark operates in local mode (spark.master: local[*]).
   • Target Workloads: Traditional single-threaded or multi-core Python libraries such as scikit-learn, pandas, statsmodels, or standalone XGBoost.
   • Advantage: Eliminates inter-node network latency, Spark serialization overhead, and the cost of idle worker nodes. All memory and CPU cores on the driver instance are dedicated directly to the local Python process.

2. Multi-Node Clusters:

   • Architecture: Allocates 1 driver node and 1 or more worker nodes connected via high-speed cluster networking.
   • Target Workloads: Distributed Spark MLlib pipelines processing terabyte-scale Delta tables, distributed deep learning using Horovod or PyTorch Distributed Data Parallel (DDP), and hyperparameter sweeps using Hyperopt with SparkTrials.

CPU vs. GPU Acceleration

• CPU Clusters: Optimal for tabular data, classical machine learning algorithms (scikit-learn, LightGBM, XGBoost), exploratory data analysis with PySpark, and small-scale hyperparameter tuning. Provides the lowest cost-per-compute hour.
• GPU Clusters: Required for training deep convolutional neural networks (CNNs), Transformers, Large Language Models (LLMs), high-dimensional embedding generation, and CUDA-accelerated gradient boosting. DBR ML GPU images automatically configure NVIDIA drivers and container runtimes.

---

Library Management Hierarchy & Scoping

Managing package dependencies across development and production environments requires selecting the appropriate installation scope. Databricks provides three distinct mechanisms:

+---------------------------------------------------------------------------------------+
|                               LIBRARY SCOPING HIERARCHY                               |
+---------------------------------------------------------------------------------------+
| 1. Cluster-Scoped Libraries                                                           |
|    - Installed via Cluster UI, Compute API, or Init Scripts                          |
|    - Available to all users and all notebooks running on that compute cluster         |
+---------------------------------------------------------------------------------------+
| 2. Notebook-Scoped Libraries (%pip / %conda)                                         |
|    - Installed inside the notebook execution context via `%pip install <package>`      |
|    - Isolated to the specific notebook session; does not mutate cluster environment    |
+---------------------------------------------------------------------------------------+
| 3. Repo / Workspace-Scoped Libraries                                                  |
|    - Managed in Git Folders via `requirements.txt` or wheel packaging                  |
|    - Loaded dynamically via `%pip install -r requirements.txt`                         |
+---------------------------------------------------------------------------------------+

Cluster-Scoped Libraries

• Configuration: Installed via the Databricks Compute UI (under the Libraries tab), cluster initialization scripts (Init Scripts), or cluster creation JSON via the REST API.
• Scope & Persistence: Installed across both driver and all worker nodes. Persists across cluster restarts.
• Best Use Case: Core organization-wide base packages, custom enterprise drivers, or standardized internal SDKs.

Notebook-Scoped Libraries (%pip Magic)

• Configuration: Executed directly inside a notebook cell using the %pip magic command:

# Notebook cell: Installing specific dependency versions
%pip install shap==0.44.1 category_encoders==2.6.3

# Restart Python process automatically to load new packages into kernel
dbutils.library.restartPython()

• Scope & Isolation: The %pip magic command installs packages specifically into the active notebook's Python environment on the driver and automatically replicates them to Spark worker nodes for Python UDF execution. Other notebooks running concurrently on the same shared cluster remain unaffected.
• Best Use Case: Rapid experimentation, testing conflicting package versions, and ensuring notebook self-containment.

Repo-Scoped Requirements

• Configuration: Developers place a standard requirements.txt or pyproject.toml file at the root of their Databricks Git Folder.
• Execution: At the top of a production or training notebook, execute:

# Install exact pinned dependencies from repository root
%pip install -r ../requirements.txt

• Best Use Case: Production scheduled jobs where deterministic dependency resolution across Git release tags is required.

---

Summary of Library Scoping Options