2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions

2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions

DP-750 Exam Focus: Two skill-area 1 bullets land squarely on this section - "Configure compute feature settings, including Photon acceleration, Azure Databricks runtime/Spark version, and machine learning" and "Configure access permissions to a compute resource." Section 2.2 already covered Photon and compute policies. This section closes the loop on runtime versioning and on who is allowed to attach to, restart, or reconfigure a cluster.

---

1. What a Databricks Runtime Version Actually Pins

A Databricks Runtime (DBR) version is not a Spark version label. It is a complete, immutable machine image containing:

• A hardened Ubuntu Linux base and the Databricks container runtime
• A Databricks-optimized build of Apache Spark (the Spark version is a property of the DBR, not something you choose independently)
• Delta Lake, including the set of Delta table features the image can read and write
• Language runtimes: a specific Python version, Scala, Java (JDK), and R
• Preinstalled libraries (pandas, NumPy, PyArrow, the ABFSS/Hadoop Azure connectors, JDBC drivers)
• The Photon native execution binaries on Photon-capable images

Because all of that is baked into one image, upgrading a runtime can change results, not just speed: a newer DBR may write a Delta protocol version an older reader cannot open, or ship a pandas major release that changes type coercion in a UDF.

Runtime Variants You Can Select

Read the dropdown carefully. Photon and the ML runtime are both chosen from the Databricks Runtime version selector, but they are distinct images. You select one runtime; the Photon toggle only appears for versions that ship the Photon engine. Serverless compute has no runtime selector at all, which is why exam scenarios that say "serverless" also remove every runtime-version distractor from the answer set.

---

2. LTS Versus Non-LTS: the Production Stability Decision

Databricks publishes most runtimes on a short support window and designates specific releases as LTS (Long-Term Support) with a substantially longer one. The operational rule that follows from this is simple and heavily testable:

• Scheduled jobs and pipelines pin a named LTS version. A job that runs nightly for two years must not silently roll onto a new Spark build.
• Interactive/development compute may track the newest runtime so the team can validate new table features before production adopts them.
• Never let a production job float onto "latest." When a runtime reaches end of support, jobs pinned to it must be migrated deliberately, with a test run, rather than discovering the change in a 3 a.m. failure.

Runtime Version Gates Table Features

The exam likes scenarios where a feature "does not work" purely because the runtime is too old. Known minimums worth memorizing:

Enabling deletion vectors upgrades the Delta table protocol. After the upgrade, clients on older runtimes can no longer read the table - a classic "we upgraded one table and the downstream Synapse reader broke" scenario.

---

3. Databricks Runtime for Machine Learning

The ML runtime is the standard runtime plus a curated machine-learning stack, preinstalled and version-matched so the libraries actually work together:

• MLflow with autologging wired into the workspace tracking server
• Deep learning: PyTorch, TensorFlow/Keras, and their distributed training helpers
• Classical ML: scikit-learn, XGBoost, LightGBM
• GPU images: NVIDIA drivers, CUDA, and cuDNN preinstalled and matched to the GPU VM SKU

When not to use it. The ML image is much larger than the standard image, so clusters take noticeably longer to start and cost more per run. A nightly medallion ETL job that runs MERGE statements and writes Delta tables gains nothing from it. Choose the ML runtime only when the workload actually imports ML libraries or needs GPU acceleration.

{
  "cluster_name": "feature-engineering-gpu",
  "spark_version": "<ml-gpu-runtime-version>",
  "node_type_id": "Standard_NC6s_v3",
  "num_workers": 2,
  "autotermination_minutes": 30
}

The spark_version field in the Clusters API is where the runtime choice lives; ML and GPU images are distinct spark_version identifiers, not flags added to a standard runtime.

---

4. Compute Access Permissions: the Four Levels

Every compute resource has exactly four permission levels. They are strictly cumulative - each level includes everything below it.

Operational Rules the Exam Tests

1. You must hold CAN MANAGE on a compute to configure its permissions. Workspace admins hold it implicitly.
2. Permissions are the exception to the restart rule. Editing any attribute of a running compute requires a restart - except the compute size and permissions, which apply live. You can also update permissions on compute that is neither running nor terminated.
3. Cloning does not copy permissions. Cloning a compute pre-populates the configuration but deliberately drops compute permissions and attached notebooks. The clone starts with an empty ACL.
4. Driver logs are locked to CAN MANAGE by default. Secrets are not redacted from the Spark driver stdout and stderr streams, so on job and all-purpose compute in dedicated or standard access mode only CAN MANAGE users can read driver logs. To open them to CAN ATTACH TO and CAN RESTART users, set spark.databricks.acl.needAdminPermissionToViewLogs false in that compute's Spark config. It is a per-compute property - there is no workspace-wide switch.
5. Autostart checks two ACLs. When a scheduled job or a JDBC/ODBC connection triggers a terminated compute to auto-restart, Databricks evaluates both the compute access control list and the job access control list before starting it.
6. Terminated compute is deleted after 30 days unless an administrator pins it (up to 100 pinned compute resources per workspace).

How to Set Permissions

Three supported paths, all equivalent:

• Workspace UI - Compute -> kebab menu on the row -> Edit permissions -> select a user, group, or service principal and pick a level.
• Permissions API - for scripted or CI-driven grants.
• Databricks Terraform provider - for infrastructure-as-code environments.

Grant to groups, not individuals, for the same reason Unity Catalog privileges go to account-level groups: onboarding and offboarding become a group membership change instead of an ACL sweep across every cluster.

---

5. Compute ACLs Versus Compute Policies Versus Pool ACLs

These three controls are constantly confused in exam distractors because they all "restrict compute."

A user granted CAN USE on a policy can create clusters shaped by that policy, but that grant says nothing about the clusters other people already created. Conversely, CAN MANAGE on one cluster does not let a user create new ones outside policy limits.

---

6. Common Exam Traps

• CAN RESTART is not CAN MANAGE. A user who needs to restart a stuck streaming cluster needs CAN RESTART. A user who needs to add two workers needs CAN MANAGE. Granting CAN MANAGE "so they can restart it" over-grants.
• CAN ATTACH TO cannot restart. If the cluster is terminated, a CAN ATTACH TO user is stuck until someone with CAN RESTART starts it - or until an autostart-eligible job does.
• A larger runtime is not a faster runtime. Scenarios that add the ML runtime to an ETL cluster to "improve performance" are wrong; Photon, right-sized nodes, and better clustering are the performance levers.
• "Latest runtime" is rarely the right answer for a production job. Look for the LTS option.