1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute

1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute

In Azure Databricks, Cluster Access Modes define the security boundary, user isolation model, language runtime capabilities, and Unity Catalog governance compatibility for compute resources.

Selecting the appropriate access mode ensures that enterprise security policies—such as row-level filters, column masking, and storage credential isolation—are strictly enforced while granting developers the language and library capabilities necessary for their workloads.

---

1. Evolution of Compute Security in Azure Databricks

Before Unity Catalog, Azure Databricks relied on legacy security mechanisms such as High Concurrency Table ACLs and Credential Passthrough. These legacy modes suffered from severe limitations:

• Language Restrictions: Table ACLs blocked Python and Scala execution unless restricted through custom wrappers, frequently breaking third-party libraries.
• Credential Leaks: Credential passthrough lacked fine-grained row/column masking and did not support automated service principal jobs cleanly.
• No Uniform Governance: Compute-level permissions were disconnected from data storage metadata.

Unity Catalog fundamentally replaced these legacy models with standardized, hardware- and process-isolated Cluster Access Modes that enforce centralized identity and access control across all compute types.

---

2. Single User Access Mode (Dedicated Compute)

Single User Access Mode (formerly called Dedicated mode) allocates the compute cluster exclusively to one designated user identity or one Service Principal.

                    SINGLE USER (DEDICATED) ACCESS MODE

  +-------------------------------------------------------------------------+
  | Assigned Identity: data_engineer@company.com (or Service Principal)     |
  |                                                                         |
  |   [ Spark Driver JVM ] <---------------------------------------------+  |
  |     - Full Unrestricted Access: Python, Scala, SQL, R                |  |
  |     - Custom Init Scripts, C++ Extensions, JVM Bytecode Allowed      |  |
  |     - Direct OS / DBFS Root Access Allowed                           |  |
  |                                                                         |
  |   Unity Catalog Governance: Evaluated ONLY for the Assigned Identity.   |
  |   Other users CANNOT attach notebooks or execute commands here.         |
  +-------------------------------------------------------------------------+

Key Characteristics & Language Support

• Full Polyglot Capabilities: Supports Python, Scala, SQL, and R without sandboxing or restriction.
• Custom Extensions & JVM Manipulation: Developers can install custom C/C++ libraries, use low-level Java Virtual Machine (JVM) reflection, execute shell commands (%sh), and run custom cluster-scoped Init Scripts.
• Unity Catalog Authorization: All queries against Unity Catalog tables, views, and volumes are authorized strictly using the assigned user's Entra ID identity and Unity Catalog privileges.

Primary Use Cases

• Data Science & Advanced Machine Learning: Workloads requiring specialized deep learning frameworks (PyTorch, TensorFlow), GPU libraries (CUDA), or custom R packages.
• Legacy Scala & Java Spark Pipelines: Automated production pipelines containing legacy Scala/Java JARs or low-level Spark RDD manipulation.
• Dedicated Single-User Development: Engineers requiring root filesystem access or custom native binary debugging.

Security Warning: Because Single User mode grants unrestricted access to the underlying VM and Spark JVM, no other user can be allowed to execute code on this cluster. Doing so would allow session snooping and privilege escalation.

---

3. Shared Access Mode (Standard / Multi-Tenant Compute)

Shared Access Mode (formerly called Standard mode) is the enterprise multi-tenant compute model designed for concurrent team collaboration under strict Unity Catalog governance.

                      SHARED (STANDARD) ACCESS MODE

  +-------------------------------------------------------------------------+
  | Multi-Tenant Users: Alice, Bob, Carol (Concurrent Sessions)             |
  |                                                                         |
  |   [ Process Isolation Sandbox ]                                         |
  |     - Supported Languages: Python & ANSI SQL ONLY                       |
  |     - Blocked: Arbitrary Scala, R, Shell (%sh), Unsafe JVM Calls        |
  |     - Blocked: Direct Local Root Disk Access & Raw Cloud Storage Keys   |
  |                                                                         |
  |   Unity Catalog Governance: Evaluated DYNAMICALLY per active user.      |
  |   - Alice sees masked data; Bob sees unmasked data (ABAC Enforced).     |
  +-------------------------------------------------------------------------+

User Isolation & Sandboxing Mechanics

• Process & Language Sandboxing: Multiple users attach their individual notebooks to the shared cluster simultaneously. Each user session runs in an isolated execution sandbox, preventing cross-user variable contamination or memory snooping.
• Supported Languages: Supports Python and ANSI SQL. Arbitrary Scala and R execution is blocked/restricted by default because the JVM does not provide native multi-tenant process isolation, and unrestricted Scala could allow a user to bypass Unity Catalog security.
• Fine-Grained Dynamic Governance: When a query executes, Unity Catalog dynamically evaluates permissions against the active querying user's Entra ID token:
  • Row Filters: Applied on-the-fly (WHERE region = 'EMEA').
  • Column Masking: Applied dynamically (hash(ssn) for unauthorized analysts).
  • Tag-Based Policies: Attribute-Based Access Control (ABAC) rules are strictly enforced.

Blocked Operations in Shared Mode

To guarantee multi-tenant security, Shared access mode intentionally blocks:

1. Execution of arbitrary shell scripts (%sh) and direct root filesystem writes.
2. Access to legacy DBFS root mount points containing plain-text credentials.
3. Modification of internal Spark JVM configuration parameters at runtime.
4. Direct acquisition of cloud storage master keys or IAM instance profile tokens.

---

4. Access Modes & Features Compatibility Comparison

The following matrix outlines the functional differences across all access modes:

---

5. Serverless Compute Architecture

Serverless Compute represents the next generation of compute execution in Azure Databricks, available for Serverless SQL Warehouses, Serverless Notebooks, Serverless Jobs, and Serverless Delta Live Tables.

+-------------------------------------------------------------------------+
|                        CUSTOMER AZURE SUBSCRIPTION                      |
|                                                                         |
|   [ Customer ADLS Gen2 Storage Account ] <-------------------+          |
|     - Bronze, Silver, Gold Delta Tables                      |          |
+--------------------------------------------------------------|----------+
                                                               | (Private Link / NCC)
+--------------------------------------------------------------|----------+
|                   DATABRICKS-MANAGED SERVERLESS PLANE        |          |
|                                                              |          |
|   +----------------------------------------------------------v-------+  |
|   | Hardware-Isolated Container Sandboxes                            |  |
|   |   - Instant Startup Fleet (<10 seconds)                          |  |
|   |   - Automated OS, DBR, and Security Patching                     |  |
|   |   - Dynamic Elastic Scaling (Instant scale-to-zero)              |  |
|   |   - Single-Tenant Network Virtualization per Workspace           |  |
|   +------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

Core Architectural Principles

1. Shift of Compute Plane: In classic compute, virtual machines reside in the customer's Azure VNet. In Serverless compute, containerized compute fleets run in a secure, Databricks-managed cloud environment located in the same Azure region.
2. Hardware-Level Sandboxing: Workloads are isolated using enterprise micro-VM and container virtualization boundaries. No container is ever shared across different customer workspaces or corporate tenants.
3. Network Connectivity Configurations (NCC): Serverless compute communicates securely with customer storage accounts (ADLS Gen2) over private network backbones using Azure Private Endpoints or managed Network Connectivity Configurations.
4. Zero Management Overhead: Cloud infrastructure provisioning, OS image hardening, Spark configuration tuning, and security vulnerability patching are managed automatically by Databricks without user intervention.

---

6. Credential Isolation & Token Downscoping in Unity Catalog

One of the most critical security functions of Unity Catalog across Shared and Serverless compute modes is Credential Isolation through Token Downscoping.

In legacy architectures, compute nodes required storage account keys or broad IAM Managed Identity access. If an analyst ran malicious Python code, they could extract those keys and gain unfettered access to all cloud storage data.

Step-by-Step Downscoping Execution Workflow

  [ Data Analyst ]
         |
         | 1. Submits Query: SELECT * FROM sales.orders WHERE region = 'US'
         v
  [ Cluster Driver / Executor ]
         |
         | 2. Requests Data Access Token for target table path
         v
  [ Unity Catalog Metastore ]
         |
         | 3. Validates Entra ID identity, Table Permissions, and Row Filters
         | 4. Requests Downscoped Temporary SAS / Bearer Token from Azure Storage
         v
  [ Azure Storage Security Broker ]
         |
         | 5. Issues Short-Lived Downscoped Token (Strictly path- & time-limited)
         v
  [ Spark Executor ]
         |
         | 6. Directly reads underlying Parquet files via ABFSS using Downscoped Token
         v
  [ ADLS Gen2 Storage ]

1. Query Submission: An analyst submits a SQL or PySpark query targeting a Unity Catalog table (e.g., main.finance.payments).
2. Metastore Authorization: The cluster driver intercepts the storage request and communicates with the Unity Catalog Metastore service.
3. Policy Evaluation: The metastore authenticates the user's Entra ID token and verifies that the user holds SELECT privileges. It dynamically injects any active row filters or column masks into the query execution plan.
4. Downscoped Credential Generation: The metastore generates a short-lived, downscoped temporary credential (such as an Azure Storage SAS token or Entra ID bearer token) restricted exclusively to the specific ADLS Gen2 folder path housing that table for the duration of the query.
5. Secure Execution: The downscoped credential is passed internally to the Spark executor storage subsystem over secure channels. The user's notebook code has no access to inspect, print, or capture this token.
6. Strict Expiration: Once the query finishes, the temporary token expires automatically, ensuring zero persistent credential exposure.