3.2 Azure Managed Identity, Storage Credentials, & External Locations

3.2 Azure Managed Identity, Storage Credentials, & External Locations

DP-750 Exam Focus: Master the two-tier cloud storage security model in Unity Catalog: Cloud-level IAM (Azure Access Connector, User-Assigned Managed Identities, and Azure RBAC) combined with Catalog-level Securables (Storage Credentials and External Locations). Understand the exact DDL commands, Azure RBAC roles (Storage Blob Data Contributor), and URL validation rules required to establish secure external storage paths.

---

The Need for Credential-less Cloud Storage Architecture

In legacy Azure Databricks configurations, connecting to Azure Data Lake Storage Gen2 (ADLS Gen2) typically relied on one of three patterns:

1. Storage Account Access Keys: Broad account-level keys stored in Databricks Secrets or notebooks, presenting a severe credential leak risk.
2. Shared Access Signature (SAS) Tokens: Time-bounded tokens requiring ongoing rotation and management overhead.
3. DBFS Mounts (dbfs:/mnt/...): Global filesystem mounts that exposed underlying storage to all users on shared compute clusters, bypassing table-level security.

Unity Catalog eliminates all hardcoded secrets, keys, and global mount points by introducing a native, credential-less identity model built on Azure Managed Identities and Azure Resource Manager (ARM).

---

Azure Access Connector Architecture

An Azure Access Connector (Microsoft.Databricks/accessConnectors) is a native first-party Azure resource that acts as the identity bridge between the Azure Databricks control/data planes and Azure Storage.

+-----------------------------------------------------------------------------------------+
|                                      Azure Cloud                                        |
|                                                                                         |
|   +--------------------------+                         +----------------------------+   |
|   |     Azure Databricks     |                         |     ADLS Gen2 Storage      |   |
|   |    Unity Catalog Plane   |                         | (container@account.dfs...) |   |
|   +--------------------------+                         +----------------------------+   |
|                |                                                     ^                  |
|                | Assumes Identity via ARM                            |                  |
|                v                                                     |                  |
|   +---------------------------------------+                          |                  |
|   |        Azure Access Connector         |                          |                  |
|   |  (Microsoft.Databricks/               |                          |                  |
|   |   accessConnectors)                   |                          |                  |
|   |                                       |                          |                  |
|   |  Managed Identity:                    |                          |                  |
|   |  - System-Assigned (SAMI), or         |                          |                  |
|   |  - User-Assigned (UAMI)               |                          |                  |
|   +---------------------------------------+                          |                  |
|                        |                                             |                  |
|                        +--- Assigned RBAC: Storage Blob Data --------+                  |
|                             Contributor on Target Container                             |
+-----------------------------------------------------------------------------------------+

Managed Identity Types

When deploying an Access Connector in Azure, administrators choose between two managed identity strategies:

---

Azure RBAC Configuration on ADLS Gen2

For Unity Catalog to read, write, and manage files in an ADLS Gen2 container or path, the Access Connector's managed identity must be granted appropriate Azure Role-Based Access Control (RBAC) permissions in the Azure Portal or via Azure CLI/Terraform.

Required Azure RBAC Roles

• Storage Blob Data Contributor (Mandatory for Read/Write/Manage): Grants full read, write, and delete permissions to blob data within the target container. This role is required for metastore root storage, managed table storage, and writable external locations.
• Storage Blob Data Reader (Read-Only Locations): Grants read-only access. Used when an external location is strictly configured for source ingestion and must be protected against accidental writes or deletes.
• Storage Blob Data Owner: Grants full control including POSIX ACL modifications. Generally unnecessary unless advanced ACL delegation is required.

Exam Trap: Assigning traditional management roles such as Contributor or Owner at the Resource Group or Storage Account level does NOT grant data-plane access to ADLS Gen2 blob storage. Unity Catalog will fail to access data unless the specific Storage Blob Data Contributor (or Reader) data-plane role is assigned.

---

Unity Catalog Storage Credentials

A Storage Credential is a top-level securable object in the Unity Catalog metastore that encapsulates authentication to Azure storage. It represents the Databricks-side abstraction of the Azure Access Connector.

+-------------------------------------------------------------------+
|              Unity Catalog Storage Credential Object              |
|                                                                   |
|  Name: `azure_prod_credential`                                    |
|  Cloud Provider: Azure Managed Identity                           |
|  Azure Connector ID:                                              |
|   `/subscriptions/<sub>/resourceGroups/<rg>/                      |
|    providers/Microsoft.Databricks/accessConnectors/prod-connector`|
|  User-Assigned Identity ID (Optional):                            |
|   `/subscriptions/<sub>/.../userAssignedIdentities/uami-prod-data`|
+-------------------------------------------------------------------+

DDL Syntax for Creating Storage Credentials

1. Using System-Assigned Managed Identity (SAMI):

CREATE STORAGE CREDENTIAL azure_prod_credential
  AZURE_MANAGED_IDENTITY (
    CONNECTOR_ID = '/subscriptions/11111111-2222-3333-4444-555555555555/resourceGroups/rg-databricks-prod/providers/Microsoft.Databricks/accessConnectors/prod-access-connector'
  )
  COMMENT 'Primary storage credential for production ADLS Gen2 containers';

2. Using User-Assigned Managed Identity (UAMI):

CREATE STORAGE CREDENTIAL azure_enterprise_uami_cred
  AZURE_MANAGED_IDENTITY (
    CONNECTOR_ID = '/subscriptions/11111111-2222-3333-4444-555555555555/resourceGroups/rg-databricks-prod/providers/Microsoft.Databricks/accessConnectors/prod-access-connector',
    MANAGED_IDENTITY_ID = '/subscriptions/11111111-2222-3333-4444-555555555555/resourceGroups/rg-identities/providers/Microsoft.ManagedIdentity/userAssignedIdentities/uami-databricks-storage'
  )
  COMMENT 'Credential using dedicated enterprise User-Assigned Managed Identity';

Ownership and Permissions on Storage Credentials

• By default, only Metastore Admins can create storage credentials.
• Metastore Admins can delegate credential creation using: GRANT CREATE STORAGE CREDENTIAL ON METASTORE TO group_principal;
• Users who possess the CREATE EXTERNAL LOCATION privilege on a Storage Credential can use it to register External Locations.

---

Unity Catalog External Locations

An External Location is a securable object in Unity Catalog that combines a cloud storage URI path with a Storage Credential. It establishes a governed boundary over a specific container or subdirectory in ADLS Gen2.

+-------------------------------------------------------------------+
|               Unity Catalog External Location Object              |
|                                                                   |
|  Name: `raw_landing_location`                                     |
|  Storage URL: `abfss://raw-data@adlsprod.dfs.core.windows.net/`   |
|  Storage Credential: `azure_prod_credential`                      |
|  Encryption / Validation: Validated on creation                   |
+-------------------------------------------------------------------+

DDL Syntax for Creating External Locations

CREATE EXTERNAL LOCATION raw_landing_location
  URL 'abfss://raw-data@adlsprod.dfs.core.windows.net/landing/'
  WITH (STORAGE CREDENTIAL azure_prod_credential)
  COMMENT 'External location for raw landing ingestion files';

Automated URL Validation and Overlap Rules

When a CREATE EXTERNAL LOCATION statement is executed, Unity Catalog performs automated checks:

1. Active URL Validation: Unity Catalog immediately contacts Azure Storage using the specified Storage Credential to verify that it can list, read, and write a test file at the designated URL path. If the Azure RBAC assignment is missing or the storage account firewall blocks traffic, creation aborts with an authorization error.
2. Bypassing Validation (Optional): In scenarios where storage will become available later, validation can be skipped with WITHOUT VALIDATION:

   CREATE EXTERNAL LOCATION staged_location
     URL 'abfss://staged@adlsprod.dfs.core.windows.net/'
     WITH (STORAGE CREDENTIAL azure_prod_credential)
     WITHOUT VALIDATION;
   

3. Overlap Prevention: Unity Catalog prevents overlapping external location URLs unless strict parent-child containment hierarchy rules are respected. You cannot define two external locations pointing to the exact same URI with different credentials.

---

Privilege Hierarchy on Storage Securables

Unity Catalog uses standard ANSI SQL GRANT and REVOKE statements to control access to Storage Credentials and External Locations:

-- Granting team privileges on an External Location
GRANT READ FILES, WRITE FILES ON EXTERNAL LOCATION raw_landing_location TO `data_engineers`;
GRANT CREATE EXTERNAL TABLE ON EXTERNAL LOCATION raw_landing_location TO `data_engineers`;
GRANT CREATE EXTERNAL VOLUME ON EXTERNAL LOCATION raw_landing_location TO `data_engineers`;

---

End-to-End Authentication and Access Flow

Understanding the runtime authentication flow is essential for diagnosing access issues on the DP-750 exam:

1. Data Engineer issues query: 
   SELECT * FROM prod_catalog.raw.iot_telemetry;

2. Databricks Runtime queries Unity Catalog Metastore:
   - Metastore verifies user has SELECT privilege on table.
   - Metastore identifies table is External, residing at abfss://raw@adls.../iot
   - Metastore checks External Location & Storage Credential.

3. Databricks Control Plane requests short-lived token:
   - Calls Azure Resource Manager (ARM) / Entra ID using Azure Access Connector.
   - Entra ID validates Access Connector's Managed Identity.
   - Issues temporary down-scoped storage token.

4. Databricks Compute Cluster accesses ADLS Gen2 directly:
   - Uses temporary storage token to read raw Parquet/Delta files.
   - Cluster never receives or handles persistent credentials or keys.