4.2 Account-Level Identities, SCIM Sync, & Group Management
Key Takeaways
- Unity Catalog centralizes identity management at the Databricks Account level (accounts.azuredatabricks.net), decoupling users, groups, and service principals from individual workspaces.
- Automated SCIM (System for Cross-domain Identity Management) synchronization from Microsoft Entra ID provisions users and security groups directly to the Databricks Account.
- Legacy workspace-local groups cannot be referenced in Unity Catalog access control grants; all governance policies require centrally synchronized Account Groups.
- Service principals authenticated via OAuth 2.0 machine-to-machine (M2M) credentials replace personal access tokens (PATs) for production CI/CD pipelines, Lakeflow Jobs, and automated ETL.
- The core RBAC best practice in Unity Catalog is to grant privileges exclusively to functional Account Groups rather than individual user accounts.
4.2 Account-Level Identities, SCIM Sync, & Group Management
DP-750 Exam Focus: Master identity architecture in Azure Databricks. Understand the differences between Account-level identities and legacy workspace-local identities, how System for Cross-domain Identity Management (SCIM) automates provisioning from Microsoft Entra ID, how to configure and utilize Service Principals with OAuth M2M authentication for automated pipelines, and why granting privileges to Account Groups is the required enterprise best practice.
The Account-Level Identity Plane
In legacy Azure Databricks deployments, identity management was decentralized. Each workspace maintained its own isolated directory of users and groups. If an enterprise operated five workspaces across development, staging, and production, user onboarding, offboarding, and group permission updates had to be manually repeated in all five workspaces.
Unity Catalog introduces a centralized Account-level identity plane managed through the Databricks Account Console (accounts.azuredatabricks.net).
+-----------------------------------------------------------------------------------------+
| Microsoft Entra ID |
| (Enterprise Identity Provider: Users, Groups, App Registrations) |
+-----------------------------------------------------------------------------------------+
|
| Automated SCIM 2.0 Provisioning
v
+-----------------------------------------------------------------------------------------+
| Azure Databricks Account Console (Central Plane) |
| accounts.azuredatabricks.net |
| |
| +-----------------------+ +-----------------------+ +-----------------------------+ |
| | Account Users | | Account Groups | | Account Service Principals | |
| | (jane.doe@corp.com) | | (`data-engineers`) | | (sp-lakeflow-prod-etl) | |
| +-----------------------+ +-----------------------+ +-----------------------------+ |
+-----------------------------------------------------------------------------------------+
| |
Workspace | | Workspace
Assignment v v Assignment
+---------------------------------------+ +---------------------------------------+
| Workspace: Dev-EastUS | | Workspace: Prod-EastUS |
| - Workspace Access Entitlement | | - Workspace Access Entitlement |
| - Databricks SQL Access | | - Serverless Compute Access |
+---------------------------------------+ +---------------------------------------+
The Three Identity Types in Unity Catalog
- Users: Individual human identities authenticated via Microsoft Entra ID Single Sign-On (SSO). Represented by their corporate email address (e.g.,
alex.smith@company.com). - Service Principals: Non-human automated identities representing applications, CI/CD tools (GitHub Actions, Azure DevOps), orchestration systems (Azure Data Factory), or automated Lakeflow Jobs. Service principals do not consume interactive user licenses.
- Groups: Logical collections of users and service principals. Groups simplify access control by allowing administrators to grant privileges to a single collective entity rather than managing hundreds of individual user grants.
Microsoft Entra ID SCIM Synchronization
SCIM (System for Cross-domain Identity Management 2.0) is an open standard protocol that automates the exchange of user and group identity data between identity providers and cloud applications. In Azure Databricks, SCIM synchronizes identities directly from Microsoft Entra ID (formerly Azure Active Directory) into the Databricks Account Console.
Account-Level SCIM vs. Legacy Workspace-Level SCIM
| Architectural Attribute | Account-Level SCIM (Recommended) | Workspace-Level SCIM (Legacy) |
|---|---|---|
| Sync Target | Databricks Account Console (accounts.azuredatabricks.net) | Individual workspace instance URL (adb-xxx.azuredatabricks.net) |
| Entra ID App Type | Databricks Account SCIM Provisioning Connector | Workspace-specific Enterprise Application |
| Identity Scope | Creates Account Groups and Account Users available across all workspaces and metastores | Creates workspace-local users and groups visible only in that single workspace |
| Unity Catalog Compatibility | Fully Compatible: Account groups can be granted privileges on any UC securable | Incompatible: Workspace-local groups cannot be used in Unity Catalog GRANT statements |
| Offboarding Speed | Immediate global deactivation across all workspaces and metastores upon Entra ID disablement | Requires manual or per-workspace synchronization runs to revoke access |
How SCIM Synchronization Operates
- An Azure Cloud Administrator registers a Databricks SCIM Provisioning Application in Microsoft Entra ID.
- The administrator generates a SCIM token in the Databricks Account Console and configures the Tenant URL:
https://accounts.azuredatabricks.net/api/2.0/accounts/<account-id>/scim/v2 - Specific Entra ID Security Groups (e.g.,
AAD-DataEngineers,AAD-DataAnalysts,AAD-FinanceTeam) are assigned to the provisioning application. - Entra ID automatically provisions users, group memberships, and identity attributes into the Databricks Account every 40 minutes (or on-demand via Provision on demand).
- When an employee departs and is disabled in Entra ID, the SCIM engine deactivates their Databricks account identity, instantly terminating all active sessions, token authentications, and Unity Catalog data access.
Account Groups vs. Legacy Workspace-Local Groups
A frequent source of confusion on the DP-750 exam is the distinction between Account Groups and Workspace-Local Groups.
ACCOUNT GROUPS (Unity Catalog Standard) WORKSPACE-LOCAL GROUPS (Legacy)
+-------------------------------------+ +-------------------------------------+
| - Created in Account Console or SCIM| | - Created inside Workspace Admin UI |
| - Synchronized across all workspaces| | - Isolated to one single workspace |
| - Usable in Unity Catalog GRANTs | | - CANNOT be used in UC GRANTs |
| - Example: `data_engineers_prod` | | - Example: `local_analysts_ws1` |
+-------------------------------------+ +-------------------------------------+
| |
v v
GRANT SELECT ON TABLE ... GRANT SELECT ON TABLE ...
[SUCCESS] [ERROR: Invalid Principal]
Why Workspace-Local Groups Cannot Govern Unity Catalog
Unity Catalog metastores span multiple workspaces. Because a workspace-local group exists only within the internal database of a single workspace, other workspaces bound to the same metastore cannot resolve its membership. Consequently, Unity Catalog strictly enforces that only Account Groups can be granted privileges on Unity Catalog securables.
Exam Trap: If you attempt to execute
GRANT SELECT ON TABLE prod.silver.orders TO local_team_group;, Unity Catalog will return an error stating that the principallocal_team_groupdoes not exist or is not an account-level identity. Workspace groups must be migrated to Account Groups via Account SCIM or the Databricks Account Console.
Service Principals & Automated Machine-to-Machine (M2M) Workloads
A Service Principal is a headless, non-interactive identity used by automated systems, scheduled workflows, and external applications. In production Azure Databricks lakehouses, all scheduled ingestion, Delta Live Tables / Lakeflow pipelines, and CI/CD operations must run under a service principal rather than a human user account.
Why Production Pipelines Must NOT Run Under Personal User Identities
- Employee Turnover & Offboarding: When a data engineer leaves the organization, their Entra ID account is deactivated. If production Lakeflow Jobs or Delta Live Tables were scheduled under their personal identity, all associated production pipelines immediately fail with authentication errors.
- Security & Auditability: Service principals can be restricted to least-privilege data access without granting interactive workspace login rights.
- Token Rotation & Expiration: Personal Access Tokens (PATs) expire or can be accidentally exposed in code. Service principals use managed enterprise OAuth 2.0 tokens.
Service Principal Types in Azure Databricks
- Microsoft Entra ID Service Principals: An Application Registration in Microsoft Entra ID (
Application (client) ID). Best practice for enterprise multi-resource Azure integration (e.g., accessing Azure Key Vault, Azure Event Hubs, and Databricks). - Databricks Managed Service Principals: Created directly in the Databricks Account Console. Useful for Databricks-native automation when direct Entra ID tenant registration is restricted.
OAuth 2.0 Machine-to-Machine (M2M) Authentication
Databricks supports native OAuth 2.0 M2M authentication for service principals, eliminating the need for long-lived Personal Access Tokens (PATs):
+--------------------------+ +-------------------------------+ +----------------------------+
| Automated CI/CD / Job | | Databricks OAuth Endpoint | | Databricks REST API / UC |
| (e.g., GitHub Actions) | | (accounts.azuredatabricks...)| | (Cluster / Query Engine) |
+--------------------------+ +-------------------------------+ +----------------------------+
| | |
| 1. POST /oidc/v1/token | |
| (client_id + client_secret) | |
+-------------------------------------->| |
| | |
| 2. Returns short-lived OAuth token | |
| (Valid for 60 minutes) | |
|<--------------------------------------+ |
| |
| 3. API Request with Bearer Token (Authorization: Bearer <token>) |
+------------------------------------------------------------------------------>|
| |
| 4. Executes pipeline / queries UC securable under Service Principal identity |
|<------------------------------------------------------------------------------+
-- Granting Unity Catalog Data Privileges to a Service Principal
GRANT USE CATALOG ON CATALOG prod_lakehouse TO `00000000-1111-2222-3333-444444444444`;
GRANT USE SCHEMA, CREATE TABLE, MODIFY ON SCHEMA prod_lakehouse.raw TO `00000000-1111-2222-3333-444444444444`;
-- Or better: Add the Service Principal to an Account Group and grant to the group!
GRANT USE CATALOG ON CATALOG prod_lakehouse TO `etl_pipeline_runners`;
GRANT USE SCHEMA, MODIFY ON SCHEMA prod_lakehouse.silver TO `etl_pipeline_runners`;
Enterprise Best Practice: Grant to Groups, Never to Users
The fundamental governance rule for the DP-750 exam is: Never assign direct Unity Catalog privileges to individual user accounts. All privileges must be assigned to Account Groups.
ANTI-PATTERN (Direct User Grants) RECOMMENDED PATTERN (Group-Based RBAC)
User: alice@corp.com Microsoft Entra ID Security Group:
--> GRANT SELECT ON TABLE ... `AAD-Finance-Analysts`
User: bob@corp.com |
--> GRANT SELECT ON TABLE ... | (Automated SCIM Sync)
User: charlie@corp.com v
--> GRANT SELECT ON TABLE ... Databricks Account Group: `finance_analysts`
|
* When 50 analysts join/leave, v
50 individual SQL statements must be run. GRANT SELECT ON SCHEMA finance.gold TO `finance_analysts`;
* Auditing requires parsing thousands of ACLs. * Adding Alice to Entra ID group instantly grants access.
Recommended Group Taxonomy Pattern
| Account Group Name | Typical Membership | Assigned Unity Catalog Privileges |
|---|---|---|
data_engineers_core | Data engineers, pipeline developers | CREATE CATALOG (Dev), USE CATALOG, USE SCHEMA, CREATE TABLE, MODIFY (Staging/Prod) |
data_analysts_bi | BI developers, reporting analysts | USE CATALOG, USE SCHEMA, SELECT on gold schemas and reporting views |
ml_engineers_nlp | Data scientists, ML engineers | USE CATALOG, USE SCHEMA, SELECT, READ VOLUME, CREATE MODEL |
lakeflow_pipeline_runners | Production Service Principals | USE CATALOG, USE SCHEMA, MODIFY, READ VOLUME, WRITE VOLUME on bronze/silver |
governance_auditors | Compliance officers, internal auditors | USE CATALOG, USE SCHEMA, SELECT on system.access, system.billing, system.lineage |
An enterprise with 500 data engineers across multiple regional Azure Databricks workspaces needs to ensure user onboarding, role updates, and offboarding are fully automated from Microsoft Entra ID without managing credentials in individual workspaces. What is the recommended identity architecture?
A data engineer attempts to grant read permissions to a team using the following SQL statement in Databricks SQL:
GRANT SELECT ON TABLE marketing_dw.gold.roi_metrics TO local_marketing_group;
The command fails with an error indicating the principal cannot be resolved or is invalid. What is the root cause of this failure?
A mission-critical nightly Lakeflow ETL job fails with authentication errors because the data engineer who scheduled the job left the company and their corporate Entra ID account was disabled. How should the engineering team reconfigure the production job to prevent future failures due to personnel changes?