5.3 Data Isolation across Workspaces & Multiple Metastore Patterns

5.3 Data Isolation across Workspaces & Multiple Metastore Patterns

DP-750 Exam Focus: Understand enterprise patterns for isolating data across Azure Databricks workspaces. Master the architectural mechanics of catalog-to-workspace binding (Standard vs. Isolated catalog modes), environment segregation (Dev, Staging, Prod) under a single regional metastore, read-only workspace catalog bindings, and the specific compliance scenarios that justify multi-metastore topologies.

---

1. The Multi-Workspace Lakehouse Architecture

In enterprise cloud deployments, organizations operate multiple Azure Databricks workspaces to achieve operational segregation between software development lifecycles (SDLC) and departmental cost centers. A standard deployment topology comprises:

• Development Workspace (ws-dev): Interactive data science, rapid pipeline prototyping, experimental feature branch testing.
• Staging / Test Workspace (ws-staging): Automated integration testing, release candidate verification, Databricks Asset Bundle (DAB) pre-deployment validation.
• Production Workspace (ws-prod): Mission-critical Lakeflow Jobs, 24/7 streaming ingestion, executive BI dashboards, strict SLA-driven workloads.

                               ENTERPRISE MULTI-WORKSPACE TOPOLOGY
                               
                               +---------------------------------+
                               |     Azure Databricks Account    |
                               |  (accounts.azuredatabricks.net) |
                               +---------------------------------+
                                                |
                                                v
                               +---------------------------------+
                               |   Regional Metastore (East US)  |
                               +---------------------------------+
                                                |
               +--------------------------------+--------------------------------+
               |                                |                                |
               v                                v                                v
    +--------------------+           +--------------------+           +--------------------+
    |   Workspace: DEV   |           | Workspace: STAGING |           |  Workspace: PROD   |
    | (Interactive/EDA)  |           | (CI/CD Integration)|           | (Automated Jobs/BI)|
    +--------------------+           +--------------------+           +--------------------+

The Golden Architectural Rule: One Metastore Per Region

A fundamental tenet of Azure Databricks Unity Catalog is that each Azure region should have exactly one Unity Catalog metastore. All workspaces provisioned within that region link to the shared regional metastore.

Historically, legacy teams attempted to isolate environments by creating separate metastores for Dev, Test, and Prod. In Unity Catalog, this is an anti-pattern that introduces severe complications:

• Prevents centralized data governance and unified metadata indexing.
• Breaks automated column-level data lineage across environments.
• Precludes cross-catalog querying and centralized audit logging.
• Requires complex Delta Sharing configurations just to share reference lookup tables between environments.

Instead, Unity Catalog achieves robust environmental isolation within a single metastore using Catalog-to-Workspace Binding.

---

2. Catalog-to-Workspace Binding (Standard vs. Isolated Modes)

By default, when a catalog is created in Unity Catalog, it operates in Standard (Unbound) Access Mode. This means every workspace currently or subsequently assigned to the metastore can view and query the catalog (subject to standard SQL GRANT privileges).

To enforce strict environment boundaries, Unity Catalog provides Isolated Catalog Mode (Workspace Binding). In this mode, the catalog is explicitly bound to a whitelist of specific Azure Databricks workspace IDs. Compute clusters in non-assigned workspaces cannot see, discover, or query the catalog, even if a user holds SELECT privileges.

+-----------------------------------------------------------------------------------------+
|                        CATALOG-TO-WORKSPACE BINDING PATTERNS                            |
+-----------------------------------------------------------------------------------------+
|                                                                                         |
|  1. STANDARD ACCESS (All Workspaces):                                                  |
|     - Catalog is accessible by all workspaces attached to the regional metastore.       |
|     - Ideal for global reference data (e.g., 'enterprise_lookup', 'samples', 'system'). |
|                                                                                         |
|  2. ISOLATED ACCESS (Assigned Workspaces Only):                                         |
|     - Catalog is locked down exclusively to designated Workspace IDs.                   |
|     - Workspaces not on the binding list cannot view or resolve catalog objects.        |
|     - Guarantees Dev workspaces cannot interact with 'prod_catalog'.                     |
|                                                                                         |
|  3. READ-ONLY BINDING:                                                                  |
|     - Non-production workspaces can query tables in a shared catalog, but all DML writes|
|       (INSERT, UPDATE, DELETE, CREATE TABLE) are blocked at the engine level.           |
+-----------------------------------------------------------------------------------------+

ANSI SQL DDL for Managing Workspace Bindings

Workspace bindings can be managed via the Databricks Account Console, Catalog Explorer UI, or ANSI SQL statements executed by a Metastore Admin or Catalog Owner:

-- Step 1: Create an isolated catalog for production workloads
CREATE CATALOG prod_enterprise_catalog
  MANAGED LOCATION 'abfss://prod-data@adlsprod.dfs.core.windows.net/managed'
  COMMENT 'Production catalog isolated to production workspace compute';

-- Step 2: Restrict the catalog to Isolated mode and bind to Production Workspace ID
-- (Workspace IDs are numeric strings found in the workspace URL: ?o=1234567890123456)
ALTER CATALOG prod_enterprise_catalog 
SET ISOLATION ISOLATED;

-- Step 3: Add explicit workspace assignments
ALTER CATALOG prod_enterprise_catalog 
ADD WORKSPACE 1234567890123456; -- Production Workspace ID

-- Step 4: Verify active workspace bindings
DESCRIBE CATALOG EXTENDED prod_enterprise_catalog;

Creating and Binding Development Catalogs

-- Create development catalog
CREATE CATALOG dev_sandbox_catalog
  MANAGED LOCATION 'abfss://dev-data@adlsdev.dfs.core.windows.net/managed';

ALTER CATALOG dev_sandbox_catalog 
SET ISOLATION ISOLATED;

-- Bind dev catalog exclusively to Dev and Staging workspaces
ALTER CATALOG dev_sandbox_catalog 
ADD WORKSPACE 9876543210987654; -- Dev Workspace ID

ALTER CATALOG dev_sandbox_catalog 
ADD WORKSPACE 5555555555555555; -- Staging Workspace ID

Revoking Workspace Access

-- Remove staging workspace access from development catalog
ALTER CATALOG dev_sandbox_catalog 
REVOKE WORKSPACE 5555555555555555;

---

3. Granular Workspace Access Modes: Read-Only Workspace Bindings

In many enterprise scenarios, data scientists in the Development Workspace need to train machine learning models using clean, historical data residing in the Production Catalog, but security policies strictly prohibit non-production compute from writing to, deleting, or altering production tables.

Unity Catalog supports configuring Read-Only Workspace Bindings on isolated catalogs:

                               READ-ONLY BINDING ARCHITECTURE
                               
                     +---------------------------------------+
                     |     Catalog: prod_reference_data      |
                     |    (Storage: adlsprod.dfs.core...)    |
                     +---------------------------------------+
                                   /           \
         (Read-Write Binding)     /             \\\     (Read-Only Binding)
                                 v               v
                  +-------------------+     +-------------------+
                  |  Workspace: PROD  |     |  Workspace: DEV   |
                  |  - SELECT (OK)    |     |  - SELECT (OK)    |
                  |  - INSERT (OK)    |     |  - INSERT (BLOCK) |
                  |  - UPDATE (OK)    |     |  - UPDATE (BLOCK) |
                  |  - DROP (OK)      |     |  - DROP (BLOCK)   |
                  +-------------------+     +-------------------+

Operational Guarantees of Read-Only Bindings

1. Zero Write Exposure: Even if a user in the Dev workspace is assigned the MODIFY, CREATE TABLE, or OWNER privilege within the catalog, all write operations executed from the Dev workspace fail immediately with a workspace permission violation.
2. Safe Feature Engineering: Data engineers and ML practitioners can perform interactive SQL queries, run Spark DataFrame operations, and train models against live production reference tables without duplicating data across storage accounts.

---

4. Storage Credential & External Location Isolation Patterns

True enterprise data isolation requires securing not just the logical catalog namespace, but also the physical cloud storage infrastructure underneath.

+-----------------------------------------------------------------------------------------+
|                        PHYSICAL STORAGE ISOLATION TOPOLOGY                              |
+-----------------------------------------------------------------------------------------+
|                                                                                         |
|  +-------------------------------------+   +-------------------------------------+      |
|  |   Dev Azure Resource Group          |   |   Prod Azure Resource Group         |      |
|  |   - Storage Account: adlsdev        |   |   - Storage Account: adlsprod       |      |
|  |   - Access Connector: conn-dev      |   |   - Access Connector: conn-prod     |      |
|  +-------------------------------------+   +-------------------------------------+      |
|                     |                                         |                         |
|                     v                                         v                         |
|  +-------------------------------------+   +-------------------------------------+      |
|  | Storage Credential: cred_dev        |   | Storage Credential: cred_prod       |      |
|  | (Uses conn-dev Managed Identity)    |   | (Uses conn-prod Managed Identity)   |      |
|  +-------------------------------------+   +-------------------------------------+      |
|                     |                                         |                         |
|                     v                                         v                         |
|  +-------------------------------------+   +-------------------------------------+      |
|  | External Location: loc_dev_landing  |   | External Location: loc_prod_landing |      |
|  | Path: abfss://landing@adlsdev...    |   | Path: abfss://landing@adlsprod...   |      |
|  +-------------------------------------+   +-------------------------------------+      |
|                     |                                         |                         |
|                     v                                         v                         |
|  +-------------------------------------+   +-------------------------------------+      |
|  | Catalog: dev_catalog (Isolated)     |   | Catalog: prod_catalog (Isolated)    |      |
|  | Bound to: Dev Workspace ID Only     |   | Bound to: Prod Workspace ID Only    |      |
|  +-------------------------------------+   +-------------------------------------+      |
+-----------------------------------------------------------------------------------------+

Best Practices for Storage Isolation

• Separate Azure Storage Accounts: Provision distinct ADLS Gen2 storage accounts in separate Azure subscriptions or resource groups for Dev, Staging, and Production.
• Dedicated Azure Access Connectors: Create a dedicated Azure Access Connector for each environment. Assign Storage Blob Data Contributor on adlsdev strictly to conn-dev, and adlsprod strictly to conn-prod.
• No Cross-Credential Grants: Never grant permissions on cred_prod or loc_prod to non-production data engineering roles.

---

5. Metastore Topologies: Single-Metastore vs. Multi-Metastore Patterns

While the single metastore per region rule is the standard recommendation, specific compliance and architectural boundaries dictate when multi-metastore topologies are required.

Metastore Architecture Decision Matrix

                               MULTI-REGION METASTORE ARCHITECTURE
                               
  +-----------------------------------------+   +-----------------------------------------+
  |       Azure Region: North America       |   |          Azure Region: Europe           |
  |                                         |   |                                         |
  |  +-----------------------------------+  |   |  +-----------------------------------+  |
  |  |   UC Metastore: East US Region    |  |   |  |  UC Metastore: West Europe Region |  |
  |  |  (Root: abfss://uc-root@adls-us)  |  |   |  |  (Root: abfss://uc-root@adls-eu)  |  |
  |  +-----------------------------------+  |   |  +-----------------------------------+  |
  |                  |                      |   |                  |                      |
  |        +---------+---------+            |   |        +---------+---------+            |
  |        v                   v            |   |        v                   v            |
  |  +------------+     +------------+      |   |  +------------+     +------------+      |
  |  |  WS: US-Dev|     | WS: US-Prod|      |   |  |  WS: EU-Dev|     | WS: EU-Prod|      |
  |  +------------+     +------------+      |   |  +------------+     +------------+      |
  +-----------------------------------------+   +-----------------------------------------+
                         \                                 /
                          \                               /
                           +--- Delta Sharing (UC-to-UC)-+
                               (Zero-Copy Data Exchange)

---

6. End-to-End Governance Checklist for Environment Isolation

When configuring environment isolation for the DP-750 exam, ensure the following sequence is executed:

1. Deploy Regional Metastore: Account Admin provisions the regional metastore in the primary Azure region.
2. Link All Regional Workspaces: Assign Dev, Staging, and Prod workspaces to the shared regional metastore.
3. Create Environment Catalogs: Provision dev_catalog, staging_catalog, and prod_catalog with dedicated ADLS Gen2 managed storage locations.
4. Apply Workspace Isolation: Set each catalog's isolation mode to ISOLATED and bind them strictly to their corresponding workspace IDs.
5. Configure Read-Only Bindings: If Dev compute requires access to curated enterprise data, bind the prod_curated catalog to the Dev workspace in READ-ONLY mode.
6. Establish RBAC Privileges: Grant schema and table privileges to Entra ID SCIM-synchronized account groups (e.g., dev_data_engineers, prod_service_principals).