5.4 Secure Data Sharing within Azure Databricks

5.4 Secure Data Sharing within Azure Databricks

DP-750 Exam Focus: Master internal data sharing using Delta Sharing in Azure Databricks. Understand the architectural mechanics of Databricks-to-Databricks (UC-to-UC) sharing, including the Provider and Recipient model, creating and managing Shares (CREATE SHARE, ALTER SHARE), configuring Recipients with Sharing Identifiers, mounting shares into consumer Unity Catalog metastores (CREATE CATALOG ... USING SHARE), and sharing advanced securables like Dynamic Views, Volumes, and Change Data Feed (CDF).

---

1. Internal Enterprise Data Sharing: The Zero-Copy Paradigm

Historically, when business units, regional branches, or distinct Azure Databricks workspaces needed to share datasets, engineering teams relied on data replication pipelines:

                               LEGACY REPLICATION ANTI-PATTERN
                               
  [ Provider Workspace ]                                     [ Consumer Workspace ]
  +--------------------+                                     +--------------------+
  | Source Delta Table | ---> [ Azure Data Factory / ETL ] --> | Copy Delta Table   |
  | (10 TB Dataset)    |      - Scheduled Batch Copy         | (Storage Duplicate)|
  +--------------------+      - Network Egress Billed        +--------------------+
                               - Stale Data (Sync Lag)         - 2x Storage Cost
                               - Complex Error Recovery        - Out-of-Sync Lineage

This replication approach introduces severe operational liabilities:

• Storage Duplication & High Cloud Costs: Paying double or triple storage costs for identical multi-terabyte datasets.
• Network Egress Fees: Continuous cross-region data transfer incurs significant network bandwidth charges.
• Data Drift & Stale Analytics: Downstream consumers work on snapshot copies that lag hours or days behind production updates.
• Governance Gaps: Compliance teams lose track of duplicate datasets scattered across disparate storage accounts.

Delta Sharing is an open, secure protocol developed by Databricks for real-time, zero-copy data exchange. Within Azure Databricks, Databricks-to-Databricks (UC-to-UC) Sharing allows a data provider to grant direct, governed query access on live Delta tables to consumer workspaces—across regions, subscriptions, and Entra ID tenants—without copying a single byte of underlying data.

---

2. Databricks-to-Databricks (UC-to-UC) Sharing Architecture

In Databricks-to-Databricks sharing, both the data provider and data consumer operate Unity Catalog-enabled Azure Databricks environments. Authentication is handled natively through Unity Catalog trust handshakes, eliminating the need to generate or exchange external credential activation URLs or bearer token files.

+-----------------------------------------------------------------------------------------+
|                       DATABRICKS-TO-DATABRICKS (UC-TO-UC) SHARING                       |
+-----------------------------------------------------------------------------------------+
|                                                                                         |
|  PROVIDER ENVIRONMENT (Metastore East US):                                              |
|  1. Data Provider creates a Share:                                                      |
|     CREATE SHARE finance_share;                                                         |
|     ALTER SHARE finance_share ADD TABLE prod.finance.fact_revenue;                      |
|                                                                                         |
|  2. Data Provider creates a Recipient using Consumer's Sharing Identifier:              |
|     CREATE RECIPIENT marketing_team_recipient                                           |
|     USING ID 'azure:eastus:11111111-2222-3333-4444-555555555555:metastore-id';         |
|                                                                                         |
|  3. Provider Grants Share Access:                                                       |
|     GRANT SELECT ON SHARE finance_share TO RECIPIENT marketing_team_recipient;          |
|                                                                                         |
|  -------------------------------------------------------------------------------------  |
|                                                                                         |
|  CONSUMER ENVIRONMENT (Metastore West Europe / Other Workspace):                        |
|  4. Consumer discovers shared data & creates a local catalog:                           |
|     CREATE CATALOG shared_finance USING SHARE provider_metastore.finance_share;         |
|                                                                                         |
|  5. Consumer Queries Data Live (Zero-Copy):                                             |
|     SELECT * FROM shared_finance.finance.fact_revenue;                                  |
+-----------------------------------------------------------------------------------------+

Provider vs. Consumer Responsibilities

---

3. Core Delta Sharing Securables & DDL Syntax

Setting up UC-to-UC sharing involves four primary objects: Shares, Securables in the Share, Recipients, and Share Grants.

                               DELTA SHARING OBJECT MODEL
                               
  +-------------------------------------------------------------------------+
  |                         Unity Catalog Provider                          |
  |                                                                         |
  |  +-------------------------------------------------------------------+  |
  |  |                          SHARE: sales_share                       |  |
  |  |  - Table: prod.sales.customers (AS client_directory)              |  |
  |  |  - Table: prod.sales.orders (WITH CHANGE DATA FEED)                |  |
  |  |  - Table: prod.sales.regional_metrics (PARTITION: region = 'EMEA') |  |
  |  |  - Volume: prod.sales.raw_contracts                              |  |
  |  +-------------------------------------------------------------------+  |
  |                                   |                                     |
  |                                   | GRANT SELECT                        |
  |                                   v                                     |
  |  +-------------------------------------------------------------------+  |
  |  |                 RECIPIENT: analytics_division_recipient            |  |
  |  |                 (Bound via Metastore Sharing Identifier)           |  |
  |  +-------------------------------------------------------------------+  |
  +-------------------------------------------------------------------------+

Step 1: Create a Share

A Share is a container for data assets managed by the provider metastore:

CREATE SHARE global_sales_share
COMMENT 'Live enterprise sales and customer metrics shared with regional analytics teams';

Step 2: Add Tables and Partitions to the Share

Providers can add full tables, table aliases, partitioned slices, or tables enabled for Change Data Feed (CDF):

-- Method A: Add an entire table
ALTER SHARE global_sales_share 
ADD TABLE prod.sales.customer_dimension;

-- Method B: Add a table with a sanitized alias (hides internal schema name)
ALTER SHARE global_sales_share 
ADD TABLE prod.sales.fact_transactions AS sales_data.transactions;

-- Method C: Add a specific partition slice (Partition Sharing)
ALTER SHARE global_sales_share 
ADD TABLE prod.sales.regional_inventory 
PARTITION (inventory_region = 'EMEA');

-- Method D: Share with Change Data Feed (CDF) enabled
ALTER SHARE global_sales_share 
ADD TABLE prod.sales.order_events 
WITH CHANGE DATA FEED;

Step 3: Add Volumes and Models to the Share

Delta Sharing is not restricted to tabular data; providers can share Volumes (unstructured files) and ML Models:

-- Share an unstructured Unity Catalog Volume containing market research PDFs
ALTER SHARE global_sales_share 
ADD VOLUME prod.marketing.research_reports;

-- Share a registered MLflow model
ALTER SHARE global_sales_share 
ADD MODEL prod.ml_models.churn_prediction_v2;

Step 4: Register the Recipient

In Databricks-to-Databricks sharing, the consumer provides their Sharing Identifier (obtained from their local Unity Catalog metastore). The provider creates the recipient object using this identifier:

-- Create recipient using the consumer metastore sharing ID
CREATE RECIPIENT emea_analytics_recipient
USING ID 'azure:westeurope:a1b2c3d4-e5f6-7890-abcd-ef1234567890:metastore-998877'
COMMENT 'Recipient for West Europe analytics team';

Step 5: Grant Share Permissions to the Recipient

GRANT SELECT ON SHARE global_sales_share 
TO RECIPIENT emea_analytics_recipient;

---

4. Consumer Experience: Mounting a Share into Unity Catalog

On the consumer side, the recipient accesses the shared data through standard Unity Catalog SQL commands without needing access keys or API credentials.

Step 1: Discover Available Providers and Shares

-- List all data providers who have shared data with this metastore
SHOW PROVIDERS;

-- Inspect shares offered by a specific provider
SHOW SHARES IN PROVIDER eastus_finance_provider;

Step 2: Create a Local Catalog from the Share

The consumer creates a new local catalog that points directly to the provider's share:

-- Mount the share into the consumer's 3-level namespace
CREATE CATALOG shared_global_sales 
USING SHARE eastus_finance_provider.global_sales_share;

-- Grant access to internal consumer groups
GRANT USE CATALOG ON CATALOG shared_global_sales TO `emea_data_analysts`;
GRANT USE SCHEMA, SELECT ON SCHEMA shared_global_sales.sales_data TO `emea_data_analysts`;

Step 3: Query Shared Tables In-Place

Consumers can run standard ANSI SQL queries, Spark DataFrames, and cross-catalog joins:

-- Query the shared Delta table live
SELECT 
  c.customer_name,
  t.transaction_id,
  t.transaction_amount,
  t.transaction_date
FROM shared_global_sales.sales_data.transactions t
INNER JOIN local_emea_catalog.analytics.dim_store s
  ON t.store_id = s.store_id
WHERE t.transaction_date >= '2026-01-01';

---

5. Advanced Sharing Capabilities: Dynamic Views & Change Data Feed

1. Sharing Governed Dynamic Views

When sharing data with external departments, providers often need to apply row filters and column masks. In Delta Sharing, providers can add Dynamic Views to a share. When consumers query the view, the provider's masking logic evaluates transparently, guaranteeing that consumers only see redacted or authorized subsets.

-- Provider creates a sanitized view
CREATE VIEW prod.sales.sanitized_orders_view AS
SELECT 
  order_id,
  customer_id,
  mask_credit_card(credit_card_num) AS credit_card_num,
  order_amount
FROM prod.sales.orders
WHERE order_status != 'CANCELLED';

-- Provider adds view to share
ALTER SHARE global_sales_share 
ADD VIEW prod.sales.sanitized_orders_view;

2. Consuming Change Data Feed (CDF) Incrementally

When a table is shared WITH CHANGE DATA FEED, downstream consumer pipelines can read incremental inserts, updates, and deletes using Structured Streaming or batch table_changes():

-- Consumer queries table changes incrementally between commit versions
SELECT 
  order_id,
  order_amount,
  _change_type,
  _commit_version,
  _commit_timestamp
FROM table_changes('shared_global_sales.sales_data.transactions', 105, 120);

# PySpark Structured Streaming consumer reading live updates from shared table
df_stream = (spark.readStream
  .format("deltaSharing")
  .option("readChangeFeed", "true")
  .option("startingVersion", 100)
  .table("shared_global_sales.sales_data.transactions"))

---

6. Security, Auditing, & Delta Sharing Governance

Delta Sharing provides enterprise security and end-to-end auditability:

• Audit Logging via system.access.audit: Every query executed by a recipient against a shared table is logged in the provider's Unity Catalog audit logs (system.access.audit). The logs record the recipient name, IP address, queried table, timestamp, and accessed partitions.
• Direct Storage Authorization: When a consumer cluster queries a shared table, Databricks generates short-lived, down-scoped SAS URL tokens that allow the consumer's Spark executors to stream Parquet chunks directly from the provider's ADLS Gen2 storage over TLS without passing through third-party servers.
• Instant Revocation: Executing REVOKE SELECT ON SHARE global_sales_share FROM RECIPIENT emea_analytics_recipient; terminates access immediately. All active queries on the consumer cluster fail upon their next batch token refresh.