6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks

6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks

DP-750 Exam Focus: Understand the open Delta Sharing protocol architecture, contrasting Open Sharing (credential profile JSON, bearer tokens, REST API for Power BI/pandas/Spark) with Databricks-to-Databricks (Managed) Sharing (native Metastore Global Sharing ID, zero-token management). Master the SQL lifecycle for CREATE SHARE, CREATE RECIPIENT, partition filtering, Change Data Feed sharing, and security governance (token expiration, IP access lists, and audit logging).

---

1. The Delta Sharing Paradigm & Zero-Copy Architecture

Traditional data sharing across organizations or business units relied on inefficient, error-prone patterns:

• Nightly SFTP / Cloud Storage Drops: High ETL maintenance, duplicate cloud storage costs, and severe data staleness.
• Vendor-Locked Sharing: Proprietary data sharing platforms require both provider and consumer to license the identical closed database software.
• Direct Cloud IAM Delegation: Sharing cloud storage containers directly via cross-account Azure IAM/SAS tokens introduces high security risk and exposes all unmanaged files in the bucket.

Delta Sharing is an open-source, vendor-neutral standard developed by Databricks and the Linux Foundation. It enables organizations to share live, read-only Delta Lake tables, views, volumes, and models across any cloud (Azure, AWS, GCP), region, or client platform without data copying or replication.

+-----------------------------------------------------------------------------------+
|                         DELTA SHARING CORE PROTOCOL FLOW                          |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|    [ Recipient Client ]                 [ Provider Unity Catalog ]                |
|  (Power BI / Pandas / Spark)               (Governance Metastore)                 |
|             |                                       |                             |
|             |  1. HTTPS Query + Bearer Token / ID   |                             |
|             |-------------------------------------->|                             |
|             |                                       | (Validates permissions &    |
|             |                                       |  partition filters)         |
|             |  2. Returns Parquet Metadata +        |                             |
|             |     Short-Lived Azure SAS URLs        |                             |
|             |<--------------------------------------|                             |
|             |                                                                     |
|             |  3. Direct Read via TLS / ABFSS Driver                              |
|             +---------------------------------------------------+                 |
|                                                                 |                 |
|                                                                 v                 |
|                                                  [ Provider ADLS Gen2 Storage ]   |
|                                                  (Underlying Parquet Data Files)  |
+-----------------------------------------------------------------------------------+

Why Delta Sharing Does Not Suffer Compute Bottlenecks

In Delta Sharing, the provider's compute clusters are not involved in streaming data to the consumer.

1. The recipient makes an HTTPS request to the provider's Unity Catalog server.
2. Unity Catalog validates authorization, evaluates partition filters, and generates short-lived presigned Azure SAS URLs (typically valid for 15 minutes to 1 hour) for the specific Parquet files needed.
3. The recipient client uses these URLs to read data directly from the provider's Azure Data Lake Storage Gen2 account over high-throughput cloud network backbones. The provider incurs zero query compute costs.

---

2. Open Delta Sharing (External Non-Databricks Consumers)

Open Delta Sharing enables external recipients who do not use Azure Databricks to consume live data directly within their preferred analytical environments (e.g., Microsoft Power BI, Python Pandas, Apache Spark, DuckDB, Tableau, Excel, or custom REST applications).

                               OPEN DELTA SHARING LIFECYCLE

  1. Provider Creates Recipient
  +-------------------------------------------------------------------------+
  | CREATE RECIPIENT external_partner;                                      |
  | (Generates one-time activation URL with secure token download)          |
  +-------------------------------------------------------------------------+
                                     |
                                     v
  2. Consumer Downloads Credential Profile File (`config.share`)
  +-------------------------------------------------------------------------+
  | {                                                                       |
  |   "shareCredentialsVersion": 1,                                         |
  |   "bearerToken": "dapi891240abc98412...",                               |
  |   "endpoint": "https://eastus.azuredatabricks.net/api/2.0/delta-sharing/"|
  |   "expirationTime": "2027-01-01T00:00:00Z"                              |
  | }                                                                       |
  +-------------------------------------------------------------------------+
                                     |
                                     v
  3. Client Consumes Data Programmatically
  +-------------------------------------------------------------------------+
  | import delta_sharing                                                    |
  | profile = "/path/to/config.share"                                       |
  | df = delta_sharing.load_as_pandas(f"{profile}#share_sales.gold.orders")  |
  +-------------------------------------------------------------------------+

Open Sharing Client Ecosystem

# Consuming Delta Sharing via Apache Spark
spark_df = (spark.read.format("deltaSharing")
            .load("/secrets/config.share#partner_share.analytics.quarterly_revenue"))

# Consuming Delta Sharing via Python Pandas
import delta_sharing
client = delta_sharing.SharingClient("/secrets/config.share")
shares = client.list_shares()
pandas_df = delta_sharing.load_as_pandas("/secrets/config.share#partner_share.analytics.quarterly_revenue")

Security Controls for Open Sharing

• One-Time Activation URL: The activation URL can only be accessed once to download the credential profile JSON file. Once downloaded, the activation link expires permanently.
• Token Rotation & Expiration: Providers configure strict token expiration horizons (e.g., 30, 60, or 90 days) using ALTER RECIPIENT ... SET TOKEN_EXPIRATION_DAYS = 90; and can rotate tokens instantly.
• IP Access Lists: Providers restrict recipient queries to approved corporate CIDR IP blocks:

  ALTER RECIPIENT external_partner SET IP_ACCESS_LIST ('198.51.100.0/24', '203.0.113.50/32');
  

---

3. Databricks-to-Databricks (Managed) Delta Sharing

When both the data provider and consumer use Azure Databricks with Unity Catalog, they utilize Databricks-to-Databricks Managed Sharing.

                    DATABRICKS-TO-DATABRICKS MANAGED SHARING

  [ Provider Unity Catalog Metastore ]        [ Consumer Unity Catalog Metastore ]
  Metastore ID: azure:eastus:prov-uuid        Metastore ID: azure:westus:cons-uuid
                 |                                             |
                 +---------------------------------------------+
                                 |
                 (Direct UC-to-UC Trust Federation)
                 - No Credential JSON Files
                 - Zero Bearer Token Management
                 - Shares Mount Natively as Catalogs

The Metastore Global Sharing Identifier

In Databricks-to-Databricks sharing, no credential profile files or bearer tokens are created. Authentication occurs via mutual cryptographic verification between Unity Catalog metastores using the recipient's Global Sharing Identifier:

Sharing Identifier Format: <cloud>:<region>:<metastore-uuid> Example: azure:eastus:8f7e2a10-3b4c-4d5e-9f0a-1a2b3c4d5e6f

Creating a Managed Recipient

-- Provider executes in Unity Catalog:
CREATE RECIPIENT enterprise_subsidiary 
USING ID 'azure:westus:8f7e2a10-3b4c-4d5e-9f0a-1a2b3c4d5e6f';

Consumer Mounting Shared Catalogs Natively

Once the provider grants access to a share, the recipient's Metastore Admin mounts the share as a native catalog inside their workspace:

-- Consumer Metastore Admin executes in recipient workspace:
CREATE CATALOG shared_finance_hub 
USING SHARE provider_org_metastore.finance_share;

-- Consumer users can now query shared tables using standard 3-level namespace!
SELECT * FROM shared_finance_hub.gold.monthly_pnl;

---

4. Comparison Matrix: Open Sharing vs. Databricks-to-Databricks

---

5. Provider-Side Object Lifecycle & Governance

To share data securely, data providers create and manage three core Unity Catalog securables: SHARE, RECIPIENT, and PROVIDER.

-- Step 1: Create the SHARE object container
CREATE SHARE regional_sales_share
COMMENT 'Live sales data for regional retail partners';

-- Step 2: Add tables to the share with optional partition filtering
ALTER SHARE regional_sales_share ADD TABLE prod_sales.gold.orders
COMMENT 'Consolidated global orders';

-- Step 2b: Add a table with partition filtering (Restricting to US and CA regions)
ALTER SHARE regional_sales_share ADD TABLE prod_sales.gold.regional_inventory
AS prod_sales.gold.regional_inventory
WITH (PARTITION (country = 'US'), PARTITION (country = 'CA'));

-- Step 2c: Share Delta Change Data Feed (CDF) for incremental synchronization
ALTER SHARE regional_sales_share ADD TABLE prod_sales.gold.pricing_updates
WITH CHANGE DATA FEED;

-- Step 3: Grant SELECT on the SHARE to the RECIPIENT
GRANT SELECT ON SHARE regional_sales_share TO RECIPIENT external_partner;

Auditing Shared Data Consumption

Every data read request initiated by Delta Sharing recipients is logged in system.access.audit under service_name = 'deltaSharing', recording recipient identifiers, IP addresses, accessed tables, and byte transfer volumes.