5.1 Delta Sharing Architecture & Governance

5.1 Delta Sharing Architecture & Governance

Delta Sharing is an open standard protocol developed by Databricks to enable the secure, real-time exchange of large datasets without the need for data replication. Historically, organizations struggled with data sharing because proprietary platforms forced both the data provider and the data recipient to adopt the exact same technology stack. Alternatively, companies resorted to building complex, error-prone Extract, Transform, Load (ETL) pipelines or relying on file transfers via protocols like SFTP. These approaches necessitated copying data from one location to another, which inevitably resulted in stale data, increased storage and compute costs, and a heightened risk of data breaches.

Delta Sharing solves these longstanding issues by providing a vendor-neutral REST API protocol that abstracts the complexities of the underlying storage. By utilizing this protocol, the data provider grants secure, direct access to the underlying cloud storage objects (such as the Parquet files that comprise a Delta table). This allows the recipient to read the data directly in its native format without requiring a Databricks workspace or any proprietary compute engine. The resulting architecture is incredibly efficient, highly scalable, and completely open, allowing users to collaborate seamlessly across different organizations and clouds.

Delta Sharing Open Protocol Architecture

The Delta Sharing architecture is built around two primary entities: the Data Provider and the Data Recipient.

1. Data Provider: This is the organization or entity that hosts and owns the data. In a Databricks environment, Unity Catalog assumes the role of the Delta Sharing server. The Data Provider is responsible for defining what data is shared, who gets access to it, and under what conditions. They have full control over the lifecycle of the shared assets.
2. Data Recipient: This is the external entity (which could be a user, a team, or a system) that consumes the shared data. The Data Recipient authenticates to the Delta Sharing server using a secure bearer token in order to request access to specific tables, views, or partitions.

The data retrieval workflow is designed for maximum performance and massive scalability. When a Data Recipient requests data, the Delta Sharing server first validates their credentials and permissions. If the request is authorized, the server queries the table's metadata (such as the Delta Lake transaction log) to determine which data files are relevant to the query. Instead of streaming the actual data through the server—which would create a massive network bottleneck—the server returns short-lived, pre-signed cloud storage URLs pointing directly to the underlying Parquet files. The recipient's client then uses these URLs to download the data directly from the cloud storage layer in parallel. This decentralized approach bypasses the Delta Sharing server for the heavy lifting of data transfer, thereby achieving immense throughput and concurrency.

Comprehensive Data Provider Setup

To initiate the sharing of data, a provider must configure two distinct objects in Unity Catalog: a Share and a Recipient. These objects act as the logical constructs governing data access.

1. Creating and Managing a Share

A share is a logical container for the data assets you wish to expose. It can encompass a wide variety of assets, including tables, views, and specific table partitions.

-- Create a new share to organize shared assets
CREATE SHARE IF NOT EXISTS vendor_share
COMMENT 'Secure data share intended for our external vendors';

-- Add an entire table to the share
ALTER SHARE vendor_share 
ADD TABLE main.sales.daily_transactions;

-- Add a specific partition to restrict access
ALTER SHARE vendor_share
ADD TABLE main.sales.historical_data
PARTITION (region = 'North America');

By leveraging partitions or views, Data Providers can implement fine-grained access control. For example, you can filter out Personally Identifiable Information (PII) or restrict a tenant's access to only their specific customer data. This ensures compliance with data privacy regulations while still enabling valuable collaboration. Furthermore, Unity Catalog supports sharing dynamic views, which evaluate the recipient's identity at runtime to filter data dynamically.

2. Creating and Authorizing a Recipient

A recipient represents the verified identity of the external entity consuming the share. Creating a recipient establishes the secure channel for credential exchange.

-- Create a new recipient entity in Unity Catalog
CREATE RECIPIENT external_vendor
COMMENT 'Vendor X analytics team';

-- Grant the recipient explicit read access to the share
GRANT SELECT ON SHARE vendor_share TO RECIPIENT external_vendor;

This clear separation of concerns—where shares manage what is shared and recipients manage who can access it—simplifies administration and allows providers to easily audit and revoke access at any time.

Managing Recipients and Credentials

When you create a recipient in Unity Catalog, the mechanism for managing credentials depends entirely on whether the recipient is utilizing a Databricks workspace (Databricks-to-Databricks sharing) or an external client tool (Open sharing).

Open Sharing (Databricks-to-Open)

For recipients who do not use Databricks, the system employs the open sharing workflow. When the provider executes the CREATE RECIPIENT command, Unity Catalog generates a unique, one-time-use activation URL.

1. Distribution: The provider securely transmits this one-time activation URL to the designated recipient through a trusted out-of-band channel (such as a secure email or messaging system).
2. Activation: The recipient navigates to the URL and downloads a .share credential profile file. This JSON-formatted file contains the Delta Sharing endpoint URL and a secure bearer token.
3. Invalidation: Immediately after the file is successfully downloaded, the activation URL is permanently invalidated. This prevents malicious actors from intercepting the URL and downloading duplicate credentials. If the recipient loses the .share file, or if corporate security policies dictate that the token must be rotated, the provider must manually intervene. They do this by executing the ALTER RECIPIENT external_vendor ROTATE TOKEN command, which generates a brand new activation URL for the recipient to download a fresh credential file.

Databricks-to-Databricks Sharing (D2D)

If the recipient is also operating within a Databricks environment powered by Unity Catalog, the sharing experience becomes significantly more seamless and secure. In this scenario, the recipient provides the Data Provider with a unique Sharing Identifier associated with their Databricks workspace.

The Data Provider creates the recipient using this specific identifier. Once established, Unity Catalog handles all authentication and authorization securely behind the scenes. There are no activation URLs to distribute, no .share files to download or manage, and no manual token rotations to perform. The platform automatically manages the secure connection, dramatically reducing administrative overhead and eliminating the security risks associated with handling static credentials. The shared data automatically appears in the recipient's Unity Catalog as a read-only foreign catalog, ready to be queried using standard Spark SQL.

Consuming Shared Data Across Diverse Clients

One of the most powerful aspects of Delta Sharing is its open protocol nature. Because it relies on standard REST APIs and Parquet files, recipients are not locked into any specific ecosystem. They can utilize a vast array of open-source and commercial tools to consume the data by simply pointing their client to the downloaded .share profile.

For example, a data scientist can effortlessly load a shared table into a pandas DataFrame in Python for local analysis:

import delta_sharing

# Define the path to the downloaded credential profile
profile_path = "/path/to/profile.share"
client = delta_sharing.SharingClient(profile_path)

# List all available tables accessible via this profile
print(client.list_all_tables())

# Load the shared table directly into a pandas DataFrame for immediate analysis
table_url = f"{profile_path}#vendor_share.default.daily_transactions"
pdf = delta_sharing.load_as_pandas(table_url)

Similarly, BI analysts can use native connectors in tools like Microsoft Power BI or Tableau to connect directly to the Delta Sharing endpoint, enabling them to build interactive dashboards on live, shared data without writing any code. Data engineers can use Apache Spark to process massive shared datasets in a distributed manner.

Robust Audit Logging and Governance

When exposing corporate data to external entities, governance, auditing, and security monitoring become paramount concerns. Unity Catalog provides a robust, built-in auditing framework specifically designed for Delta Sharing.

Every single interaction with a shared table is meticulously recorded in the account-level audit logs. This includes events such as a recipient querying a table, a provider creating a share, or a token being rotated. Data Providers can query these comprehensive logs to gain granular visibility into exactly who is accessing their data, which specific tables or partitions were queried, the exact timestamp of the request, and the IP addresses from which the requests originated.

This level of detailed visibility ensures that organizations can confidently meet strict regulatory compliance requirements (such as GDPR or HIPAA) and quickly detect any anomalous or unauthorized access patterns. Furthermore, providers can implement IP access lists to restrict Delta Sharing access to known, trusted IP ranges, adding an additional layer of network-level security to their data collaboration initiatives.