1.2 Databricks Marketplace & Data Sharing Basics

Databricks Marketplace & Data Sharing Basics

In modern enterprise data analytics, data isolation creates severe business friction. Traditionally, sharing data between organizations, business units, or different cloud platforms required complex Extract, Transform, Load (ETL) pipelines, File Transfer Protocol (FTP) servers, or custom API endpoints. These legacy methods introduce data duplication, stale datasets, security vulnerabilities, and significant engineering maintenance costs. The Databricks Data Intelligence Platform addresses these challenges through Delta Sharing and the Databricks Marketplace.

---

The Evolution of Open Data Sharing

Modern data architectures require zero-copy data sharing. Instead of copying gigabytes or terabytes of data across cloud storage accounts, modern architectures share direct access to underlying data files governed by strict security policies.

+-----------------------------------------------------------------------+
|                         DELTA SHARING ARCHITECTURE                    |
|                                                                       |
|   +-------------------+    REST Protocol    +---------------------+   |
|   |  Data Provider    |  =================> |    Data Recipient   |   |
|   |  (Unity Catalog)  |    Short-Lived SAS  |  (Databricks, Spark, |   |
|   |                   |    Presigned URLs   |   Pandas, Power BI) |   |
|   +-------------------+                     +---------------------+   |
|             |                                          |              |
|             v                                          v              |
|   +-------------------+                     +---------------------+   |
|   | Cloud Storage S3  | <------------------ | Read Direct Parquet/ |   |
|   | (Provider Account)|  Direct Data Fetch  | Delta Data Files    |   |
|   +-------------------+                     +---------------------+   |
+-----------------------------------------------------------------------+

Open vs. Proprietary Sharing Protocols

• Proprietary Sharing: Many traditional cloud data warehouses allow data sharing only if both the data provider and data recipient use the exact same vendor product in the exact same cloud region.
• Open Sharing (Delta Sharing): Databricks created Delta Sharing as an open-source REST protocol. Recipients do not need to use Databricks, run on the same cloud platform, or subscribe to any vendor lock-in. Recipients can query shared data natively using open tools like Apache Spark, Python Pandas, Microsoft Power BI, Tableau, Excel, or Rust.

---

Delta Sharing Protocol & Core Architecture

Delta Sharing operates natively within Unity Catalog. It establishes a secure channel between a Data Provider and a Data Recipient using short-lived pre-signed URLs.

Key Architectural Entities

1. Share: A logical container created by a Data Provider inside Unity Catalog. A Share groups together data assets intended for distribution. A single Share can contain:
   • Delta Lake Tables (read-only)
   • Table Partitions & Dynamic Views
   • Volumes (non-tabular files like PDFs or CSVs)
   • AI/ML Models and Notebooks
2. Recipient: A object in Unity Catalog representing an external user, team, or organization that receives data. Unity Catalog tracks recipient authentication credentials.
3. Provider: An object representing the entity sharing the data.

How Delta Sharing Executes Under the Hood

When a Data Recipient queries a shared Delta table:

1. The Recipient client sends an authentication request to the Provider's Delta Sharing server (managed by Unity Catalog).
2. Unity Catalog verifies privileges, parses the Delta table commit log, and generates short-lived, pre-signed cloud storage URLs (e.g., AWS S3 pre-signed URLs valid for 1 hour).
3. The Recipient client reads the underlying Parquet/Delta data files directly from the Provider's cloud object storage using those pre-signed URLs.
4. No data is copied, and compute costs for executing the query are borne entirely by the Recipient.

---

Databricks Marketplace: Discovery & Monetization

The Databricks Marketplace is an open commercial and public data exchange built directly on top of Delta Sharing. It allows data providers to showcase, distribute, and monetize data products, while providing data analysts with immediate access to third-party datasets.

Types of Marketplace Assets

Unlike traditional data exchanges that only offer static tabular CSV files, Databricks Marketplace supports a rich variety of intelligence assets:

• Tabular Datasets: Financial market feeds, geographic demography, point-of-sale retail data, and weather telemetry.
• AI / ML Models: Pre-trained machine learning models and large language model (LLM) weights.
• Databricks Notebooks & Solution Accelerators: Ready-to-run analytical code, visualization templates, and industry workflow reference guides.
• AI/BI Genie Spaces: Pre-curated conversational natural language analytics environments ready for instant querying.

Marketplace Exchange Listings

Listings on Databricks Marketplace fall into two categories:

1. Standard (Instant) Listings: Free or public datasets that consumers can instantly request and mount into their Unity Catalog metastore with a single click.
2. Personalized Listings: Commercial datasets requiring provider approval, contract negotiation, or customization before access is granted.

---

Provider vs. Recipient Experience

Sharing data in Databricks requires zero ETL setup. Administrators use standard Data Control Language (DCL) SQL commands to govern shares.

Databricks-to-Databricks Sharing vs. Databricks-to-Open Sharing

Databricks supports two recipient authentication models:

• Databricks-to-Databricks (D2D): If the recipient also uses Databricks with Unity Catalog, sharing is seamless. The provider shares data using the recipient's unique Unity Catalog Metastore Identifier. The shared catalog appears natively in the recipient's Catalog Explorer without token management.
• Databricks-to-Open (D2O): If the recipient uses an external non-Databricks platform (such as Python scripts or Power BI), Unity Catalog generates a secure one-time activation link. The recipient downloads a secure credential file (.share credential file containing a bearer token) used to authenticate REST API calls.

Managing Shares with SQL

Data Analysts and Data Governance Leads inspect and manage shares using standard Databricks SQL commands:

sql\n-- Create a share object in Unity Catalog\nCREATE SHARE regional_sales_share\nCOMMENT 'Contains quarterly sales metrics for external distributors';\n\n-- Add Delta tables to the share\nALTER SHARE regional_sales_share ADD TABLE main.sales_schema.quarterly_summary;\n\n-- Create a recipient entity for an external distributor\nCREATE RECIPIENT distributor_acme\nUSING ID 'aws:us-west-2:a1b2c3d4-e5f6-7890-1234-56789abcdef0';\n\n-- Grant read access on the share to the recipient\nGRANT SELECT ON SHARE regional_sales_share TO RECIPIENT distributor_acme;\n\n-- Inspect active shares and detailed metadata\nSHOW SHARES;\nDESCRIBE SHARE regional_sales_share;\n

---

Data Clean Rooms: Secure Multi-Party Collaboration

For privacy-sensitive industries (such as healthcare, banking, and retail), sharing raw datasets is legally prohibited due to regulations like GDPR and HIPAA. Databricks Clean Rooms provide a secure, privacy-preserving environment where multiple parties collaborate without sharing raw data.

Clean Room Key Principles

• Zero Raw Data Exposure: Participants can run joint analytical queries (e.g., matching retail purchase records against advertising exposure logs), but neither party can see the other party's raw records.
• Privacy Overlap Queries: Clean rooms enforce differential privacy algorithms and minimum aggregate threshold restrictions to prevent reverse-engineering of individual identity records.
• No Cloud Co-location Required: Built on Delta Sharing, Clean Room participants can reside in different cloud providers (e.g., AWS vs. Azure) and different metastores seamlessly.

---

Delta Sharing & Marketplace Comparison Table