3.3 Ingesting Data via Delta Sharing & External Connections

3.3 Ingesting Data via Delta Sharing & External Connections

In traditional data architecture, acquiring external datasets or integrating with remote transactional databases required complex Extract, Transform, Load (ETL) pipelines. These pipelines created physical data copies, leading to data staleness, ballooning cloud storage costs, security vulnerabilities, and governance complexity. Modern data intelligence platforms solve these issues by introducing zero-copy data sharing and lakehouse federation. Databricks provides Delta Sharing for secure cross-organization data exchange and Lakehouse Federation for querying external database systems directly in-place.

---

Zero-Copy Data Access Paradigm

The zero-copy data access model eliminates data movement entirely. Instead of copying gigabytes or terabytes of data across cloud buckets or database instances, analysts query remote datasets live at the source.

Traditional ETL Model:
Source DB / Provider ---> [Extract File] ---> [Network Transit] ---> [Load to Lakehouse] ---> Query Data
(High latency, duplicate storage costs, stale data)

Zero-Copy Sharing & Federation Model:
Source DB / Provider <------------------- Live Direct Query ------------------- Analyst Workspace
(Zero storage duplication, real-time access, centralized governance)

Key advantages of zero-copy architectures include:

• Immediate Data Freshness: Queries reflect source updates instantaneously without waiting for batch ETL completion.
• Zero Storage Overhead: Recipients do not pay cloud storage fees for duplicated data.
• Centralized Security: Providers maintain complete control and can revoke access instantly.

---

Delta Sharing Protocol & Architecture

Delta Sharing is an open-source protocol developed by Databricks and donated to the Linux Foundation. It enables secure real-time data sharing across organizations regardless of the underlying cloud platform, computing framework, or data storage engine.

Open Sharing vs. Databricks-to-Databricks Sharing

Delta Sharing operates in two primary modes:

1. Databricks-to-Databricks Sharing: When both provider and recipient use Databricks with Unity Catalog, sharing occurs natively using Unity Catalog Metastore IDs. No credential files or bearer tokens are exchanged. Databricks handles authentication seamlessly, allowing recipients to mount shared data as Unity Catalog catalogs instantly.
2. Databricks-to-Open Sharing: When recipients use non-Databricks platforms (such as Microsoft Power BI, Tableau, Python pandas, Apache Spark, or Excel), access is governed through a secure open REST API. The provider generates a credential file (.share) containing a short-lived bearer token. The recipient's application uses the token to authenticate with the Delta Sharing server, which returns pre-signed cloud storage URLs for reading underlying Parquet/Delta data files directly from cloud storage.

---

Governed Data Sharing in Unity Catalog

Unity Catalog serves as the control plane for Delta Sharing, organizing sharing entities into a structured hierarchy:

• Share: A logical container created by a data provider in Unity Catalog that groups read-only assets to be shared. A share can contain Delta tables, materialized views, standard views, dynamic views, and Unity Catalog Volumes.
• Provider: An entity representing the data provider who creates shares and manages recipient permissions.
• Recipient: An entity representing an external consumer authorized to access specific shares.

Shared Object Capabilities

Providers can share complex assets beyond raw tables:

• Views with Row Filters & Column Masks: Providers can enforce fine-grained access control before data leaves their metastore, ensuring recipients only see records permitted by row predicates or masked columns.
• Unity Catalog Volumes: Allows sharing non-tabular datasets, such as machine learning model checkpoints, image files, or PDF documents.

Analyst Workflow for Accessing Shared Data

To consume a Delta Share, a Databricks data analyst inspects available shares and mounts them directly into their local Unity Catalog namespace:

-- View shares published by an external provider
SHOW SHARES IN PROVIDER external_vendor_provider;

-- Create a local catalog mapped to the external share
CREATE CATALOG vendor_analytics_cat
USING SHARE external_vendor_provider.market_intelligence_share;

-- Query shared tables live with zero copy overhead
SELECT 
  region, 
  product_category, 
  total_market_revenue 
FROM vendor_analytics_cat.market_data.regional_summaries
WHERE summary_year = 2026;

---

Lakehouse Federation for External Databases

While Delta Sharing connects organizations across lakehouses, Lakehouse Federation enables data analysts to query external relational database management systems (RDBMS) and data warehouses directly from Databricks SQL Warehouses without moving data.

Lakehouse Federation supports connectivity to popular database engines including PostgreSQL, MySQL, Snowflake, Amazon Redshift, Google BigQuery, Microsoft SQL Server, and Oracle.

Unity Catalog Federation Entities

Lakehouse Federation integrates external database systems into Unity Catalog's 3-level namespace using two key abstractions:

1. Connection: A Unity Catalog object that encapsulates connection details and credentials (hostname, port, username, password/secrets) for an external database system.
2. Foreign Catalog: A Unity Catalog catalog object that maps directly to a database inside the remote system. All schemas, tables, and views in the remote database become visible inside Unity Catalog as read-only objects under foreign_catalog.schema_name.table_name.

-- Step 1: Create connection to remote PostgreSQL database
CREATE CONNECTION postgres_prod_db
TYPE POSTGRESQL
OPTIONS (
  host 'postgres-prod.internal.net',
  port '5432',
  user secret('scope_credentials', 'pg_user'),
  password secret('scope_credentials', 'pg_pass')
);

-- Step 2: Create Foreign Catalog mapping to remote database
CREATE FOREIGN CATALOG pg_sales_catalog
USING CONNECTION postgres_prod_db
OPTIONS (database 'sales_production');

-- Step 3: Query remote database directly inside Databricks SQL
SELECT customer_id, order_status, total_amount
FROM pg_sales_catalog.public.orders
WHERE order_date >= '2026-01-01';

---

Query Pushdown Optimization

A key engineering challenge of federated queries is network latency and data transfer overhead. Lakehouse Federation solves this through Query Pushdown Optimization.

When an analyst runs a SQL query against a Foreign Catalog, the Databricks SQL query optimizer evaluates the query execution plan and pushes filtering (WHERE), projections (SELECT), joins, and aggregations (GROUP BY) down to the source database engine.

Databricks SQL Warehouse                    External Database (PostgreSQL / Snowflake)
├── Executes Query ------------------------> Receives Pushed-Down SQL Query
│                                            ├── Executes filter & aggregation locally
└── Receives Filtered Result Set <-----------└── Returns small aggregated summary table

By processing filters and aggregations at the source engine, Lakehouse Federation transfers only the compact final result set across the network, minimizing latency and compute costs.

---

Comparative Matrix: Ingestion & Sharing Patterns

Analysts must select the appropriate integration strategy based on data location, update frequency, and architecture: