5.3 Zero-Copy Data Federation & Snowflake/BigQuery/Databricks Sharing

Zero-Copy Data Federation & Snowflake/BigQuery/Databricks Sharing

For decades, enterprise data engineering has been trapped in the "Copy Problem". When an organization needed enterprise data warehouse records inside a customer data platform, data teams built complex Extract, Transform, Load (ETL) or Reverse ETL pipelines: copying gigabytes or terabytes of data across cloud boundaries, duplicating storage costs, establishing brittle scheduled sync jobs, and introducing hours or days of synchronization lag.

Salesforce Data Cloud solves this fundamental architectural bottleneck through Zero-Copy Data Architecture. By leveraging open table formats like Apache Iceberg and Delta Lake, Data Cloud can federate queries directly against external cloud data platforms—including Snowflake, Google BigQuery, Databricks, and AWS Redshift—and share Data Cloud Customer 360 data back to those platforms with zero physical data movement.

---

Principles of Zero-Copy Architecture vs. Traditional ETL/ELT

To succeed on the Data Cloud Consultant exam, you must distinguish between physical data ingestion and virtual data federation.

Traditional ETL/ELT Pipeline (Physical Copy):
[External Warehouse] --> [Nightly Batch Extract] --> [S3 / GCS Staging] --> [Data Cloud Ingest] --> [Duplicate Storage]
  * High Latency (Hours/Days)    * High Egress & Storage Costs    * Brittle Pipeline Maintenance

Zero-Copy Inbound Federation (Virtual Access):
[External Warehouse] <---------------- Query Pushdown ----------------- [Salesforce Data Cloud]
 (Snowflake/BigQuery) ----------------- Iceberg / Arrow Metadata -------> (Virtual DLO / Canonical DMO)
  * Zero Data Movement           * Zero Duplicate Storage         * Near-Real-Time Data Freshness

The Core Pillars of Zero-Copy

1. Data Virtualization: Rather than copying raw bytes from an external table into Data Cloud's Hyperforce lakehouse storage, Data Cloud catalogs the external table's metadata and schema definition. It registers a Virtual Data Lake Object (vDLO) that behaves identically to a standard physical DLO within the platform.
2. Query Pushdown & Distributed Execution: When a user runs a segment, builds a Calculated Insight, or executes a query involving a vDLO, Data Cloud's query engine translates the operation into a native SQL query and pushes execution down into the external data platform's compute engine. Only the final aggregated or filtered result set is returned across the secure network link.
3. Open Table Formats (Apache Iceberg & Delta Sharing): Zero-Copy relies on open table standards. By utilizing Apache Iceberg metadata specifications, Data Cloud and modern lakehouses speak a universal, language-agnostic data representation that decouples storage from compute.

Comparative Architectural Matrix: ETL Ingestion vs. Zero-Copy Federation

---

Inbound Data Federation: Snowflake, BigQuery, Databricks & Redshift

Inbound Data Federation allows Data Cloud to query external enterprise datasets in-place. The external tables appear directly within Data Cloud as Virtual Data Lake Objects (vDLOs), allowing consultants to map them to standard Customer 360 Data Model Objects (DMOs).

1. Snowflake Inbound Federation

• Mechanics: Utilizes Snowflake's support for Apache Iceberg Tables and cross-cloud data sharing.
• Connection: Established via a secure Snowflake Partner Integration using OAuth or key-pair authentication. Data Cloud reads Snowflake metadata catalogs and executes pushed-down SQL against a designated Snowflake Virtual Warehouse.
• Use Case: Ingesting 10 years of historical Point-of-Sale (POS) transactions or financial ledger data directly into the Sales Order canonical DMO without duplicating 500 million historical rows into Data Cloud.

2. Google BigQuery Inbound Federation

• Mechanics: Integrates with Google Cloud via the BigQuery Storage Read API and Google Cloud IAM service account federation.
• Use Case: Quoting real-time ad campaign performance metrics, Google Analytics 4 (GA4) raw event tables, or enterprise data warehouse tables directly inside Data Cloud segmentation without building GCS export pipelines.

3. Databricks Inbound Federation

• Mechanics: Leverages the open Delta Sharing protocol and Databricks Unity Catalog. Data Cloud securely connects to Databricks recipients, querying Delta Lake tables natively.
• Use Case: Accessing machine learning feature stores, churn propensity scores, and product recommendation matrices generated by Databricks data science pipelines.

4. Amazon Redshift Inbound Federation

• Mechanics: Connects to AWS Redshift clusters or Redshift Serverless via the Amazon Redshift Data API and AWS IAM cross-account authentication.

---

Outbound Data Sharing: Exposing Data Cloud Customer 360 Assets

Zero-Copy is truly bi-directional. While Inbound Federation brings warehouse data into Data Cloud, Outbound Data Sharing allows enterprise data scientists, business intelligence teams, and data engineers to access Data Cloud's harmonized Customer 360 assets directly from their external analytics platform without reverse-ETL tools or file exports.

┌────────────────────────────────────────────────────────────────────────┐
│                      Salesforce Data Cloud Tenant                      │
│  - Unified Individual Profiles (Consolidated Identity Graph)           │
│  - Calculated Insights (Recency, Frequency, Monetary Scores)           │
│  - Harmonized DMOs (Contact Points, Cases, Orders, Products)           │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │ Outbound Zero-Copy Share
                                    │ (Apache Iceberg Metadata Sharing)
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│                     External Data Cloud Platforms                      │
│                                                                        │
│   Snowflake Data Share               Databricks Delta Sharing          │
│   - Appears as a native read-only    - Appears as an external Unity    │
│     database in Snowflake              Catalog Delta Share             │
│   - Joined directly with ERP tables  - Fed directly into PySpark and   │
│     using standard SQL                 MLflow model training pipelines │
└────────────────────────────────────────────────────────────────────────┘

How Outbound Data Sharing Operates

1. Data Share Creation: In Data Cloud Setup, an administrator creates a Data Share, selecting the specific Data Model Objects (DMOs), Unified Profiles, or Calculated Insights to expose.
2. Data Share Target: The administrator associates the Data Share with a Data Share Target (e.g., a specific Snowflake Account Identifier or Databricks recipient token).
3. Native Database Mount: In the external platform (such as Snowflake), the share appears instantly as a native, read-only database. Analysts can query Data Cloud tables using standard SQL: SELECT * FROM DATA_CLOUD_SHARE.CUSTOMER_360.UNIFIED_INDIVIDUAL__DMO;.

---

Bi-Directional Zero-Copy Integration: The Closed-Loop Enterprise Lifecycle

The full power of Zero-Copy manifests when an enterprise unites inbound federation and outbound data sharing into a continuous, closed-loop customer intelligence cycle:

1. Capture & Harmonize: Data Cloud captures real-time web/app interactions via the Web & Mobile SDK and streams CRM records via standard connectors.
2. Outbound Share to Snowflake: Data Cloud shares harmonized Unified Individual records and real-time behavioral signals to Snowflake via Zero-Copy Data Sharing.
3. Advanced Machine Learning Scoring: Within Snowflake or Databricks, enterprise data scientists run sophisticated predictive AI models (e.g., Customer Churn Probability, Next-Best-Action Propensity), combining Data Cloud's live Customer 360 profile with 15 years of on-premise ERP billing and supply chain history.
4. Write Predictions to Analytics Table: The data science model writes the generated scores to a Snowflake table (CUSTOMER_ML_SCORES).
5. Inbound Zero-Copy Federation: Data Cloud connects to CUSTOMER_ML_SCORES as an Inbound Virtual DLO (vDLO) and maps the churn score to the Individual DMO.
6. Real-Time Operational Activation: The churn score is immediately accessible in the Data Cloud Segmentation Canvas! Marketers build a segment: Churn_Risk_Score > 0.85 AND Recent_Service_Case_Count > 2, activating the audience to Marketing Cloud for retention campaigns and triggering real-time Alerts for Service Cloud agents.

All of this occurs without a single byte of data being extracted to flat files, SFTP drops, or intermediate ETL storage.

---

Performance, Latency, Governance, and Cost Tradeoffs

While Zero-Copy architecture provides extraordinary strategic agility, consultants must evaluate critical architectural tradeoffs when deciding between Zero-Copy federation and physical batch ingestion:

1. External Warehouse Compute Consumption & Cost

Inbound federated queries execute in the remote system's compute layer. When Data Cloud evaluates a segment or calculates an insight involving an Inbound vDLO from Snowflake, it spins up the designated Snowflake Virtual Warehouse.

• Cost Impact: If an enterprise configures 20 complex segments that refresh every hour against unindexed, multi-billion-row Snowflake tables, Snowflake virtual warehouse credits will burn continuously, resulting in unexpected compute invoices.

2. Query Latency & Interactive Personalization

Because federated queries traverse cloud networks and depend on remote query compilation, vDLO queries typically exhibit higher latency (seconds to minutes) than queries against locally cached lakehouse Parquet partitions. For millisecond real-time web personalization, physical ingestion or caching is superior to federated queries.

3. Data Governance, Data Spaces & Row-Level Security

When external tables are federated into Data Cloud, external row-level security (RLS) policies and dynamic masking applied in Snowflake or BigQuery may not automatically translate into Data Cloud. Consultants must enforce data segregation inside Data Cloud by provisioning dedicated Data Spaces and applying role-based permission sets.

Decision Framework: Zero-Copy vs. Physical Ingestion

---

Critical Exam Traps & Consultant Pitfalls

[!WARNING] The Snowflake Virtual Warehouse "Bill Shock" Trap An architect implements Inbound Zero-Copy Federation to query a 1.5-billion-row order history table residing in Snowflake. The marketing team creates 10 segments referencing this virtual DLO and sets each segment schedule to refresh every 1 hour. At the end of the first billing cycle, the client is shocked by an astronomical Snowflake credit bill!

Exam Reality: Segment refreshes on virtual DLOs actively wake up and consume compute credits on the external Snowflake Virtual Warehouse. To prevent bill shock, certified consultants must:

1. Right-size and auto-suspend Snowflake virtual warehouses (e.g., auto-suspend after 60 seconds of inactivity).
2. Ensure Snowflake source tables are clustered on frequently filtered attributes (e.g., Customer_Id, Transaction_Date).
3. Optimize segment schedules to daily or 12-hour cadences rather than rapid hourly polling.

[!CAUTION] The Direct Identity Resolution on High-Volume vDLOs Trap A consultant maps an Inbound Virtual DLO containing 200 million unindexed records directly to the Individual DMO and includes it in an Identity Resolution ruleset with multiple complex fuzzy match rules. During identity resolution runs, jobs fail consistently with cross-network query timeout errors.

Consultant Mandate: Identity resolution requires intensive, iterative graph traversal. Attempting to execute full-scale graph clustering across remote, un-materialized virtual tables creates massive network serialization bottlenecks. Best practice is to physically ingest core profile identifiers into native DLOs, reserving Zero-Copy federation for transactional and contextual analytical enrichment.