5.2 Cloud Storage (S3, GCS, Azure Blob) & Streaming Ingestion (Web/Mobile SDK, Ingestion API)

Cloud Storage (S3, GCS, Azure Blob) & Streaming Ingestion (Web/Mobile SDK, Ingestion API)

While Salesforce CRM connectors handle transactional core records, modern enterprise Customer Data Platforms must ingest petabyte-scale data originating outside the core Salesforce perimeter. This includes nightly ERP inventory dumps stored in Amazon S3, behavioral telemetry streamed from digital storefronts via the Salesforce Interactions SDK, and real-time point-of-sale (POS) events pushed via the Ingestion API. A certified Data Cloud consultant must master the architectural patterns, security handshakes, schema definitions, and ingestion schedules across both file-based batch storage and low-latency streaming endpoints.

---

Cloud Storage Connectors: Architecture & Configuration

Salesforce Data Cloud provides native, high-throughput batch connectors for major enterprise cloud object stores and file transfer protocols:

• Amazon Web Services (AWS) S3
• Google Cloud Storage (GCS)
• Microsoft Azure Blob Storage
• Secure File Transfer Protocol (SFTP)

Supported File Formats & Compression

Cloud storage connectors extract structured tabular data from object storage buckets. Consultants must align source export pipelines with supported formats and compression types:

Directory Structures, File Naming Patterns & Wildcards

Cloud storage connectors locate files within buckets using directory pathing and wildcard matching expressions:

• Bucket and Subdirectory: e.g., s3://enterprise-customer-data/ingestion/pos_transactions/
• File Pattern (Wildcard): Using asterisks (*) to capture timestamped or partitioned files, such as pos_transactions_*.csv.gz or orders_daily_????-??-??.parquet.
• Partitioned Directories: Data Cloud can navigate partitioned hierarchies, such as /year={YYYY}/month={MM}/day={DD}/*.parquet.

The File Processing Lifecycle & Idempotency

Data Cloud tracks processed files by persisting an internal metadata catalog of ingested file names and their corresponding object modification timestamps.

[!CAUTION] The Stale File Overwrite Trap: If an external ETL job modifies an existing file in-place (e.g., re-uploading customers_daily.csv without renaming the file), Data Cloud's scheduled sync may evaluate the file as already processed and skip it, leading to missed updates! Enterprise export pipelines must always employ unique timestamped or sequential file naming conventions (e.g., customers_20260921_120000.csv.gz).

---

IAM Authentication, Role Delegation & Security Architecture

Connecting Data Cloud to external public cloud infrastructure requires enterprise-grade security. Data Cloud never requires hardcoding permanent administrative root credentials; instead, it enforces cross-account role delegation and scoped identity tokens.

Amazon S3 Authentication: AWS IAM AssumeRole & External ID

The industry-standard pattern for connecting Data Cloud to Amazon S3 is cross-account IAM Role Assumption:

1. Salesforce Account ID & External ID: Within Data Cloud Setup, navigating to the S3 Connector interface generates a 12-digit Salesforce AWS Account ID and a unique, cryptographically random External ID specific to your Data Cloud tenant.
2. AWS IAM Trust Policy: In the customer's AWS management console, an IAM Role is created with a Trust Relationship allowing the Salesforce AWS Account to execute the sts:AssumeRole action, guarded by an sts:ExternalId condition to prevent the "confused deputy" problem:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "DataCloud-Tenant-Unique-External-ID-98765"
        }
      }
    }
  ]
}

1. IAM Permissions Policy: The role requires scoped permissions on the target S3 bucket: s3:GetObject, s3:ListBucket.
2. AWS KMS Encryption Keys (SSE-KMS): If the S3 bucket is encrypted using AWS Key Management Service (KMS), the assumed role must also be granted explicit access to the customer-managed KMS key: kms:Decrypt and kms:DescribeKey. Omitting KMS permissions is the single most frequent cause of S3 ingestion connection failures.

Google Cloud Storage (GCS) Authentication

• Authenticates via a GCP Service Account. Data Cloud provides a service account principal that the customer binds to the target GCS bucket with the Storage Object Viewer IAM role.

Microsoft Azure Blob Storage Authentication

• Utilizes either Shared Access Signature (SAS) Tokens (scoped to the container level with Read and List permissions and an explicit validity timeframe) or Microsoft Entra ID (Azure AD) service principals assigned the Storage Blob Data Reader role.

---

Salesforce Interactions Web & Mobile SDK

Capturing real-time digital engagement across web properties and mobile applications is fundamental for real-time personalization, cart-abandonment triggers, and behavioral segmentation. The Salesforce Interactions SDK provides client-side libraries for JavaScript (Web) and native mobile environments (iOS and Android).

Browser / Mobile App Event
  │ (Clicks "Add to Cart", SKU: 88412, Price: $149.00)
  ▼
Salesforce Interactions SDK
  │ Parses Web Sitemap / Mobile Event Schema
  ▼
Identity Capture Tier
  ├── Anonymous Visitor? --> Generates / Persists _anonId: "anon-uuid-77182"
  └── Authenticated User? --> Captures Known ID: "cust-9941" (Email / CRM ID)
  ▼
Real-Time Streaming Edge Ingestion
  ▼
Data Cloud Data Lake Object (DLO: Web_Engagement__dlm)
  │ EventTime: 2026-09-21T10:14:02Z | Category: Engagement
  ▼
Streaming Data Transform / Streaming Insights / Real-Time Data Actions

Declarative Web Sitemap Architecture

The Web SDK relies on a declarative Sitemap deployed via JavaScript. The sitemap monitors page URLs and DOM elements to determine:

• Page Type: Home, Category, Product Detail Page (PDP), Cart, Checkout, Order Confirmation.
• Contextual Metadata: Product SKU, price, currency, category breadcrumb, search query.
• Interaction Events: viewItem, viewCategory, addToCart, purchase, search.

Anonymous-to-Known Profile Stitching

When a first-time user visits an e-commerce website, they have no established customer identity. The Interactions SDK automatically generates a pseudonymous cookie identifier (_anonId). All browsing actions, category views, and cart additions are ingested into Data Cloud as engagement records tied to this anonymous ID.

When the user subsequently authenticates (e.g., registers, logs into their portal, or completes a purchase providing an email address), the SDK captures the authenticated identifier (partyId). Data Cloud's Identity Resolution engine links the _anonId to the authenticated Individual profile, bridging the user's historical anonymous browsing journey with their master Customer 360 profile.

---

Ingestion API: Streaming vs. Bulk Architecture

The Ingestion API enables external applications, custom enterprise backends, mobile applications, microservices, and ETL platforms (such as MuleSoft, Informatica, or Talend) to push data directly into Data Cloud without intermediary cloud storage staging.

Schema Provisioning via OpenAPI Specification (OAS 3.0)

Before transmitting data through the Ingestion API, an administrator or developer must upload an OpenAPI 3.0 (OAS) schema file in YAML or JSON format. This schema defines:

• The object name and fields.
• Data types (string, number, date, dateTime).
• Primary Key definition.

Once the OAS file is uploaded in Data Cloud Setup, the platform automatically generates an Ingestion API connector and provisions the underlying Data Lake Objects (DLOs), ready to receive HTTP payloads.

Streaming vs. Bulk Ingestion API Endpoints

The Ingestion API exposes two distinct operational endpoints designed for divergent data velocity and volume requirements:

---

Refresh Schedules & Ingestion Cadences

For batch-oriented connectors (Cloud Storage, SFTP, and Bulk Ingestion API), consultants must configure the synchronization frequency:

• Standard Refresh Cadences: Hourly, Daily, Weekly, or Monthly scheduled runs.
• Manual Execution: Administrators can trigger an on-demand "Refresh Now" from the Data Stream interface for ad-hoc backfills or testing.
• Event-Driven Triggers: Cloud storage buckets can be paired with automated notification services (such as AWS SNS/SQS event notifications) to trigger Data Cloud ingestion pipelines immediately upon the arrival of a new file drop.

---

Critical Exam Traps & Consultant Pitfalls

[!WARNING] The AWS KMS "Access Denied" Trap An enterprise configures an S3 connector. The IAM Role is created with full s3:GetObject and s3:ListBucket permissions, and the AssumeRole Trust Relationship matches the External ID perfectly. When testing the connection, the bucket connects, but every scheduled Data Stream execution fails with an opaque "Access Denied / Extraction Failed" error.

The Exam Cause: The enterprise S3 bucket is encrypted using AWS Key Management Service (SSE-KMS) with a Customer Managed Key (CMK). The IAM role lacked permissions to decrypt the data! A certified consultant must update the AWS KMS Key Policy to grant the IAM Role permissions for kms:Decrypt and kms:DescribeKey on the key ARN.

[!CAUTION] The Ingestion API Schema Immutability Trap Once an Ingestion API schema is deployed via an OpenAPI 3.0 specification and mapped to canonical DMOs, you cannot arbitrarily modify, rename, or delete existing fields in the OpenAPI definition. Removing or altering fields requires deleting downstream DMO mappings, retiring the Data Stream, and deploying an updated OAS definition. Always validate your schema contracts thoroughly in sandbox environments before deploying to production.