4.1 Data Kits & Package Automation for Cross-Org Deployment

Data Kits & Package Automation for Cross-Org Deployment

In modern enterprise IT environments, customer data architectures rarely exist in isolation. Organizations frequently maintain multi-tier deployment pipelines spanning developer sandboxes, quality assurance (QA) environments, user acceptance testing (UAT) sandboxes, staging environments, and production orgs. Furthermore, Independent Software Vendors (ISVs) and Salesforce System Integrators (SIs) routinely build standardized, industry-specific data models and analytical pipelines designed for repeated deployment across hundreds of client instances.

Historically, replicating data models, custom objects, field mappings, and analytical formulas across multiple customer data platform instances required manual configuration or custom integration scripts—an error-prone approach that introduced configuration drift and operational instability. In Salesforce Data Cloud, Data Kits solve this architectural challenge by providing a standardized, declarative framework for bundling, packaging, versioning, and deploying Data Cloud metadata assets across Salesforce organizations.

---

What is a Data Kit?

A Data Kit is a specialized metadata container within Data Cloud designed to package and distribute design-time assets. It decouples the abstract definition of a data pipeline from its physical run-time infrastructure, enabling developers and architects to author a data solution once and deploy it consistently into any subscriber org.

When bundled into a Data Kit, Data Cloud components are converted into portable metadata definitions. When installed into a subscriber org, these definitions serve as blueprints that instantiate native Data Cloud objects and configurations without requiring manual schema creation or manual field-by-field mapping.

Core Metadata Components Supported in Data Kits

A Data Kit can bundle a wide array of Data Cloud metadata entities:

1. Custom Data Model Objects (Custom DMOs): Custom entities created in the Customer 360 canonical data model to represent domain-specific concepts (e.g., Vehicle_Telemetry__c, Policy_Coverage__c, Hotel_Reservation__c).
2. Custom Fields on Standard DMOs: Custom attributes added to standard canonical DMOs (e.g., adding Loyalty_Tier__c or Preferred_Language__c to the standard Individual__dlm DMO).
3. Data Stream Templates: Blueprints of data streams that define source object schemas, target field data types, primary keys, event timestamps, and data categories (Profile, Engagement, or Other).
4. DLO-to-DMO Mapping Definitions: Pre-configured mapping configurations that define how fields from incoming Data Lake Objects (DLOs) map into canonical Data Model Objects (DMOs), including data type conversions and relationship mappings.
5. Calculated Insights (CIs): Declarative and SQL-based multidimensional analytical models that compute complex customer metrics (e.g., Customer_Lifetime_Value__c, Recency_Frequency_Monetary_Score__c, Churn_Risk_Index__c).
6. Custom Relationships: Foreign key associations and cardinality rules (1:1, 1:N, N:1) established between custom and standard DMOs.

---

What CAN and CANNOT Be Packaged in a Data Kit

A primary focus of the Salesforce Certified Data Cloud Consultant examination is recognizing the rigid boundaries between packageable metadata templates and unpackageable runtime assets. A consultant must never assume that an entire operational Data Cloud instance can be cloned via a Data Kit.

The following comparison table details the exact capabilities and limitations of Data Kits:

Consultant Exam Warning: Exam scenarios frequently describe a scenario where an administrator expects customer records or active S3 ingestion schedules to appear automatically after installing a package containing a Data Kit. Always remember: Data Kits transfer metadata blueprints, never physical data or external authentication credentials.

---

The Data Kit Lifecycle: Authoring, 2GP Packaging & Subscriber Deployment

Moving Data Cloud assets from a development environment to a production or subscriber organization follows a strict five-stage lifecycle:

Stage 1: Data Kit Authoring in Data Cloud Setup

Within the source org (typically a Developer org or Developer Sandbox):

1. Navigate to Data Cloud Setup > Data Kits.
2. Click New to create a Data Kit record, defining its name, API name, and description.
3. Add components to the kit: select custom DMOs, standard DMO custom fields, data stream definitions, and Calculated Insights.
4. Ensure all dependent mappings and relationships are fully included. Data Cloud automatically analyzes dependencies and prompts the author to include referenced entities.

Stage 2: Second-Generation Packaging (2GP)

A Data Kit cannot be distributed across orgs by itself; it must be encapsulated inside a Salesforce Second-Generation Package (2GP):

• Unlocked Packages (Enterprise DevOps):
  • Used for internal application development and enterprise multi-org pipelines (Dev -> Staging -> Prod).
  • Highly flexible: supports version upgrades, metadata overwrites, and dependency management.
  • Non-namespaced or namespaced based on enterprise architecture standards.
• Managed Packages (ISV Distribution):
  • Used by AppExchange partners to distribute commercial data solutions to external customers.
  • Enforces intellectual property (IP) protection (protecting proprietary Calculated Insight SQL logic).
  • Requires a registered namespace, preventing naming collisions with subscriber custom fields.

Using the Salesforce CLI (sf), developers link the Data Kit metadata to the package directory and generate an immutable package version:

sf package version create \
  --package "Enterprise-Customer360-DataKit" \
  --installation-key-bypass \
  --wait 30 \
  --code-coverage

Stage 3: Package Installation in Subscriber Org

The subscriber org administrator installs the generated package version via the Salesforce CLI or standard web browser installation URL:

sf package install \
  --package "04t..." \
  --target-org production-org \
  --wait 20

Upon package installation, the Data Kit metadata becomes available in the subscriber org, but it does not immediately deploy or alter the active Data Cloud environment.

Stage 4: Deploying Components from the Data Kit

After package installation, an administrator in the target org must deploy the assets from the Data Kit into active Data Cloud runtime:

1. Navigate to the Data Kits tab in Data Cloud.
2. Select the installed Data Kit.
3. Review the available components (DMOs, Stream Templates, Calculated Insights).
4. Select the target Data Space where the components should reside (e.g., Default Data Space or a custom Data Space like EMEA).
5. Click Deploy. Data Cloud instantiates the custom DMOs, mapping templates, and insight definitions.

Stage 5: Connector Binding and Stream Activation

Because external credentials and physical connections cannot be packaged, data streams instantiated from Data Kit templates are created in an Inactive / Unbound state.

1. The administrator navigates to Data Streams.
2. Opens the newly created stream template.
3. Selects an active, authenticated connector in the subscriber org (e.g., selecting an active Amazon S3 connection, Salesforce CRM org connection, or Ingestion API endpoint).
4. Selects the source file/object name and verifies the schedule.
5. Clicks Save and Deploy to activate the stream and begin physical data ingestion.

---

2GP Packaging Comparison: Unlocked vs. Managed Packages

Understanding when to use Unlocked Packages versus Managed Packages for Data Kits is a frequent decision point on the consultant exam:

---

Package Automation & CI/CD Pipelines for Data Cloud Metadata

In mature enterprise DevOps environments, manual point-and-click deployment through the Data Cloud UI is replaced by automated Continuous Integration and Continuous Delivery (CI/CD) pipelines. Data Cloud metadata types are fully integrated into the Salesforce Metadata API and Salesforce CLI.

Key Metadata Types for Data Cloud

When authoring deployment manifests (package.xml) or utilizing source tracking, developers interact with specific metadata types:

• DataKitDefinition: Represents the Data Kit container and its bundled asset relationships.
• DataModelObject: Represents custom DMO definitions and canonical field extensions.
• CalculatedInsight: Represents declarative and SQL-based insight definitions.
• DataStreamDefinition: Defines stream schemas and transformation mappings.

Automated Deployment Flow

A modern CI/CD pipeline (e.g., utilizing GitHub Actions, GitLab CI, Azure DevOps, or Copado) executes the following sequence:

1. Linting and Validation: Validate Data Cloud metadata files and SQL syntax against canonical schema standards.
2. Target Org Prerequisite Verification: A pre-deployment validation script checks that target org dependencies exist:
   • Verifies that the connected Salesforce CRM connector is authorized.
   • Checks that required source custom fields exist in the connected CRM org.
   • Confirms that target Data Spaces are provisioned.
3. Automated Metadata Deployment:

   sf project deploy start \
     --manifest manifest/package.xml \
     --target-org uat-environment \
     --wait 30
   

4. Post-Deployment Automated Configuration: Utilizing Salesforce CLI plugins or REST APIs, the pipeline binds deployed stream templates to pre-provisioned external credentials and activates batch ingestion schedules.

---

Dependencies, Prerequisites & Version Management

Deploying Data Kits into downstream environments introduces complex dependency trees that must be resolved in a specific order to prevent deployment failures:

1. Upstream Connector Prerequisites

If a Data Kit includes a Data Stream template configured for a Salesforce CRM source, the target org must already possess an active connection to that Salesforce CRM instance. If the connection does not exist, the Data Kit deployment will fail when attempting to validate the stream blueprint.

2. Source CRM Field Dependencies

If a Data Stream template references specific fields on the CRM Contact object (e.g., Loyalty_ID__c), those exact fields must exist in the source CRM org, and the Salesforce CRM Connector Integration User must have Field-Level Security (FLS) Read access to them. Missing fields cause schema validation errors during stream binding.

3. Canonical Schema Dependencies & Deletion Protection

Once a custom DMO or custom field deployed from a Data Kit is mapped to an active Data Lake Object, referenced in an Identity Resolution ruleset, or used in a Calculated Insight, it cannot be deleted or modified destructively. Attempting to deploy a new Data Kit version that removes a referenced field will trigger a dependency validation error in the subscriber org.

4. Version Upgrades and Backward Compatibility

When updating an Unlocked Package containing a Data Kit:

• Additive Changes (adding new custom fields to a DMO, adding a new Calculated Insight) deploy seamlessly without impacting running data pipelines.
• Destructive Changes (renaming fields, changing data types from Number to Text) require deprecation planning: mappings must be removed and insights updated prior to deploying the updated package version.