14.1 DMF Concepts, Projects & Sequencing

14.1 DMF Concepts, Projects & Sequencing

Quick Answer: The Data Management Framework (DMF)—also known historically as the Data Import/Export Framework (DIXF)—is Dynamics 365 Finance and Operations' enterprise engine for high-volume data migration, integration, and batch transformation. DMF employs a three-tier architecture: Source Data -> Staging Table -> Target Application Tables. Data operations are managed via Data Projects (Import/Export) supporting CSV, Excel, XML, and packaged ZIP files. Relational integrity during bulk loads is enforced through Execution Sequencing configured across three tiers: Execution Unit (outer batch grouping), Level in Execution Unit (dependency stage within a unit), and Sequence in Level (execution order within a level). For maximum ingestion throughput, entities can enable Set-based processing, which translates transfers into direct SQL set-based operations (INSERT_RECORDSET), intentionally bypassing row-level X++ methods (insert(), validateWrite()).

---

1. DMF Architecture: The Three-Tier Pipeline

In enterprise ERP implementations, migrating master records (such as 500,000 customers, 1 million products, or 5 million inventory transactions) directly into normalized, relational database tables causes severe table lock contention, transaction log saturation, and catastrophic integrity failures when external formats contain malformed data. DMF isolates these operations using a decoupled, staged processing pipeline.

Data Management Framework (DMF) Three-Tier Architecture

┌─────────────────────────────────────────────────────────────────┐
│                     Source Data Layer                           │
│  • File formats: Delimited (CSV/TSV), Excel (.xlsx), XML        │
│  • Data Package (.zip): Contains manifest.xml & data files      │
└────────────────────────────────┬────────────────────────────────┘
                                 │ Phase 1: Source-to-Staging (Parsing & Validation)
                                 ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Staging Table Layer                          │
│  • Auto-generated physical SQL table (EntityName + 'Staging')   │
│  • Denormalized staging columns mirror source data structure    │
│  • Records marked with TransferStatus (Selected/Completed/Error)│
│  • Allows manual data cleansing and error correction in UI      │
└────────────────────────────────┬────────────────────────────────┘
                                 │ Phase 2: Staging-to-Target (Business Logic & Commit)
                                 ▼
┌─────────────────────────────────────────────────────────────────┐
│                Target Application Tables Layer                  │
│  • Normalized OLTP tables (CustTable, DirPartyTable, etc.)      │
│  • Evaluates relations, financial dimensions, and numbering    │
│  • Mode: Set-based (Direct SQL) OR Row-by-row (X++ Methods)     │
└─────────────────────────────────────────────────────────────────┘

The Staging Table Layer

When a developer creates a Data Entity in Visual Studio using the Data Entity Wizard, the framework automatically generates a corresponding Staging Table (suffixed with Staging, such as CustCustomerV3Staging).

• Isolation & Troubleshooting: If an incoming file contains 10,000 records and 15 contain formatting defects, the 9,985 valid rows can proceed while the 15 invalid rows remain in the staging table. Users can inspect the exact parsing errors in the Execution log, edit the values directly in the View staging data form, and re-execute target copying without re-uploading the source file.
• Performance Buffer: Ingesting flat files into physical staging tables is executed using high-speed bulk database operations (e.g., SQL Server bulk copy bcp / SqlBulkCopy), minimizing the duration of external I/O locks.
• Staging Table Regeneration: When a developer alters the underlying data entity schema (such as adding new fields or modifying data types), the staging table metadata in SQL Server becomes out of sync. Developers and administrators must regenerate the staging table from Data management > Data entities > Modify target mapping / Regenerate staging table to ensure the physical SQL table reflects the updated entity metadata.
• Staging Data Maintenance: Staging tables can accumulate millions of historical staging rows, consuming substantial database storage. F&O provides automated cleanup routines under Data management > Framework parameters > Staging clean-up or via the system batch job DMFStagingCleanupExecutionService to purge staging records older than a configured retention period.

The Target Table Layer

Once staging rows are validated, DMF copies data from the staging table to normalized destination tables (e.g., populating CustTable, DirPartyTable, and LogisticsPostalAddress from a single CustCustomerV3Staging record). This transfer executes via the entity's copyTarget() pipeline.

---

2. Data Projects & Source Data Formats

A Data Project is an administrative configuration in the Data Management workspace that encapsulates one or more data entities, source data format definitions, field mappings, and execution parameters.

Import Projects vs. Export Projects

• Import Projects: Pull external source files into staging tables, validate constraints, and push verified records into target application tables.
• Export Projects: Extract records from normalized application tables into staging entities and serialize them into external formats (or push them into external target stores such as Bring Your Own Database / BYOD).

Supported Source Data Formats

Before adding entities to a data project, administrators configure Source data formats under Data management > Configure source data formats:

---

3. Entity Execution Parameters & Sequencing Hierarchy

In complex data migrations, records cannot be imported in arbitrary order. For instance, importing Sales Orders before Customers exist, or importing Customers before Customer Groups and Currencies exist, results in foreign-key referential integrity errors. DMF enforces execution dependencies through a strict three-tier sequencing hierarchy.

Execution Sequencing Hierarchy

Execution Unit 1 (Master Setup Data)
 ├── Level 1: Currency, Payment Terms, Units of Measure (Executed in Parallel or Sequence)
 └── Level 2: Customer Groups, Vendor Groups, Item Groups (Depends on Level 1)
      │
      ▼ (Unit 1 must complete fully before Unit 2 begins)
Execution Unit 2 (Master Entities)
 ├── Level 1: Customers V3, Vendors V2 (Depends on Groups and Currencies)
 └── Level 2: Released Products V2 (Depends on Item Groups and Units)
      │
      ▼ (Unit 2 must complete fully before Unit 3 begins)
Execution Unit 3 (Transactional Documents)
 ├── Level 1: Sales Order Headers V2
 └── Level 2: Sales Order Lines (Depends on Headers, Customers, and Products)

Sequencing Parameters Defined

1. Execution Unit: The top-level execution boundary. All entities assigned to Unit 1 must complete processing (both staging and target copying) before any entity in Unit 2 begins. Execution units run strictly sequentially.
2. Level in Execution Unit: Defines sub-dependencies within an individual execution unit. Entities configured at Level 1 execute first. Once Level 1 completes, entities configured at Level 2 execute. Entities with the same Level number can run concurrently in parallel if multiple batch threads are allocated.
3. Sequence in Level: Defines the prioritized sequence order (e.g., 10, 20, 30) for entities sharing the same Level within an Execution Unit.

Real-World Execution Blueprint Table

[!IMPORTANT] Exam Rule: Parallel Execution Behavior Entities assigned the same Execution Unit and same Level can execute in parallel across multiple batch threads if the batch job is configured with multiple tasks. If entity B strictly depends on entity A, they must never share the same Level; entity B must be placed on a higher Level number or in a subsequent Execution Unit.

Batch Task Grouping and Parallelism Thresholds

To maximize hardware utilization, administrators configure Entity execution parameters (under Data management > Framework parameters > Entity settings). This includes specifying the Import threshold record count and Task count:

• Threshold Record Count: The minimum number of staging records required before DMF splits the staging-to-target job across parallel batch tasks (e.g., threshold set to 10,000 records).
• Task Count: The number of parallel batch tasks spawned across available AOS batch threads (e.g., splitting 100,000 records into 8 tasks of 12,500 records each).

---

4. Set-Based Processing vs. Row-by-Row Processing

When copying data from staging tables to target application tables, developers can toggle the entity property Set-based processing = Yes inside the data project or entity settings.

The Performance Divide

[!WARNING] Real-World Exam Trap: Bypassed Validations in Set-Based Processing MB-500 test questions frequently describe a scenario where custom validation code written in validateWrite() or custom calculation logic in insert() was completely ignored during a large DMF import. The root cause is almost always that Set-based processing was enabled. When set-based processing is active, the AOS pushes data from staging to target using SQL set operations, bypassing single-record X++ CRUD methods.

Implementing Set-Based Delegates

When set-based processing is required for performance but custom transformation logic must still occur, developers override set-based delegates on the data entity in Visual Studio:

[SubscribesTo(tableStr(CustCustomerV3Entity), delegateStr(CustCustomerV3Entity, insertEntityDataSourceSetBased))]
public static void onInsertEntityDataSourceSetBased(CustCustomerV3Entity _entity, Common _targetTable, boolean _isTargetRecordNew)
{
    // Custom set-based SQL transformations executed prior to bulk commit
}

---

5. Field Mapping: Visual, Auto, Manual & Transformations

The bridge between external source files and staging table fields is governed by the Field Mapping tool within the data project.

Field Mapping Design Surface

Source File (e.g., CSV/Excel)             Staging Table (CustCustomerV3Staging)
┌────────────────────────────┐            ┌────────────────────────────────────┐
│ AccountNumber              │───────────>│ CustomerAccount                    │
│ CustomerName               │───────────>│ OrganizationName                   │
│ CustGrp                    │───────────>│ CustomerGroupId                    │
│ Currency                   │───────────>│ CurrencyCode                       │
│ [Missing in File]          │            │ SalesCurrencyCode (Default: 'USD') │
└────────────────────────────┘            └────────────────────────────────────┘
               ▲                                             ▲
               │                                             │
               └───────── Transformation & Conversion ───────┘

Mapping Mechanisms

1. Auto-Mapping: When a source file is uploaded, DMF automatically maps file columns to staging fields whose names or labels match. If column headers match exactly, auto-mapping achieves 100% completion.
2. Visual Mapping Tool: An interactive graphical design surface where developers can draw connection lines from source columns on the left to staging attributes on the right.
3. Mapping Details (Grid View): Allows granular control over mapping parameters, including:
   • Default Values: Specifies a hardcoded fallback value (e.g., defaulting SalesType to Journal or CurrencyCode to USD) whenever the source file column is empty or unmapped.
   • Conversion Rules (Value Mapping): Enables value substitution (e.g., translating external vendor code VEND_DOMESTIC to internal code DOM).
   • Transformations (XSLT): Applies Extensible Stylesheet Language Transformations to inbound XML structures before loading staging tables.

Source-to-Staging vs. Staging-to-Target Mapping

• Source-to-Staging Mapping: Dictates how physical file contents populate staging table columns. This is fully configurable within the Data Management web workspace without requiring code changes.
• Staging-to-Target Mapping: Dictates how staging table columns translate into normalized application tables. This logic is defined in Visual Studio metadata within the Data Entity definition (mapEntityToDataSource() and entity field bindings).

---

6. Scenario Walk-Through: Enterprise Master & Transactional Migration

Scenario Background

An enterprise retailer is migrating from an on-premises legacy AS/400 system to Dynamics 365 Finance. The cutover plan requires importing 80,000 customer records, 150,000 released products, and 450,000 open sales order lines over a 12-hour weekend maintenance window.

Step-by-Step Implementation Flow

1. Data Project Structure: Create a dedicated import project named Cutover_Master_Data.
2. Sequencing Configuration:
   • Execution Unit 1, Level 1: Currencies, Units of Measure, Payment Terms.
   • Execution Unit 1, Level 2: Customer Groups, Item Model Groups, Commission Groups.
   • Execution Unit 2, Level 1: Customers V3 (Row-by-Row = No / Set-based = Yes; continuous number sequences disabled).
   • Execution Unit 2, Level 2: Released Products V2 (Set-based = Yes).
   • Execution Unit 3, Level 1: Sales Order Headers V2 (Row-by-row = Yes, to execute business logic for tax groups and credit limits).
   • Execution Unit 3, Level 2: Sales Order Lines (Row-by-row = Yes, to ensure inventory reservation and pricing calculation methods fire).
3. Staging Error Handling: During the pilot run, 32 customer records fail with invalid ISO country codes. The migration team opens the execution log, selects View staging data, updates the country codes to valid values directly in the grid, and triggers Copy data to target for selected error records without re-importing the entire source file.

---

7. Real-World Exam Traps: DMF Concepts, Projects & Sequencing

[!WARNING] Exam Trap 1: Set-Based Processing Bypassing Validations When an exam question states that custom business logic in validateWrite() or insert() on a target table failed to execute during a bulk data migration, the immediate answer is that Set-based processing was enabled. Set-based processing operates at the SQL layer and never calls row-by-row X++ methods.

[!WARNING] Exam Trap 2: Incorrect Execution Sequencing Causing Dependency Failures If an import project fails with foreign key lookup errors (e.g., Sales Orders failing because Customers do not exist), placing all entities in the same Execution Unit and Level with differing Sequence numbers will not prevent race conditions during multi-threaded batch runs. Entities with hard relational dependencies must reside on different Levels or different Execution Units.

[!WARNING] Exam Trap 3: File Column Added but Ignored by Staging When a source CSV file has new columns added after the data project was initially configured, DMF ignores the new columns by default. Developers must open the entity mapping in the data project and update the source-to-staging mapping or click Re-map to bind the new columns.

[!WARNING] Exam Trap 4: Continuous Number Sequences Blocking Set-Based Processing Set-based processing cannot allocate continuous number sequences dynamically during SQL INSERT_RECORDSET operations. If an entity relies on a continuous number sequence, set-based processing will fail or fall back to row-by-row mode unless continuous numbering is turned off or numbers are supplied in the source file.