6.3 Data Entity Design & Modeling

6.3 Data Entity Design & Modeling

Quick Answer: A Data Entity is an abstraction layer that de-normalizes complex relational database tables into a single, cohesive conceptual business model. Data entities power two primary integration patterns: synchronous real-time OData v4 REST endpoints (when IsPublic = Yes, requiring an Entity Key, PublicEntityName, and PublicCollectionName) and asynchronous high-volume batch processing via the Data Management Framework (DMF) using Staging Tables. Entity fields can be Mapped (bound to table datasources), Unmapped (held in staging for transformation), or Virtual (computed dynamically in X++ via postLoad or mapEntityToDataSource without database persistence). Composite Data Entities assemble parent-child entity hierarchies (e.g., Sales Order Header + Lines) into a single transactional tree for XML/DMF package operations, but are strictly unsupported over OData.

---

1. Data Entity Architecture & Relational Abstraction

In Dynamics 365 Finance and Operations, relational tables are highly normalized to third normal form (3NF) to optimize transactional performance and eliminate update anomalies. For instance, customer master data is distributed across CustTable, DirPartyTable, LogisticsPostalAddress, DirPartyLocation, and LogisticsElectronicAddress.

While normalization benefits online transaction processing (OLTP), it creates severe friction for external integrations, reporting, and data migration. External systems should not need to orchestrate five interdependent database tables just to create a customer. A Data Entity solves this by encapsulating the normalized relational schema into a single, de-normalized, developer-friendly interface.

Normalized Relational Storage (Azure SQL)           Data Entity Abstraction Layer
┌──────────────────────────────────────┐          ┌─────────────────────────────┐
│ CustTable (Account, Group, PaymMode) │◄─────────┤                             │
├──────────────────────────────────────┤          │                             │
│ DirPartyTable (Name, Organization)   │◄─────────┤   CustCustomerV3Entity      │
├──────────────────────────────────────┤          │                             │
│ LogisticsPostalAddress (Street, City)│◄─────────┤ (Exposed via OData & DMF)   │
└──────────────────────────────────────┘          └─────────────────────────────┘

Dual Integration Channels

A single Data Entity definition concurrently services two distinct enterprise architectural pipelines:

1. Synchronous Real-Time OData v4 Services: Lightweight, synchronous CRUD operations triggered by web apps, Power Apps, Power Automate flows, and third-party REST clients.
2. Asynchronous High-Volume Batch Data Management (DMF): Mass data migrations, scheduled package exports to Azure Data Lake / BYOD, and bulk file imports orchestrated via the Data Management Framework.

---

2. Read/Write Entities vs. Read-Only Aggregate Entities

Data entities are partitioned into two architectural categories based on their operational purpose and underlying datasources:

Comparison of Entity Categories

---

3. Staging Tables & The DMF Import/Export Pipeline

When importing data via the Data Management Framework (DMF), external source files (CSV, Excel, XML) are never written directly into production transactional tables. Direct writes risk data corruption, lock contention, and bypass business validations.

The Staging Architecture

1. Data Landing: The DMF import job extracts records from the uploaded source package into a dedicated Staging Table (e.g., CustCustomerStaging).
2. Isolation & Transformation: Staging tables contain flat, loose schemas where constraints are relaxed. Data can be validated, cleansed, and transformed in staging without impacting live operational tables.
3. Target Processing: An automated SSIS or set-based X++ batch job transforms verified records from the staging table into the target transactional tables, invoking entity validation rules (validateWrite, insertEntityDataSource).

Field Classifications within Data Entities

Data Entity Field Classifications
├── 1. Mapped Field      --> Direct binding: Staging Column <-> Entity Field <-> Datasource Table Column
├── 2. Unmapped Field    --> Staging Column <-> Entity Field (No direct table datasource binding)
└── 3. Virtual Field     --> In-Memory X++ Calculation (Zero database or staging persistence)

1. Mapped Fields:
   • Directly bound to a physical column on one of the entity's underlying table data sources.
   • Changes to mapped fields automatically synchronize between staging, the entity buffer, and the underlying database table.
2. Unmapped Fields:
   • Has a physical column in the Staging Table, but is not bound to any field in the underlying table data sources.
   • Use Case: Receiving an external raw value (such as an unformatted composite address or legacy system ID) that must undergo complex custom transformation in X++ before being parsed into multiple target tables.
3. Virtual Fields:
   • Exists solely in memory on the data entity; it has no physical column in the database table and no column in the staging table.
   • Computed dynamically at runtime via X++ code.
   • Read / Export Logic: Populated inside the postLoad() method when an entity record is read via OData or exported.
   • Write / Import Logic: Parsed inside the mapEntityToDataSource() method when an inbound record is received.

// Virtual Field Implementation Pattern
public class MyCustomerEntity extends common
{
    // Executed during export or OData GET to populate the virtual field
    public void postLoad()
    {
        super();
        // Concatenate dynamic status without persisting to disk
        this.VirtualCreditSummary = strFmt("%1 (Limit: %2)", 
            this.CustomerAccount, this.CreditMax);
    }

    // Executed during import or OData POST/PATCH to parse the virtual field
    public void mapEntityToDataSource(DataEntityRuntimeContext _entityCtx, DataEntityDataSourceRuntimeContext _dataSourceCtx)
    {
        super(_entityCtx, _dataSourceCtx);
        if (_dataSourceCtx.name() == dataEntityDataSourceStr(MyCustomerEntity, CustTable))
        {
            // Extract custom business values and assign to target table buffer
            CustTable custTable = _dataSourceCtx.getBuffer();
            custTable.CustomField = this.VirtualCreditSummary;
        }
    }
}

---

4. Public Entities & OData v4 REST Endpoints

To expose a data entity as a synchronous REST endpoint accessible by external applications, developers configure the entity's Public properties.

Mandatory Properties for Public Entities

1. IsPublic = Yes: Flags the entity for compilation into the OData v4 metadata catalog.
2. PublicEntityName: The unique singular name identifying the entity resource (e.g., CustomerV3).
3. PublicCollectionName: The unique plural name identifying the entity collection endpoint (e.g., CustomersV3). Must be globally unique across the entire F&O deployment.
4. Entity Key: A designated unique alternate key on the data entity (e.g., CustomerAccount + DataAreaId).

The Role of the Entity Key in OData Addressing

Under the OData v4 specification, every resource within an entity set must be uniquely addressable via its key predicate. Without an Entity Key, the OData framework cannot construct single-record URI routes.

# Querying the collection (GET)
GET https://[environment].operations.dynamics.com/data/CustomersV3?$filter=CustomerGroup eq 'US_WHOLESALE'

# Addressing a single resource via Entity Key (GET / PATCH / DELETE)
GET https://[environment].operations.dynamics.com/data/CustomersV3(CustomerAccount='US-001',dataAreaId='usmf')

# Creating a new resource (POST)
POST https://[environment].operations.dynamics.com/data/CustomersV3
Content-Type: application/json

{
    "CustomerAccount": "US-099",
    "CustomerGroup": "US_RETAIL",
    "Name": "Contoso Retail Logistics"
}

---

5. Data Entity Extensions

Like tables and views, standard data entities can be enhanced through Data Entity Extensions (EntityName.Extension):

Permitted Entity Extension Actions

• Add Fields from Existing Data Sources: Expose standard or custom table columns from existing datasources that were not included in the base entity.
• Add New Data Sources: Attach new tables to the entity hierarchy (for example, joining a custom parameters table to VendVendorV2Entity). New data sources can be joined via InnerJoin or OuterJoin.
• Add Unmapped and Virtual Fields: Declare new custom virtual fields and write business logic in extension classes.
• Extend Validation and Mapping Methods: Utilize Chain of Command (CoC) on entity methods (validateWrite, postLoad, mapEntityToDataSource, insertEntityDataSource, updateEntityDataSource).

Prohibited / Immutable Boundaries in Entity Extensions

• Cannot Change the Root Data Source: The root datasource table cannot be altered or swapped.
• Cannot Delete Base Entity Fields: Standard fields cannot be deleted or removed from the entity.
• Cannot Modify Base Field Data Types: Field data types defined in the base entity cannot be changed.
• Cannot Alter Entity Key on Base Entities: You cannot remove fields from the standard Entity Key.

---

6. Composite Data Entities & Hierarchical Boundaries

In enterprise supply chain scenarios, business documents are inherently hierarchical. A sales order consists of an order header and multiple order line items. A vendor invoice consists of an invoice header, invoice lines, and tax lines.

Anatomy of a Composite Data Entity

A Composite Data Entity is an AOT object that links multiple independent, single-concept data entities into a unified parent-child tree structure:

Composite Data Entity Hierarchy (e.g., SalesOrderCompositeEntity)
│
├── Parent Data Entity: SalesOrderHeaderV2Entity
│   │
│   └── Child Data Entity: SalesOrderLineV2Entity
│       │
│       └── Child Data Entity: SalesOrderTaxLineEntity

• Relation Binding: The parent and child entities are linked by defining relationships matching key fields (e.g., Header.SalesOrderNumber == Line.SalesOrderNumber).
• Transactional Integrity: During DMF import, the entire document (header + lines) is processed as an atomic unit. If a line item fails business validation, the framework can roll back the entire document import.

[!WARNING] Critical Exam Rule: Composite Data Entities Are STRICTLY UNSUPPORTED in OData One of the most common and heavily tested traps on the MB-500 exam involves integration protocol selection for composite entities:

• Composite Data Entities are supported ONLY in asynchronous Data Management Framework (DMF) batch operations using XML file formats or data packages.
• Composite Data Entities CANNOT be set to IsPublic = Yes and are STRICTLY UNSUPPORTED over synchronous OData REST endpoints.
• If an external client requires synchronous REST integration to create sales orders with lines, it must make separate calls against individual public entities or execute deep inserts via custom REST services.

---

7. Scenario Walk-Through: Building a Vendor Onboarding Public Entity with Virtual Address Concatenation

Scenario Description

Fabrikam needs to onboard vendors via a third-party procurement portal. The external portal requires a synchronous REST endpoint (/data/VendorOnboardingV1) to submit new vendor records. The portal provides a single combined address string ("100 Main St, Suite 400, Austin, TX 78701") which must be parsed into street, city, state, and zip code before inserting into the normalized logistics address tables.

Implementation Walk-Through

1. Create Data Entity (VendVendorOnboardingEntity):
   • Primary Data Source: VendTable.
   • Secondary Data Sources: DirPartyTable (InnerJoin), LogisticsPostalAddress (OuterJoin).
2. Configure Public Properties:
   • IsPublic = Yes.
   • PublicEntityName = VendorOnboardingV1.
   • PublicCollectionName = VendorsOnboardingV1.
3. Define Entity Key:
   • Add VendorAccountNumber and DataAreaId to the entity's Entity Key.
4. Add Virtual Field for Address Concatenation:
   • Add an unmapped, virtual string field RawAddressString to the entity.
   • In postLoad(), write X++ code to format the address components for export.
   • In mapEntityToDataSource(), write parsing logic that splits RawAddressString into street, city, state, and zip code, assigning values to the LogisticsPostalAddress data source buffer.
5. Generate Staging Table:
   • Right-click the entity and select Generate staging table (VendVendorOnboardingStaging) to ensure the entity is also fully enabled for asynchronous DMF package imports.
6. Verify OData Registration:
   • Compile the model, synchronize the database, and verify the endpoint responds at https://[env].operations.dynamics.com/data/VendorsOnboardingV1.

---

8. Real-World Exam Traps: Data Entity Modeling

[!WARNING] Exam Trap 1: Attempting to Expose Composite Data Entities via OData An exam question asks which protocol to select to expose a Composite Data Entity for real-time CRUD operations from a third-party mobile web application. Any option proposing OData is completely false. Composite data entities cannot be public and only operate via DMF XML data packages.

[!WARNING] Exam Trap 2: Confusing Virtual Fields with Unmapped Fields Exam candidates often confuse these two concepts. An Unmapped Field has a physical column in the Staging Table (useful for staging external data before transformation). A Virtual Field has no physical column anywhere—neither in the database nor in the staging table—and exists purely in memory during X++ runtime execution (postLoad and mapEntityToDataSource).

[!WARNING] Exam Trap 3: Omitting the Entity Key on Public Data Entities If an exam scenario describes a public entity that fails to deploy or cannot be queried via a single-record URL (/data/Customers('US-001')), the root cause is almost certainly a missing or misconfigured Entity Key. OData v4 requires an Entity Key to construct unique URI routes for individual entity instances.

[!WARNING] Exam Trap 4: Attempting to Use Aggregate Entities for DMF Data Migration Aggregate entities are strictly read-only analytical structures designed for Power BI and analytical workspaces. They cannot be used as target entities for importing data via the Data Management Framework.