2.3 Duplicate Detection Rules & Settings
Key Takeaways
- Duplicate detection rules evaluate records based on defined matching criteria between a Base Table and a Matching Table (which can be the same table or two distinct tables like Lead and Contact).
- Matching criteria options include Exact Match, Same First N Characters, Same Last N Characters, Same Date, and Same Date and Time, with modifiers to ignore blank values and enforce case sensitivity.
- A maximum of 5 active (published) duplicate detection rules can be enabled per table at any given time.
- Duplicate detection triggers interactively during model-driven form saves (modal warning dialog), during Data Import wizard execution, during mobile offline synchronization, and via scheduled Bulk Duplicate Detection jobs.
- Automated API and Power Automate transactions bypass duplicate detection by default unless explicitly configured via the MSCRM.SuppressDuplicateDetection header.
Duplicate Detection Rules & Settings
Data integrity is vital for accurate analytics, customer relationship management, and automated business processes. Duplicate records lead to fragmented histories, inaccurate reporting, and wasted marketing spend. Microsoft Dataverse provides a robust, configurable Duplicate Detection Engine that detects potential duplicates before or after records are created.
For the PL-200 exam, functional consultants must master duplicate detection rule construction, matching operators, cross-table comparison mechanics, trigger events, and bulk deduplication jobs.
1. Duplicate Detection Settings & Architecture
Duplicate detection operates at two layers: Environment-level settings that control when the engine executes, and Table-level rules that define the matching logic.
+-----------------------------------------------------------------------------------+
| DUPLICATE DETECTION ENGINE ARCHITECTURE |
| |
| [GLOBAL SETTINGS (PPAC / Data Management)] |
| [x] Enable Duplicate Detection across Environment |
| | |
| +---> [x] When a record is created or updated |
| +---> [x] When Dataverse for Outlook / Mobile goes offline to online |
| +---> [x] During data import |
| |
| [PUBLISHED RULES (Max 5 Active Rules per Table)] |
| - Generates match codes / hash indexes upon publishing |
| - Compares Base Table against Matching Table |
+-----------------------------------------------------------------------------------+
Global Trigger Configuration
In PPAC or the classic settings area (Settings > Data Management > Duplicate Detection Settings), administrators configure the global trigger points:
- When a record is created or updated: Evaluates rules in real time when a user saves a form.
- When Dataverse for Outlook/Mobile transitions from offline to online: Catches duplicates created on mobile devices while disconnected.
- During data import: Evaluates incoming records against existing data during Data Import Wizard runs.
2. Anatomy of a Duplicate Detection Rule
A duplicate detection rule specifies how Dataverse compares records.
+-----------------------------------------------------------------------------------+
| DUPLICATE DETECTION RULE CONFIGURATION |
| |
| RULE NAME: [Lead to Contact Email & Name Match] |
| BASE TABLE: [Lead] MATCHING TABLE: [Contact] |
| |
| CRITERIA TABLE: |
| +-------------------+----------------------+-------------------+--------------+ |
| | Base Table Field | Matching Table Field | Operator | Ignore Blank | |
| +-------------------+----------------------+-------------------+--------------+ |
| | Email | Email Address 1 | Exact Match | [x] Yes | |
| | First Name | First Name | Same First 3 Chars| [x] Yes | |
| | Last Name | Last Name | Exact Match | [ ] No | |
| +-------------------+----------------------+-------------------+--------------+ |
+-----------------------------------------------------------------------------------+
Key Components of a Rule:
- Base Table: The primary table on which the rule is defined (the record being created or updated).
- Matching Table: The target table against which the Base record is compared. The Matching Table can be identical to the Base Table (e.g., Contact to Contact for intra-table deduplication) or a different table (e.g., Lead to Contact or Lead to Account for cross-table matching).
- Matching Criteria Operators:
Exact Match: String or value must match character-for-character.Same First Characters: Matches the first N characters (e.g., first 5 characters of Postal Code or first 3 of First Name).Same Last Characters: Matches the trailing N characters (e.g., last 7 digits of a Phone Number, ignoring country/area codes).Same Date: Matches calendar date while ignoring time components.Same Date and Time: Matches both date and timestamp exactly.
- Modifiers:
- Ignore Blank Values: When checked, records with null/empty values in that field will not be considered duplicates. Always enable this for optional fields to prevent matching thousands of records with empty values.
- Case-Sensitive: Enforces exact casing (e.g., "Contoso" vs "contoso"). Typically left disabled.
The 5-Rule Limit & Matchcode Generation
[!IMPORTANT] Rule Ceiling: Dataverse allows you to define unlimited duplicate detection rules, but no more than 5 rules can be in a Published (Active) state per table at any time. When a rule is published, Dataverse creates an internal matchcode (hash index) for each record. Publishing takes time depending on table record volume.
3. Trigger Scenarios & User Experiences
Dataverse handles duplicate detection differently depending on the context in which data is entered or processed.
+-----------------------------------------------------------------------------------+
| DUPLICATE DETECTION TRIGGER SCENARIOS |
| |
| +-----------------------------------------------------------------------------+ |
| | 1. MODEL-DRIVEN FORM SAVE (Interactive User Experience) | |
| | User enters duplicate data ---> Clicks 'Save' ---> Warning Modal Appears | |
| | Modal lists matching records ---> Options: [Ignore and save] or [Cancel] | |
| +-----------------------------------------------------------------------------+ |
| | 2. DATA IMPORT WIZARD (Batch Ingestion) | |
| | Import setting: 'Enable Duplicate Detection: Yes/No' | |
| | Duplicates logged as errors/warnings or excluded based on settings | |
| +-----------------------------------------------------------------------------+ |
| | 3. BULK DUPLICATE DETECTION SYSTEM JOBS (Asynchronous Scans) | |
| | Scheduled query scan across entire table ---> Generates duplicate report | |
| +-----------------------------------------------------------------------------+ |
| | 4. WEBSERVICE / API / POWER AUTOMATE (Automated Flows) | |
| | Default: Bypasses duplicate detection silently | |
| | Enforce via HTTP Header: 'MSCRM.SuppressDuplicateDetection: false' | |
| +-----------------------------------------------------------------------------+ |
+-----------------------------------------------------------------------------------+
Trigger Scenarios Detailed:
- Model-Driven Interactive Save: When a user creates or updates a record via a form, Dataverse checks active matchcodes. If a match is detected, an interactive Duplicates detected modal displays the existing matching records. The user can review the records and click Ignore and save to create the record anyway, or Cancel to abort and merge.
- Data Import Wizard: During data import, the wizard prompts Enable Duplicate Detection. If set to No, all records import regardless of duplicates. If set to Yes, imported records matching existing records are rejected and reported in the import failure log.
- Bulk Duplicate Detection Jobs: Administrators can schedule background system jobs that run FetchXML queries against active tables. The job scans records, identifies duplicates, and presents them in a dedicated review list where administrators can inspect and merge records.
4. Merging Records in Dataverse
When duplicate records exist, Dataverse provides native Merge capabilities for standard entities: Account, Contact, Lead, and Incident (Case).
+-----------------------------------------------------------------------------------+
| RECORD MERGING WORKFLOW |
| |
| [Select 2 Records] ---> Click [Merge] on Command Bar |
| | |
| v |
| [Select Master Record] ---> Choose which record retains the Primary ID |
| | |
| v |
| [Field-by-Field Selection] ---> Choose surviving attribute values |
| | |
| v |
| [Merge Execution] ---> Dataverse performs 3 automated actions: |
| 1. Updates Master Record with selected surviving values |
| 2. Reparents all subordinate Child Records (Activities, Orders, Cases) |
| 3. Deactivates (Sets to Inactive) the Subordinate Record |
+-----------------------------------------------------------------------------------+
Core Rules for Merging:
- You can merge up to 2 records at a time in the standard user interface.
- The subordinate record is deactivated (soft-deleted), never permanently deleted, preserving historical references and audit trails.
- All related child records (appointments, tasks, emails, child cases, invoices) are automatically reparented to the master record.
5. API & Power Automate Considerations
In modern automated architectures involving Power Automate cloud flows, Azure Functions, or custom web services, developers frequently encounter unexpected duplicate behavior:
- Default API Behavior: Programmatic Create and Update requests through the Dataverse Web API and standard Power Automate connectors suppress duplicate detection by default to maximize ingestion performance.
- Enforcing Duplicate Detection in APIs: To enforce rules, the calling client must pass the following HTTP header:
MSCRM.SuppressDuplicateDetection: false - If a duplicate is detected when this header is present, the API throws an HTTP
412 Precondition Failederror containing the GUIDs of matching duplicate records.
A functional consultant is designing a duplicate detection rule on the Contact table to identify contacts with similar first names and identical email addresses. Which configuration should the consultant implement to prevent contacts with empty email addresses from matching each other?
A company wants to check incoming Leads against both existing Leads and existing Contacts to prevent sales reps from creating leads for already registered contacts. How can the consultant achieve this with duplicate detection rules?
A sales representative enters an account in a model-driven app that matches an existing account based on an active duplicate detection rule. What is the default user experience when the representative clicks Save?
An integration developer creates a cloud flow in Power Automate that inserts 5,000 Contact records into Dataverse using the standard Dataverse connector. Many contacts have identical email addresses that match active duplicate detection rules, but all 5,000 records are created without errors. Why did duplicate detection not trigger?