1.4 Duplicate CI Handling and Remediation

Duplicate CI Handling and Remediation

Data integrity is the cornerstone of a functional Configuration Management Database (CMDB). A common issue in large enterprises is the creation of duplicate configuration items (CIs)—multiple records in the CMDB representing a single physical asset or logical service. Duplicates typically occur due to misconfigured identification rules, multiple data sources (such as Discovery, Service Graph Connectors, and manual imports) writing to the CMDB without passing through the Identification and Reconciliation Engine (IRE), or manual data entry errors.

The consequences of duplicate CIs are severe. They lead to fragmented ticket history (incidents and changes attached to different records for the same server), broken service maps, inaccurate reporting, and failed software license compliance audits. Managing and remediating duplicate CIs is therefore a critical component of CMDB governance.

---

1. Detection via the Identification and Reconciliation Engine (IRE)

ServiceNow prevents and detects duplicate CIs primarily through the Identification and Reconciliation Engine (IRE). The IRE uses class-specific Identification Rules (which contain Identifier Entries) to evaluate incoming payloads. An identifier entry defines the unique attributes of a class, such as serial number, MAC address, hostname, or IP address.

When a discovery source attempts to insert or update a CI, the IRE evaluates the payload:

• If the payload match criteria point to an existing record, the IRE updates that record.
• If the payload matches multiple records with different system IDs (sys_id), the IRE identifies a duplication conflict.
• The IRE allows the update to proceed on one record, but it automatically generates a De-duplication Task in the reconciliation_duplicate_task table to flag the conflict for administrator review.

De-duplication tasks are also generated by the CMDB Health Correctness Dashboard scheduled job, which scans the CMDB periodically to identify existing duplicates that may have bypassed the IRE (e.g., through direct database writes or legacy import sets).

---

2. The Duplicate CI Remediation Wizard

ServiceNow provides a guided, wizard-driven interface called the Duplicate CI Remediation tool to resolve de-duplication tasks. This tool ensures that duplicates are resolved safely, preserving transactional history and downstream relationships.

When an administrator opens a de-duplication task, the remediation wizard guides them through the following critical steps:

1. Identify the Master CI: The administrator must select one CI from the duplicate group to serve as the single source of truth (the Master CI). ServiceNow provides recommendations based on criteria such as the record with the oldest creation date, the record discovered by the most authoritative source, or the record with the highest density of attributes and active relationships.
2. Consolidate Attributes (Merge): The wizard allows the user to compare the attributes of the duplicate CIs side-by-side. If a duplicate record contains a more accurate or recent value for an attribute (e.g., a newer firmware version or updated location), the user can choose to merge that specific value into the Master CI.
3. Consolidate Related Items and Relationships: This is the most critical step of the remediation process. The wizard automatically moves all downstream and upstream relationships, as well as operational records (such as active Incidents, Problems, Change Requests, and Outages), from the duplicate CIs to the Master CI. This prevents the loss of historical data and ensures that active support tickets remain linked to the correct configuration item.
4. Decommission Duplicate CIs: Once attributes and relationships are consolidated, the wizard prompts the user to dispose of the redundant duplicate records. The duplicates can either be permanently deleted from the database or retired (updated to lifecycle_stage = End of Life and lifecycle_stage_status = Retired) according to the organization's data retention policies.

---

3. Alternative Remediation Paths

While merging is the standard resolution path, the Duplicate CI Remediation tool supports two alternative options depending on the nature of the duplicates:

• Direct Deletion: If the duplicate records are verified to be empty, stale, or "junk" records with no active relationships or history, the administrator can bypass the merge process and delete them directly to clean up the database.
• Mark as Not Duplicate (Exclusion): In some scenarios, the IRE may flag CIs as duplicates when they are actually distinct physical devices. This occurs if identification rules are too loose or if multiple devices share non-unique identifiers (e.g., virtual machines cloned with identical MAC addresses or serial numbers). If the administrator verifies that the CIs are indeed distinct, they can mark them as "Not Duplicate".

Marking records as "Not Duplicate" creates a record in the Duplicate CI Exclusion (cmdb_duplicate_ci_exclusion) table. The IRE references this table during subsequent ingestion cycles and bypasses duplicate checks for the specified CI pair, preventing them from being flagged in future de-duplication tasks.

Duplicate Resolution Paths Comparison

By integrating the Duplicate CI Remediation wizard into governance workflows, CMDB administrators can systematically resolve data conflicts, maintain clean relationship maps, and ensure the overall accuracy of platform operations.