6.3 Match-Merge & Identity Resolution

Match-Merge and Identity Resolution

At the heart of any Master Data Management (MDM) implementation is Identity Resolution. This is the process of identifying, matching, and merging records from disparate source systems that represent the same real-world entity (e.g., the same customer, product, or location). Without robust identity resolution, an organization cannot achieve a single, reliable Golden Record (or Best Version of Truth or BVT).

The identity resolution engine operates through a structured process known as the Match-Merge Lifecycle. This lifecycle consists of five key phases: Data Standardization, Candidate Selection (Blocking), Matching, Survivorship, and Merging.

---

The Match-Merge Lifecycle Phases

1. Data Standardization and Cleansing

Before records can be matched, they must be formatted consistently. Raw data is often plagued by typos, varying formats, and noise. Standardization involves:

• Parsing: Splitting unstructured fields into structured attributes (e.g., splitting a full address string into Street Number, Street Name, Unit, City, State, and Zip Code).
• Normalization: Converting data values to standard representations (e.g., converting "Avenue", "Ave.", and "AVE" to "Ave").
• Casing and Punctuation: Removing special characters, trailing spaces, and standardizing case (e.g., "john.smith@domain.com" to "john.smith@domain.com").

2. Candidate Selection and Blocking

Comparing every record in a database against every other record is computationally expensive. For a database with N records, a naive pairwise comparison requires N(N-1)/2 comparisons—an O(N^2) complexity. If a database contains 1,000,000 records, this would require nearly 500 billion comparisons. To solve this, MDM systems use Blocking Keys (also known as match keys). Blocking partitions the database into smaller, manageable subsets (blocks) based on shared attributes (e.g., sharing the same first letter of the last name and the first three digits of the postal code). The matching algorithms are then run only within each block, drastically reducing the search space and execution time.

3. Matching Methodologies

Once candidates are grouped, the MDM engine applies matching rules to determine the likelihood that two records represent the same entity. There are two primary matching methodologies:

Deterministic Matching

Deterministic Matching relies on explicit, predefined rules and requires exact matches on specific data elements.

• How it works: Rule-based logic (e.g., if "Social Security Number (SSN) matches" or "First Name + Last Name + Date of Birth matches exactly").
• Pros: Extremely high precision (very few false positives), predictable, computationally fast.
• Cons: Low recall (misses true matches due to minor spelling errors, nicknames, or missing fields).

Probabilistic Matching

Probabilistic Matching uses statistical models to calculate the mathematical probability that two records match. It assigns positive or negative weights to individual attributes based on their uniqueness and reliability.

• String Similarity Algorithms:
  • Levenshtein Distance: Measures the minimum number of single-character edits (insertions, deletions, substitutions) required to change one string into another (e.g., "Smith" and "Smyth" has a Levenshtein distance of 1).
  • Jaro-Winkler: Measures similarity between two strings, giving higher weights to prefixes.
  • Phonetic Algorithms (Soundex / Double Metaphone): Translates words into codes representing how they sound in English, allowing matches despite spelling differences (e.g., "Robert" and "Rupert").
• Matching Thresholds:
  • Match Threshold: If the total similarity score exceeds this upper boundary, the records are matched and merged automatically.
  • Review Threshold: If the score falls between the match and non-match thresholds, the pair is routed to a Data Steward queue for manual review.
  • Non-Match Threshold: Below this boundary, the records are declared distinct.

4. Survivorship Rules

After identifying a match, the system must create the consolidated Golden Record. Survivorship Rules define how the MDM engine selects which attribute values "survive" from the source records to populate the golden record. Common survivorship strategies include:

• Source System Trust: Fields from a highly trusted system (e.g., a customer portal where the customer typed their own email) take precedence over other systems (e.g., a legacy billing system).
• Recency (Most Recent): The attribute value with the latest update timestamp is selected.
• Frequency (Most Common): The value that appears most frequently across all matched source records is chosen.
• Completeness: Choosing the value that is non-null, longest, or conforms to a validation mask.

5. Merging and Lineage

When records are merged, the MDM hub creates a new golden record or updates an existing one. Crucially, the hub must preserve the cross-reference (XREF) links to the original source records. This ensures Data Lineage is maintained. If matching rules are modified or a merge is discovered to be incorrect, the MDM system can execute an Unmerge operation, decomposing the golden record back into its constituent source records without data loss.

False Positives vs. False Negatives

In identity resolution, tuning the thresholds is a delicate balancing act:

• False Positives: The system incorrectly matches and merges two records that actually represent different entities (e.g., merging the records of two different people named "John Smith" who live in different cities). This is a critical quality issue that can lead to privacy violations (disclosing one person's data to another).
• False Negatives: The system fails to match two records that represent the same entity, leaving them as separate duplicates. This results in an incomplete view of the customer or asset. Data management professionals must continually profile data and adjust matching weights to minimize both errors, prioritizing privacy and compliance.

---

DAMA Exam Traps and Study Tips

1. Precision vs. Recall: Understand that deterministic matching favors high precision (avoiding incorrect matches) at the expense of recall. Probabilistic matching increases recall (finding more matches) but increases the risk of false positives.
2. The Purpose of Blocking: The exam frequently tests why blocking is used. Remember: blocking is a performance-optimization technique used to reduce the number of record comparisons, preventing exponential performance degradation.
3. Role of the Data Steward: Understand that data stewards do not manually match every record. They only resolve the "gray area" exceptions that fall between the automatic Match and Non-Match thresholds.