5.2 Metadata Lineage & Impact Analysis

The Concept of Data Lineage

In data management, Data Lineage describes the lifecycle of data, tracking its origin, movement, transformations, and ultimate destination across the enterprise. It answers the fundamental questions: "Where did this data come from?" and "What transformations did it undergo along the way?" Within the DAMA DMBoK2 framework, lineage is a cornerstone of metadata management, providing the transparency required for regulatory compliance, data quality root-cause analysis, and change management.

Lineage cannot be documented statically in spreadsheets for dynamic enterprise environments. It must be actively extracted from systems, code, and transformation engines. Understanding the distinct dimensions of lineage is critical for the Certified Data Management Professional (CDMP) exam. Specifically, candidates must distinguish between Business Lineage and Technical Lineage, as well as Horizontal Lineage and Vertical Lineage.

---

Business Lineage vs. Technical Lineage

The distinction between business and technical lineage lies in the abstraction level and the intended audience.

Business Lineage

Business Lineage is a high-level, conceptual representation of data flow designed for business stakeholders, data stewards, and auditors. It abstracts away technical jargon, database specificities, and minor code-level alterations. Instead, it focuses on the semantic flow of information across major business processes, departments, or applications.

• Granularity: Coarse (system-to-system or process-to-process).
• Key Detail: Shows that "Customer Contact Information" flows from the CRM system to the Billing Application, and subsequently to the Mailing Service.
• Purpose: Strategic impact analysis, regulatory reviews (e.g., showing a GDPR auditor how customer consent data flows through the business), and high-level data stewardship tracking.

Technical Lineage

Technical Lineage is a low-level, detailed representation of data flow designed for database developers, data engineers, and data quality analysts. It details the exact columns, physical tables, stored procedures, and ETL transformations that manipulate the data at every stage.

• Granularity: Fine (column-to-column, attribute-to-attribute, and table-to-table).
• Key Detail: Shows that column CRM_USERS.F_NAME and CRM_USERS.L_NAME are concatenated via an ETL expression CONCAT(F_NAME, ' ', L_NAME) and mapped to the physical database column DW_CUST.FULL_NAME in the STG_CUSTOMER database table.
• Purpose: System debugging, operational impact analysis, data integration design, and automated data quality checks.

---

Horizontal Lineage vs. Vertical Lineage

Lineage is also categorized by its orientation: horizontal (flow across systems) or vertical (mappings across abstraction layers).

graph TD
    subgraph Conceptual Layer
        A[Business Term: Customer Value]
    end
    subgraph Logical Layer
        B[Logical Attribute: Customer Lifetime Value]
    end
    subgraph Physical Layer
        C[Physical Column: T_CUST.cust_ltv_amt]
    end
    A -->|Vertical Lineage Mapping| B
    B -->|Vertical Lineage Mapping| C

    subgraph Horizontal Lineage Flow
        D[CRM Source System] --> E[ETL Staging Table]
        E --> F[Data Warehouse Fact]
        F --> G[BI Dashboard Report]
    end

Horizontal Lineage

Horizontal Lineage (often referred to as data flow lineage or source-to-target lineage) tracks the path of data chronologically as it travels horizontally from its points of origin through ingestion, staging, integration, summary storage, and ultimate consumption.

• It traces data from left to right: Source System -> ETL Ingestion -> Data Warehouse -> Data Mart -> BI Report.
• Horizontal lineage is vital for Data Provenance, proving the origin of a data point, and tracing errors back to the source.

Vertical Lineage

Vertical Lineage maps data elements across different abstraction levels within the same system or domain. It does not track chronological movement. Instead, it establishes semantic connectivity between conceptual, logical, and physical levels.

• It maps from top to bottom: Business Term -> Logical Entity/Attribute -> Physical Table/Column.
• Example: Mapping the business term "Active Customer" to the logical attribute Customer_Status_Code and then to the physical column tbl_customer.cust_status_cd VARCHAR(2).
• Vertical lineage ensures that database tables can be traced directly back to their business definitions and regulatory policies.

---

Change Impact Analysis: Upstream and Downstream

A critical operational use case for data lineage is Change Impact Analysis, which evaluates the consequences of data-related activities. This analysis is split into upstream and downstream traces.

Downstream Impact Analysis (Forward Trace)

Downstream Impact Analysis answers the question: "If I modify or delete this data element, what will break?" It traces the flow forward from a proposed change point to all downstream dependencies.

• Scenario: A database administrator wants to change the data type of T_ORDER.ord_id from an integer to a UUID string.
• Impact Evaluation: By querying the downstream lineage, the DBA discovers that this column feeds:
  1. Two ETL ingestion workflows.
  2. One monthly financial aggregation job.
  3. Three active executive Tableau dashboards.
• The downstream analysis allows the organization to coordinate changes, prevent broken reports, and estimate the true cost of schema updates.

Upstream Impact Analysis (Backward Trace / Root-Cause Analysis)

Upstream Impact Analysis answers the question: "Where did this incorrect value originate, and how was it calculated?" It traces the flow backward from a discovered issue to the upstream source systems.

• Scenario: A financial report displays a negative value for "Customer Lifetime Value," which is business-logically impossible.
• Impact Evaluation: A data quality analyst traces the value backward. They find it was calculated in a summary table from a detail table, which in turn loaded the data from an ETL process that contained a bug in its subtraction logic. The source was a database default value of -1 in the CRM system.
• This backward tracing speeds up Root-Cause Analysis and remediation of data quality issues.

---

Methods of Lineage Harvesting

To capture this metadata, organizations utilize automated lineage harvesting techniques, which fall into three main approaches:

1. Design-based Harvesting: Extracts lineage from metadata integration files created by data modeling tools (e.g., ERwin) or ETL design packages.
2. Execution-based Harvesting: Analyzes execution logs and run-time metadata during active database jobs to capture actual dynamic data flows.
3. Syntax-based / SQL Parsing: Uses specialized parsers to scan SQL scripts, database views, and stored procedures to reverse-engineer logical mappings.

[!WARNING] CDMP Exam Trap: Candidates often confuse "horizontal lineage" with "technical lineage." Remember: Technical lineage is defined by its granularity (column/table level), whereas horizontal lineage is defined by its direction of movement (source-to-target across systems). You can have a high-level "business horizontal lineage" (CRM system to Billing system) or a low-level "technical horizontal lineage" (mapping column X in table A to column Y in table B).