10.2 Data Lineage & Tagging in Unity Catalog

Data Lineage & Tagging in Unity Catalog

Effective data governance requires more than just securing data; it requires understanding exactly where data comes from (lineage) and enriching it with rich metadata for discoverability (tagging). As organizations build complex data pipelines spanning multiple teams, tools, and business units, answering questions like "Which dashboard will break if I drop this table?" or "Where did this PII column originate?" becomes critical to maintaining trust. Unity Catalog provides automated, out-of-the-box lineage tracking alongside powerful tagging capabilities to empower data stewards, data engineers, and business analysts to confidently navigate their lakehouse environment.

Automated Data Lineage

Unity Catalog automatically captures runtime data lineage across queries executed on a Databricks cluster or a Databricks SQL warehouse. Unlike traditional lineage tools that rely on fragile static code parsing or manual definitions, Unity Catalog captures lineage at runtime by directly inspecting the actual execution plan generated by the Spark engine. This means that even highly dynamic PySpark code, complex macros, or dynamically generated SQL statements will produce perfectly accurate lineage graphs.

Lineage is captured at two distinct granularities:

1. Table-level lineage: Shows which tables or views are upstream (sources) or downstream (consumers) of a given table. This is incredibly useful for impact analysis when deprecating old datasets or troubleshooting pipeline failures.
2. Column-level lineage: Provides highly granular tracking of how individual columns are derived, transformed, and loaded. It can trace a specific metric back to its origin column in a raw table, ensuring that sensitive data is appropriately tracked regardless of how it is transformed.

Lineage is captured for a variety of execution contexts, provided they interact directly with Unity Catalog assets:

• Spark SQL and PySpark: Any DataFrame operation or SQL query reading from or writing to Unity Catalog tables automatically generates lineage data without any additional configuration.
• Delta Live Tables (DLT): Executing a DLT pipeline automatically registers comprehensive lineage graph metadata into Unity Catalog, integrating perfectly with DLT's declarative pipeline definitions.
• Dashboards and Workflows: Databricks SQL queries, automated jobs, and dashboard visualizations contribute to the overall lineage graph, tracing data all the way from raw ingestion tables to the final BI consumption layers.

Limitations of Lineage

It is critically important for data engineers and those preparing for the Databricks certification exam to know what is not captured by Unity Catalog lineage:

• Non-UC Assets: Operations interacting with the legacy Hive Metastore (hive_metastore), external databases queried without Lakehouse Federation, or the DBFS root do not generate UC lineage. Legacy systems remain opaque to the modern governance engine.
• Local Driver Operations: If you convert a PySpark DataFrame to a pandas DataFrame (e.g., using .toPandas()) and write it to a local CSV file on the driver node, the lineage chain is immediately broken and not captured. Unity Catalog cannot monitor processes executing strictly within the Python runtime outside of Spark's execution plan.
• V1 Delta / Legacy Formats: Lineage is best supported on Delta Lake tables. Certain operations on legacy formats like raw Parquet or CSV may lack full column-level precision or complete tracking history.

Viewing and Querying Lineage

Users can view lineage visually through the Catalog Explorer in the Databricks UI. By navigating to any table or view, clicking the "Lineage" tab, and selecting either the "Lineage Graph" or "Lineage Table" view, users can trace the data flow interactively. The visual graph is highly intuitive, allowing users to expand upstream or downstream nodes to see exactly how tables connect.

For automated auditing, compliance reporting, and alerting, data engineers can query lineage programmatically using Unity Catalog System Tables. The system.access.table_lineage and system.access.column_lineage tables store historical lineage data, allowing you to write SQL queries to audit data pipelines at scale.

-- Querying table lineage programmatically to find downstream consumers
SELECT source_table_name, target_table_name, target_type 
FROM system.access.table_lineage 
WHERE source_table_full_name = 'prod_catalog.finance_schema.daily_revenue'
  AND event_time > current_date() - INTERVAL 7 DAYS;

Applying Tags to Data Assets

Tagging allows data stewards to categorize and label data assets (including catalogs, schemas, tables, views, and columns) with flexible key-value pairs. This metadata greatly enhances search, discovery, and governance workflows by providing business context that goes beyond standard structural schemas.

Tags are especially useful for:

• Identifying sensitive data: Tagging columns containing Personally Identifiable Information (e.g., pii = true or sensitive = high).
• Data classification: Denoting data classification levels across the enterprise (e.g., classification = confidential or classification = public).
• Ownership and Domains: Assigning ownership or aligning tables with specific business domains (e.g., department = marketing or owner = data_eng_team).

You can seamlessly apply tags using standard SQL syntax with the ALTER TABLE command, making it easy to incorporate tagging into standard deployment scripts or CI/CD pipelines.

-- Applying tags to a table
ALTER TABLE prod_catalog.sales_schema.customers 
  SET TAGS ('pii' = 'true', 'department' = 'sales', 'tier' = 'gold');

-- Applying a tag to a specific column
ALTER TABLE prod_catalog.sales_schema.customers 
  ALTER COLUMN email SET TAGS ('pii' = 'true');

-- Removing a tag from a table
ALTER TABLE prod_catalog.sales_schema.customers 
  UNSET TAGS ('tier');

Search and Discovery in Catalog Explorer

The Catalog Explorer serves as the primary user interface for data discovery and governance in the Databricks Lakehouse. It provides a powerful, Google-like search experience that scans across all Unity Catalog assets that a user has permission to view.

Because Unity Catalog actively indexes metadata, tags, table names, and column names, users can simply search for terms like pii or sales and instantly find all relevant tables, columns, or schemas that match these criteria across the entire metastore. This democratizes data access and significantly reduces the time analysts spend looking for the right datasets.

Data Asset Documentation

Beyond simple key-value tags, Unity Catalog allows users to add rich, descriptive comments to objects. These comments appear prominently in Catalog Explorer and are fully searchable. Databricks even provides AI-generated documentation features that analyze the table schema and automatically suggest detailed descriptions, accelerating the documentation process.

-- Adding documentation to a table
COMMENT ON TABLE prod_catalog.sales_schema.customers 
  IS 'Contains master customer records, including contact information. Source of truth for CRM.';

-- Adding documentation to a column
COMMENT ON COLUMN prod_catalog.sales_schema.customers.email 
  IS 'Customer email address. Highly sensitive PII data.';

Practical Scenario: The PII Audit

Consider a scenario where a corporate compliance officer requests a comprehensive list of all tables in the production environment that contain PII data to ensure GDPR compliance.

1. The data engineering team ensures that all columns containing sensitive data are appropriately tagged during the pipeline deployment using ALTER COLUMN ... SET TAGS ('pii'='true').
2. The compliance officer navigates to the Catalog Explorer and simply searches for the tag pii, instantly receiving a visual list of all restricted assets.
3. Alternatively, a data engineer queries the system.information_schema.column_tags system table to programmatically generate a comprehensive, exportable report of every column across the prod_catalog that possesses the pii tag.
4. Using the system.access.column_lineage table, the engineer can trace the lineage of these PII columns to verify that no untagged downstream tables are improperly deriving data from these restricted columns, thus proving to the compliance officer that data leaks are prevented.

By combining automated lineage, versatile key-value tags, and rich documentation, data stewards can maintain a well-governed data ecosystem where analysts can confidently discover and trust the data they use, while security teams can effortlessly audit access and flow.