2.1 Dataset Discovery & Asset Tagging

2.1 Dataset Discovery & Asset Tagging

Data discovery and asset tagging form the cornerstone of effective data governance within the Databricks Data Intelligence Platform. As organizations expand their lakehouses to house petabytes of analytical data across thousands of tables, data analysts must be equipped to quickly locate, evaluate, and categorize governed data assets. Unity Catalog provides a centralized governance layer that unifies dataset discovery, rich metadata annotation, key-value asset tagging, and automated system auditing across all workspace environments.

---

Discovering Governed Assets in Unity Catalog

Finding the right dataset quickly is critical for building accurate analytical queries, dashboards, and machine learning features. Databricks combines visual navigation with intelligent search engines to simplify dataset discovery.

Catalog Explorer and Workspace Search

Catalog Explorer serves as the primary visual interface for inspecting Unity Catalog objects. It enables data analysts to browse through the three-level namespace hierarchy (catalog.schema.table_or_view), inspect table schemas, view data previews, examine table properties, and verify access permissions.

Complementing Catalog Explorer, Databricks Workspace Search provides a universal search bar powered by Databricks IQ. The search engine indexes metadata across the workspace within 30 seconds of object creation or alteration. Key search capabilities include:

• Multi-attribute Indexing: Workspace search indexes catalog names, schema names, table names, column names, column comments, table tags, and column tags.
• Semantic Search: Users can search using plain language terms (e.g., "customer churn 2025") to find relevant tables even if exact column names differ.
• Faceted Filtering: Analysts can narrow search results by catalog, schema, asset owner, object type (table, view, volume, model), and specific tag key-value pairs.

Unity Catalog Object Hierarchy and Metadata

All discoverable assets in Databricks adhere to the three-level namespace:

1. Catalog: The high-level container for organizing data assets (e.g., main, finance_prod, sandbox).
2. Schema (Database): The logical grouping of tables, views, functions, and volumes within a catalog (e.g., sales, hr, raw).
3. Table / View / Volume: The leaf objects containing data rows or files (e.g., dim_customers, fact_transactions).

Rich documentation enhances discoverability. Analysts can add explicit comments to tables and columns using standard SQL statements:

-- Annotating table and column metadata
COMMENT ON TABLE main.finance_prod.fact_transactions 
IS 'Contains finalized enterprise sales transactions processed daily via Delta Live Tables.';

ALTER TABLE main.finance_prod.fact_transactions 
ALTER COLUMN transaction_amount COMMENT 'Net transaction amount in USD after applying discounts and taxes.';

Databricks IQ also offers AI-generated documentation, suggesting descriptions for tables and columns based on schema structure, sample data distributions, and column names. Analysts can review, edit, and accept these AI suggestions with one click in Catalog Explorer.

---

Certified Datasets in Unity Catalog

The Databricks Data Analyst Associate exam explicitly tests discovering, querying, cleaning, and managing certified datasets. In Unity Catalog, certification is not a separate object type — it is a governed trust signal applied through the system tag system.certification_status.

How analysts discover certified datasets

1. Open Catalog Explorer and look for the certification badge on tables and views.
2. Search workspace assets and filter by the certification status when available.
3. Query governance metadata to automate discovery:

-- Apply certification (requires ASSIGN on system.certification_status)
SET TAG ON TABLE main.gold.orders `system.certification_status` = `certified`;

-- Discover certified tables via tag metadata
SELECT catalog_name, schema_name, table_name, tag_name, tag_value
FROM system.information_schema.table_tags
WHERE tag_name = 'system.certification_status'
  AND tag_value = 'certified';

Workspace search also supports type:table certificationStatus:certified to filter certified tables.

Analyst workflow implications

• Prefer certified gold/silver tables for dashboards, Genie spaces, and exam-style “trusted source” scenarios.
• Certification does not replace privileges: you still need USE CATALOG, USE SCHEMA, and SELECT (or equivalent) to query the data.
• Owners or users with APPLY TAG (and governed-tag ASSIGN where required) set or clear certification; analysts primarily consume the signal.
• Pair certification with lineage and comments: a certified badge answers “is this trusted?”, while lineage answers “where did it come from?”

Implementing Metadata and Asset Tagging

While descriptive comments provide human-readable context, asset tagging enables structured categorization, programmatic policy enforcement, and regulatory compliance tracking across the entire data estate.

Tag Structure and Governance Standards

Tags in Unity Catalog are key-value pairs assigned directly to catalogs, schemas, tables, views, and individual columns. Standardizing tag conventions ensures data consistency across enterprise business units.

• Character Limits: Tag keys allow up to 255 characters; tag values allow up to 1,000 characters. You can assign a maximum of 50 tags to a single securable object, and a table can have at most 1,000 column tags across all columns.
• Case Sensitivity: Tag keys are case-sensitive. For example, Sales and sales are two distinct tags in Unity Catalog.
• Tag Inheritance: Tags applied at the catalog or schema level do not automatically propagate down to child tables or columns as active inherited attributes. However, catalog and schema tags serve as logical groupings during discovery and audit reporting.

Common enterprise tagging frameworks include:

• Data Sensitivity / PII: Key: PII, Values: true, false, Direct_Identifier, Quasi_Identifier.
• Data Quality Tier: Key: Tier, Values: Bronze, Silver, Gold.
• Asset Ownership: Key: Owner_Team, Values: Finance-Analytics, Marketing-Ops.
• Compliance Scope: Key: Compliance, Values: GDPR, CCPA, HIPAA, SOX.

Applying Tags via SQL and UI

Analysts and data stewards can apply tags interactively through the Catalog Explorer GUI or programmatically via Databricks SQL execution.

-- Applying table-level key-value tags
ALTER TABLE main.finance_prod.dim_customers 
SET TAGS (
  'Tier' = 'Gold',
  'Owner_Team' = 'Customer-Analytics',
  'Compliance' = 'GDPR'
);

-- Applying column-level key-value tags for PII tracking
ALTER TABLE main.finance_prod.dim_customers 
ALTER COLUMN ssn SET TAGS ('PII' = 'Direct_Identifier', 'Confidentiality' = 'Restricted');

ALTER TABLE main.finance_prod.dim_customers 
ALTER COLUMN email SET TAGS ('PII' = 'Quasi_Identifier', 'Confidentiality' = 'Confidential');

-- Removing specific tags from a table
ALTER TABLE main.finance_prod.dim_customers 
UNSET TAGS ('Compliance');

---

Auditing Governance via Information Schema

To verify governance compliance at scale, data analysts can query Unity Catalog's read-only Information Schema system tables. The Information Schema provides standardized SQL views detailing catalog structure, permissions, and active tag assignments.

Querying System Tables for Tag Audit

The primary system views for inspecting asset tags are information_schema.table_tags and information_schema.column_tags. These views can be joined with information_schema.tables and information_schema.columns to perform comprehensive compliance reporting across 10,000+ workspace assets.

-- Identifying all un-tagged Gold tables in the finance catalog
SELECT 
  t.table_schema,
  t.table_name,
  t.table_type
FROM main.information_schema.tables t
LEFT JOIN main.information_schema.table_tags tag
  ON t.table_schema = tag.schema_name
  AND t.table_name = tag.table_name
  AND tag.tag_name = 'Tier'
WHERE t.table_schema = 'finance_prod'
  AND tag.tag_value IS NULL;

---

Real-World Scenario: Compliance Audit for PII Discovery

Suppose a lead data analyst at a healthcare financial firm must prepare for a European GDPR compliance audit scheduled for May 2026. The auditor requires a list of all columns across the patient_billing schema that store direct personal identifiers.

1. Tag Application: The data engineering team tags candidate columns using ALTER TABLE patient_billing.claims ALTER COLUMN national_id SET TAGS ('PII' = 'Direct_Identifier').
2. Programmatic Audit: The analyst executes the following query against information_schema.column_tags to generate the compliance report:

SELECT 
  table_name,
  column_name,
  tag_value AS pii_classification
FROM main.information_schema.column_tags
WHERE schema_name = 'patient_billing'
  AND tag_name = 'PII'
ORDER BY table_name, column_name;

1. Validation: The resulting output provides audit proof of all tagged PII elements, enabling governance teams to configure downstream Unity Catalog fine-grained row/column access control policies.