1.3 Unity Catalog Namespace & Object Hierarchy

Unity Catalog Namespace & Object Hierarchy

Governance across enterprise data platforms requires a centralized, consistent naming structure and access control model. Prior to Unity Catalog, Databricks workspace governance relied on legacy workspace-local Hive metastores. This legacy model forced organizations to duplicate access policies across individual workspaces and manage fragmented table definitions. Unity Catalog solves these issues by introducing a unified governance layer anchored by a 3-tier namespace.

---

Introduction to Unified Data Governance

Unity Catalog is a centralized governance solution for data, analytics, and AI assets on the Databricks platform. It provides single-pane-of-glass access control, automated data lineage, central auditing, and cross-workspace data discovery across all cloud providers (AWS, Azure, and GCP).

+-----------------------------------------------------------------------+
|                         UNITY CATALOG METASTORE                       |
|                                                                       |
|   +---------------------------------------------------------------+   |
|   |                   CATALOG (First Tier)                        |   |
|   |   e.g., `main`, `finance_prod`, `sandbox`                     |   |
|   +---------------------------------------------------------------+   |
|                                   |                                   |
|                                   v                                   |
|   +---------------------------------------------------------------+   |
|   |                   SCHEMA / DATABASE (Second Tier)             |   |
|   |   e.g., `sales`, `hr`, `telemetry`                            |   |
|   +---------------------------------------------------------------+   |
|                                   |                                   |
|                                   v                                   |
|   +---------------------------------------------------------------+   |
|   |                   OBJECT (Third Tier)                         |   |
|   |   Tables, Views, Materialized Views, Volumes, Functions       |   |
|   +---------------------------------------------------------------+   |
+-----------------------------------------------------------------------+

---

Metastore & Multi-Workspace Binding

The Metastore is the top-level logical container within Unity Catalog. It registers data assets, metadata, security privileges, and service credentials for an entire cloud region.

Key Metastore Characteristics

• One Metastore per Cloud Region: An organization typically provisions one Unity Catalog metastore per cloud region (e.g., aws-us-east-1 metastore).
• Multi-Workspace Attachment: Multiple Databricks workspaces in the same region bind to the exact same metastore. This ensures that a table created in Workspace A is instantly discoverable and queryable in Workspace B (subject to access permissions).
• Metastore Admin: A dedicated administrative role responsible for managing storage locations, binding workspaces, and assigning catalog-level privileges.

---

The Three-Tier Namespace Architecture

To reference any data object in Databricks SQL or code notebooks, Unity Catalog enforces a strict 3-tier namespace structure:

$\text{Object Identifier} = \text{catalog\_name}.\text{schema\_name}.\text{object\_name}$

1. Catalog (Tier 1)

The highest grouping level within a metastore. Catalogs are frequently aligned with environment boundaries (e.g., prod, dev, staging) or business domains (e.g., finance, marketing, supply_chain).

2. Schema / Database (Tier 2)

A schema (used interchangeably with the term database) resides inside a catalog. Schemas organize related logical assets such as tables, views, functions, and volumes (e.g., finance_prod.accounts_receivable).

3. Object (Tier 3)

The actual data asset contained within a schema. Supported objects include:

• Tables: Delta Lake format tabular data.
• Views & Materialized Views: Saved SQL queries and pre-computed analytical results.
• Volumes: Governed file storage for non-tabular data.
• Functions: User-defined SQL or Python functions (UDFs).
• Registered Models: MLflow machine learning model artifacts.

Default Catalog Resolution & Session Scoping

Data analysts can set session-level defaults to avoid typing fully qualified 3-tier names in every SQL statement:

-- Set active catalog and schema for the current session
USE CATALOG prod_catalog;
USE SCHEMA sales_schema;

-- Querying using implied session defaults (resolves to prod_catalog.sales_schema.orders)
SELECT order_id, customer_id, total_amount 
FROM orders 
WHERE order_date >= '2026-01-01';

-- Querying using explicit fully qualified 3-tier namespace
SELECT * 
FROM dev_catalog.sandbox_schema.test_orders;

---

Managed Tables vs. External Tables

In Unity Catalog, tables are classified based on how physical data files are managed in cloud storage.

Managed Tables

Managed tables are the default and recommended table type in Unity Catalog.

• File Management: Unity Catalog fully manages both the metadata and the physical Delta Lake data files in cloud object storage.
• Storage Location: Physical data files are stored in the root storage location configured for the parent catalog or schema.
• Lifecycle on DROP TABLE: When a user executes DROP TABLE table_name, Unity Catalog deletes both the table metadata from the catalog and the underlying physical data files from cloud storage after a safety purge window.

External Tables

External tables are used when data files are produced by external systems or must reside in custom cloud storage paths.

• File Management: Unity Catalog manages table metadata and access control, but does not manage the lifecycle of physical files.
• Storage Location: Physical files reside in a specific cloud path (e.g., s3://my-company-bucket/external_sales/) registered via an External Location.
• Lifecycle on DROP TABLE: When a user executes DROP TABLE table_name, Unity Catalog deletes only the metadata registration. The underlying physical files in cloud storage remain untouched.

---

Storage Credentials & External Locations

To establish secure access to external cloud storage without exposing cloud secret keys in SQL code, Unity Catalog uses two security objects:

1. Storage Credential: An object that encapsulates a cloud IAM role (AWS IAM Role, Azure Managed Identity, or GCP Service Account).
2. External Location: An object that combines a cloud storage URI with a Storage Credential (e.g., CREATE EXTERNAL LOCATION s3_sales_location URL 's3://acme-sales-data/' WITH (STORAGE CREDENTIAL aws_iam_cred);).

Analysts and engineers must have CREATE EXTERNAL TABLE or READ FILES privileges granted on an External Location to read or write external data assets.

---

Volumes: Governed Non-Tabular Data Storage

Traditional data platforms struggle to govern unstructured and semi-structured files such as PDFs, DICOM medical images, audio recordings, video files, or raw JSON drop files. Unity Catalog introduces Volumes as first-class governed objects within the 3-tier namespace.

• Managed Volumes: Stored in the default storage location of the parent schema. Dropping a managed volume deletes all contained physical files.
• External Volumes: Point to a custom cloud storage path registered via an External Location. Dropping an external volume leaves physical files intact.

-- Accessing files stored inside a Unity Catalog Volume via SQL / POSIX paths
SELECT * 
FROM read_files('/Volumes/prod_catalog/raw_schema/unstructured_docs/*.pdf');

---

Practical SQL Syntax & Asset Creation

Below is a complete DDL script demonstrating the creation of a governed Unity Catalog hierarchy:

-- Step 1: Create top-level Catalog
CREATE CATALOG IF NOT EXISTS finance_prod
COMMENT 'Production catalog for financial reporting and analytics';

-- Step 2: Create Schema within Catalog
CREATE SCHEMA IF NOT EXISTS finance_prod.revenue_schema
COMMENT 'Contains quarterly revenue tables and views';

-- Step 3: Create Managed Table in 3-Tier Namespace
CREATE TABLE IF NOT EXISTS finance_prod.revenue_schema.quarterly_actuals (
  fiscal_year INT,
  quarter STRING,
  revenue_usd DECIMAL(18, 2),
  updated_at TIMESTAMP
)
USING DELTA
COMMENT 'Managed Delta table tracking quarterly revenue';

-- Step 4: Create Governed Managed Volume for Financial PDF Audits
CREATE VOLUME IF NOT EXISTS finance_prod.revenue_schema.audit_pdfs
COMMENT 'Managed volume storing official PDF financial statements';