5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags

5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags

DP-750 Exam Focus: Understand how Attribute-Based Access Control (ABAC) scales data governance in Azure Databricks. Master the DDL syntax for applying metadata tags (SET TAGS, UNSET TAGS) across catalogs, schemas, tables, and columns. Learn how tag-based masking policies automatically protect sensitive columns across the entire metastore, how policy inheritance operates, and how to query system.information_schema to audit classification compliance.

---

1. RBAC vs. ABAC: The Governance Scalability Challenge

As enterprise lakehouses expand to house thousands of tables and petabytes of data, traditional Role-Based Access Control (RBAC) encounters severe operational bottlenecks:

• Role Explosion: Organizations end up creating hundreds of bespoke roles (e.g., us_finance_analyst_masked, eu_marketing_lead_unmasked) to handle subtle combinations of access needs.
• Manual Schema Maintenance: Whenever a data engineer creates a new table containing Personally Identifiable Information (PII), security administrators must manually write and apply individual ALTER COLUMN ... SET MASK statements on that specific table.
• High Risk of Human Error: If a developer forgets to apply a column mask to a newly created table in a bronze or silver schema, sensitive data remains exposed until the next compliance audit.

Attribute-Based Access Control (ABAC) solves these challenges by decoupling access policies from individual table schemas. In ABAC, security decisions are determined dynamically at query runtime by comparing subject attributes (such as the user's group, department, or location) against object attributes (such as metadata classification tags applied to catalogs, tables, or columns).

+-----------------------------------------------------------------------------------------+
|                           RBAC VS. ABAC ARCHITECTURE COMPARISON                         |
+-----------------------------------------------------------------------------------------+
|                                                                                         |
|  TRADITIONAL RBAC (Role-Based):                                                         |
|  [ User ] ---> ( Assigned Role ) ---> [ Explicit GRANT on Specific Table/Column ]       |
|  * Bottleneck: Must explicitly apply masks and permissions to thousands of columns.     |
|                                                                                         |
|  MODERN ABAC (Attribute-Based):                                                         |
|  [ User Attributes ]  <--- Evaluated at Query Runtime --->  [ Securable Object Tags ]    |
|  (Group: 'compliance')                                      (Column Tag: pii='ssn')    |
|                                           |                                             |
|                                           v                                             |
|                           [ Automated Tag Masking Policy ]                              |
|                           - Applied globally across entire metastore                    |
|                           - Protects any column tagged 'ssn' automatically              |
+-----------------------------------------------------------------------------------------+

---

2. Unity Catalog Tagging Taxonomy & Mechanics

Unity Catalog supports applying metadata tags to any securable object within the 3-level namespace hierarchy: Catalogs, Schemas, Tables, Views, Volumes, and Columns.

Tag Structure: Key-Value Tags and Tag-Only Labels

• Key-Value Tags: Composed of a key and a value string (e.g., 'environment' = 'production', 'pii_classification' = 'restricted'). Used for granular policy evaluation and detailed metadata categorization.
• Tag-Only Labels (Simple Tags): Contain only a key with no value or an empty string (e.g., 'deprecated', 'gdpr_article_9').

ANSI SQL DDL Syntax for Tag Management

1. Tagging Catalogs and Schemas

-- Apply classification and ownership tags to a catalog
ALTER CATALOG prod_finance 
SET TAGS (
  'data_tier' = 'enterprise_curated',
  'compliance_scope' = 'sox_pci',
  'owner_team' = 'finance_data_eng'
);

-- Apply confidentiality tags to a schema
ALTER SCHEMA prod_finance.payroll 
SET TAGS (
  'data_classification' = 'highly_confidential',
  'retention_days' = '2555'
);

2. Tagging Tables and Views

-- Apply multiple tags to an existing Delta table
ALTER TABLE prod_finance.payroll.employee_salaries 
SET TAGS (
  'contains_pii' = 'true',
  'data_domain' = 'human_resources',
  'governance_review' = 'passed_2026_q3'
);

-- Remove a tag from a table
ALTER TABLE prod_finance.payroll.employee_salaries 
UNSET TAGS ('governance_review');

3. Tagging Individual Columns

-- Tagging sensitive columns with specific PII classification types
ALTER TABLE prod_finance.payroll.employee_salaries 
ALTER COLUMN national_insurance_number 
SET TAGS ('pii_type' = 'ssn', 'sensitivity' = 'critical');

ALTER TABLE prod_finance.payroll.employee_salaries 
ALTER COLUMN work_email 
SET TAGS ('pii_type' = 'email', 'sensitivity' = 'moderate');

ALTER TABLE prod_finance.payroll.employee_salaries 
ALTER COLUMN bank_iban 
SET TAGS ('financial_data' = 'iban', 'sensitivity' = 'critical');

4. Applying Tags during Table Creation

CREATE TABLE prod_finance.payroll.contractor_payments (
  contractor_id STRING NOT NULL,
  tax_id STRING NOT NULL COMMENT 'National Tax Identifier' WITH TAGS ('pii_type' = 'tax_id'),
  routing_number STRING WITH TAGS ('financial_data' = 'routing_number'),
  payment_amount DECIMAL(18,2)
)
SET TAGS ('business_domain' = 'procurement');

---

3. Data Classification & PII Governance Frameworks

Organizations implementing Unity Catalog ABAC establish a standardized Classification Taxonomy. This taxonomy guarantees consistent terminology across data engineering, security, and legal teams.

Enterprise Data Classification Matrix

---

4. Automated Tag-Based Masking Policies

The most powerful capability of Unity Catalog ABAC is Tag-Based Column Masking. Instead of binding a masking function to a specific table column, security administrators bind the masking function to a Metadata Tag Key or Key-Value Pair.

                               TAG-BASED MASKING ARCHITECTURE
                               
  1. Define Central Mask Function:
     CREATE FUNCTION ssn_tag_mask(val STRING) RETURNS STRING ...
                                  |
                                  v
  2. Create Tag-Based Masking Policy:
     Bind ssn_tag_mask() to Tag: [ pii_type = 'ssn' ]
                                  |
                                  v
  +-------------------------------+-------------------------------+
  |                               |                               |
  v                               v                               v
[ Table: customers ]            [ Table: employees ]            [ Table: applicants ]
Column: ssn                     Column: tax_num                 Column: ssn_identifier
Tag: pii_type = 'ssn'           Tag: pii_type = 'ssn'           Tag: pii_type = 'ssn'
  |                               |                               |
  +-------------------------------+-------------------------------+
                                  |
                                  v
  [ AUTOMATIC ENFORCEMENT: All three columns masked across all queries without table DDL ]

Step-by-Step Implementation of Tag-Based Masking

Step 1: Author the Governance Mask Function

-- Create the generic mask function in the central governance schema
CREATE OR REPLACE FUNCTION prod.governance.generic_ssn_mask(val STRING)
RETURNS STRING
LANGUAGE SQL
DETERMINISTIC
RETURN
  CASE 
    WHEN is_account_group_member('data_security_officers') THEN val
    WHEN is_account_group_member('payroll_auditors') THEN val
    WHEN val IS NULL THEN NULL
    ELSE concat('***-**-', right(regexp_replace(val, '[^0-9]', ''), 4))
  END;

Step 2: Bind the Mask Function to the Classification Tag (Catalog Explorer or Policy API)

In Unity Catalog, tag-based policies are registered such that any column possessing the tag 'pii_type' = 'ssn' automatically inherits the generic_ssn_mask policy.

Step 3: Automated Enforcement on Schema Evolution

When data engineers ingest new datasets or add columns to existing tables, they simply apply the classification tag:

-- Data engineer creates a brand new table
CREATE TABLE prod.marketing.lead_capture (
  lead_id BIGINT,
  full_name STRING,
  tax_id STRING WITH TAGS ('pii_type' = 'ssn'),
  email_address STRING WITH TAGS ('pii_type' = 'email')
);

-- Result: The tax_id column is INSTANTLY masked for unauthorized users.
-- The security team never had to write an ALTER TABLE statement.

---

5. Policy Conflict Resolution & Evaluation Precedence

When multiple security policies and tags intersect on a single table or column, Unity Catalog evaluates access rules according to strict, deterministic Precedence Rules:

+-----------------------------------------------------------------------------------------+
|                        POLICY RESOLUTION PRECEDENCE HIERARCHY                           |
+-----------------------------------------------------------------------------------------+
|                                                                                         |
|  HIGHEST PRECEDENCE                                                                     |
|  [ 1. Direct Column Mask (ALTER TABLE ... ALTER COLUMN ... SET MASK) ]                 |
|     - If a column has an explicit, direct mask assigned via DDL, it completely         |
|       overrides any tag-based masking policies.                                         |
|                                                                                         |
|  [ 2. Tag-Based Masking Policy on Column Tag ]                                          |
|     - Evaluates policies attached to tags explicitly placed on the column.              |
|                                                                                         |
|  [ 3. Tag-Based Policy on Inherited Parent Tags (Table -> Schema -> Catalog) ]          |
|     - Evaluates policies matching tags inherited from parent securable hierarchy.       |
|                                                                                         |
|  LOWEST PRECEDENCE (No Policy: Full Unmasked Value Returned if user has SELECT)         |
+-----------------------------------------------------------------------------------------+

Exam Trap: If a column has both an explicit direct column mask (applied via ALTER COLUMN SET MASK) AND an ABAC tag-based policy (applied via pii_type = 'ssn'), Unity Catalog always executes the direct column mask and ignores the tag-based policy. Always remove explicit column masks if you intend to standardize on tag-based policies.

---

6. Auditing & Compliance Monitoring with System Information Schema

Unity Catalog exposes all tagging metadata through SQL-queryable system views located in system.information_schema. Security officers and data engineers can write automated compliance queries to discover unmasked PII, verify tag coverage, and monitor data governance health.

                               INFORMATION SCHEMA TELEMETRY
                               
  +-------------------------------------------------------------------------+
  |                   system.information_schema Catalog                     |
  +-------------------------------------------------------------------------+
  |                                                                         |
  |  +--------------------+  +--------------------+  +--------------------+ |
  |  |    column_tags     |  |     table_tags     |  |    schema_tags     | |
  |  | (Column-level tags)|  | (Table-level tags) |  | (Schema-level tags)| |
  |  +--------------------+  +--------------------+  +--------------------+ |
  +-------------------------------------------------------------------------+

Essential Governance Audit Queries

1. Identify All Columns Tagged as PII Across the Entire Metastore

SELECT 
  catalog_name,
  schema_name,
  table_name,
  column_name,
  tag_name,
  tag_value
FROM system.information_schema.column_tags
WHERE tag_name = 'pii_type'
ORDER BY catalog_name, schema_name, table_name;

2. Detect Untagged Tables in Production Schemas (Compliance Gap Analysis)

SELECT 
  t.table_catalog,
  t.table_schema,
  t.table_name
FROM system.information_schema.tables t
LEFT JOIN system.information_schema.table_tags tg
  ON t.table_catalog = tg.catalog_name
  AND t.table_schema = tg.schema_name
  AND t.table_name = tg.table_name
WHERE t.table_catalog = 'prod_enterprise'
  AND t.table_type = 'MANAGED'
  AND tg.tag_name IS NULL;

3. Audit Tables Tagged as Containing PII but Lacking Column Classification

-- Find tables labeled as contains_pii = 'true' where no individual columns are tagged
SELECT 
  tt.catalog_name,
  tt.schema_name,
  tt.table_name
FROM system.information_schema.table_tags tt
WHERE tt.tag_name = 'contains_pii' AND tt.tag_value = 'true'
  AND NOT EXISTS (
    SELECT 1 
    FROM system.information_schema.column_tags ct
    WHERE ct.catalog_name = tt.catalog_name
      AND ct.schema_name = tt.schema_name
      AND ct.table_name = tt.table_name
  );