11.3 Data Governance, Cataloging, and Discovery with Dataplex

11.3 Data Governance, Cataloging, and Discovery with Dataplex

[!IMPORTANT] Data governance, fine-grained access control, and metadata management are core pillars of the Google Cloud Professional Data Engineer exam. Expect in-depth questions on how to architect decentralized data lakes using Dataplex, how to implement column-level security and dynamic data masking using Policy Tags in BigQuery, how to configure Row-Level Security policies, and how to automate data quality validation and lineage tracking without operating custom compute infrastructure.

As enterprise data architectures expand, organizations transition from monolithic data warehouses toward distributed data mesh topologies. Modern enterprises manage petabytes of data distributed across thousands of Cloud Storage buckets (containing raw Parquet, ORC, JSON, and Avro files) and hundreds of BigQuery datasets across multiple projects and regions.

This distribution introduces severe governance vulnerabilities: data silos proliferate, schema drift goes undetected, sensitive Personally Identifiable Information (PII) is copied without audit trails, and data quality degrades. Google Cloud Dataplex and its Knowledge Catalog (the catalog the exam guide calls Dataplex Catalog) provide the centralized intelligent data fabric required to govern, secure, and discover distributed data assets at scale.

---

Dataplex Hierarchical Architecture: Lakes, Zones, and Assets

Dataplex organizes distributed physical storage resources into a structured, domain-oriented logical hierarchy without requiring physical data movement, consolidation, or replication.

+-------------------------------------------------------------------------+
|                        Dataplex Lake (Domain)                           |
|               e.g., 'sales-domain-lake' or 'customer-360'               |
+-------------------------------------------------------------------------+
                                     |
         +---------------------------+---------------------------+
         |                                                       |
         v                                                       v
+---------------------------------+     +---------------------------------+
|        Raw Data Zone            |     |       Curated Data Zone         |
|  * Ingestion landing area       |     |  * Cleaned, analytical data     |
|  * Accepts varied raw formats   |     |  * Enforces schema standards    |
|  * Relaxed validation rules     |     |  * Optimized Parquet / BQ tables|
+---------------------------------+     +---------------------------------+
         |                                                       |
         v                                                       v
+---------------------------------+     +---------------------------------+
|         Assets Mapped           |     |         Assets Mapped           |
|  • Cloud Storage Bucket         |     |  • BigQuery Analytical Dataset  |
|    (gs://raw-telemetry-landing) |     |    (project:curated_sales_mart) |
|  • Cloud Storage Bucket         |     |  • Cloud Storage Curated Parquet|
|    (gs://partner-csv-drop)      |     |    (gs://curated-lake-parquet)  |
+---------------------------------+     +---------------------------------+

1. Lakes (Business Domains)

A Lake represents the top-level logical container in Dataplex, typically aligned with a business domain, functional unit, or regulatory jurisdiction (e.g., retail-operations-lake, financial-risk-lake, healthcare-analytics-lake). Lakes define the primary administrative boundary for resource policy, billing assignment, and domain-wide security defaults.

2. Zones (Storage Categories and Standards)

Inside a Lake, data assets are organized into Zones. Zones categorize data according to its processing maturity, data readiness, and access requirements. Dataplex provides two foundational zone types:

• Raw Zones: Designed for landing zones and raw ingestion pipelines. Raw zones accept unstructured, semi-structured, and structured data in arbitrary formats (raw JSON, CSV, logs). Validation rules can be relaxed, allowing upstream systems to land files without immediate schema failure.
• Curated Zones: Designed for processed, cleaned, and standardized analytical data. Curated zones enforce specific storage formats (such as Parquet, ORC, or native BigQuery managed tables). Data in curated zones must adhere to strict schema definitions and partitioning standards, making it immediately queryable by business analysts and BI dashboards.

3. Assets (Physical Storage Mapping)

An Asset maps an underlying physical Google Cloud storage resource directly into a Dataplex Zone. Assets can point to:

• A Cloud Storage bucket (or bucket prefix).
• A BigQuery dataset.

Zero Data Movement: Attaching a Cloud Storage bucket or BigQuery dataset as an asset does not move, copy, or transform the underlying bytes. The data remains in its native storage tier (Colossus or GCS), retaining its existing performance characteristics, while Dataplex overlays unified governance, automated discovery, metadata management, and centralized security access policies.

---

Automated Metadata Discovery and Cataloging

Manually documenting schemas, tables, and partitions across thousands of Cloud Storage buckets is impossible at enterprise scale. Dataplex incorporates a fully managed, serverless metadata discovery engine.

When a Cloud Storage bucket is registered as an asset, the Dataplex discovery service automatically:

1. Scans Storage Files: Inspects structured and semi-structured files (Parquet, ORC, Avro, CSV, JSON) stored within the bucket.
2. Infers Schemas: Analyzes file headers and records to infer column names, nested record definitions, and data types (such as INT64, TIMESTAMP, STRING).
3. Extracts Hive-Style Partitions: Parses directory path patterns matching Hive partitioning conventions (e.g., gs://bucket/sales/year=2026/month=09/day=14/data.parquet) and identifies the partition keys and data types.
4. Publishes External Tables to BigQuery & the Catalog: Automatically creates and updates external table definitions in the BigQuery metastore and entries in the Dataplex Catalog. Analysts can query sales immediately using standard SQL without manually executing CREATE EXTERNAL TABLE statements.
5. Detects Schema Evolution and Drift: Tracks when new columns are added or data types change across files, alerting administrators when schema drift or file corruption occurs.

---

Cataloging: From Data Catalog to Knowledge Catalog

The exam guide lists "Dataplex, Dataplex Catalog" among the tools for building a data platform, and this is an area where the product naming moved faster than most study material. Get the current state straight before anything else:

• Data Catalog — the standalone catalog product — was deprecated on February 3, 2025 and shut down on June 1, 2026. Its business glossary was deprecated on May 19, 2025 with the same June 1, 2026 shutdown.
• Its replacement is Dataplex Universal Catalog, which as of April 10, 2026 is called Knowledge Catalog. The API, client library, CLI, and IAM names are unchanged, which is why the exam guide's "Dataplex Catalog" wording, the dataplex.googleapis.com API, and the roles/dataplex.* IAM roles all still line up.
• Policy tags and policy tag taxonomies are explicitly not deprecated. They remain the mechanism for BigQuery column-level access control, covered in the next part of this section. Do not let the catalog transition confuse you into thinking column-level security changed — it did not.

The Concept Mapping You Must Know

Older material describes Data Catalog's tag model. Google documents a direct mapping into Knowledge Catalog's aspect model:

If an exam question uses "tag template," read it as "aspect type." If it uses "policy tag," it means exactly what it has always meant.

Search Across the Catalog

The catalog indexes BigQuery tables and views, Dataplex assets, Pub/Sub topics, Spanner databases, and Cloud Storage data, and supports faceted search across physical resource names, column names, project IDs, and attached metadata:

• Search by attached metadata: tag:DataClassification=RESTRICTED
• Search by column: column:credit_card_number
• Search by resource type and parent: type=TABLE parent:project.sales_dataset

Structured Metadata Instead of Free Text

Rather than relying on unstructured descriptions, governance is enforced with a formal schema — a tag template in the legacy vocabulary, an aspect type in the current one:

Aspect type: ComplianceGovernance
  - DataClassification (ENUM: PUBLIC, INTERNAL, CONFIDENTIAL, RESTRICTED)
  - ContainsPII        (BOOLEAN: TRUE, FALSE)
  - DataOwner          (STRING: security-team@example.com)
  - RetentionPeriodDays(INT64: 2555)
  - LastAuditedDate    (TIMESTAMP: 2026-09-14 00:00:00 UTC)

Structured metadata is what makes catalog search a governance control rather than a documentation exercise: an auditor can enumerate every asset where ContainsPII = TRUE in one query, which is impossible when the same fact lives in a free-text description.

Separately, the catalog distinguishes metadata that everyone with read access can see from metadata gated behind its own fine-grained IAM permissions. That separation is what lets security, audit, and compliance teams attach sensitive annotations — vulnerability assessments, legal-hold markers, detailed PII findings — to an asset without exposing those annotations to every analyst who can query the underlying table.

---

Column-Level Security and Dynamic Data Masking

Enterprise data warehouses frequently store sensitive columns (such as Social Security Numbers, credit card numbers, salaries, or email addresses) alongside non-sensitive operational attributes in the same table. Granting table-level access exposes sensitive PII, while duplicating tables into "sanitized" versions creates maintenance overhead and data sprawl.

BigQuery addresses this using Policy Tags managed through Dataplex Catalog taxonomy hierarchies (policy tags and policy tag taxonomies are explicitly not deprecated).

Step-by-Step Architecture for Policy Tag Enforcement

1. Define Taxonomy & Policy Tags (Dataplex Catalog)
   Taxonomy: PII_Taxonomy
      ├── High_Sensitivity (Taxonomy Node)
      │      └── SSN_Policy_Tag
      └── Low_Sensitivity (Taxonomy Node)
             └── Email_Policy_Tag

2. Associate Policy Tag with BigQuery Column
   CREATE TABLE customer_records (
     customer_id STRING,
     signup_date DATE,
     email_address STRING OPTIONS(policy_tags=['projects/.../policyTags/Email_Policy_Tag']),
     ssn STRING OPTIONS(policy_tags=['projects/.../policyTags/SSN_Policy_Tag'])
   );

3. Assign IAM Permissions & Masking Rules
   - Compliance Officer -> roles/datacatalog.categoryFineGrainedReader (Unmasked Plaintext)
   - Data Analyst      -> roles/datacatalog.maskedUser + SHA-256 Masking Rule (Hashed Output)
   - Unprivileged User -> No Policy Tag Role (Query Fails if column referenced)

Dynamic Data Masking Rules

When users without plaintext access query a table, standard column-level security blocks the entire query with an authorization failure. Dynamic Data Masking allows non-privileged users to query the table successfully while masking the sensitive values dynamically at query execution time.

To enable dynamic data masking, administrators grant the user or group roles/datacatalog.maskedUser on the policy tag and configure a Data Masking Rule on the tag:

1. Default Masking Value: Returns the standard system default value based on the column's data type: "" (empty string) for STRING, 0 for integer types, 0.0 for floats, 0001-01-01 for DATE, and false for BOOL.
2. Nullify: Returns a SQL NULL for the masked column across all rows.
3. Hash (SHA-256): Computes a deterministic SHA-256 cryptographic hash of the input value.
   • Why this is critical for analytics: Because SHA-256 hashing is deterministic ($H(x) = H(x)$), analysts can perform COUNT(DISTINCT ssn), GROUP BY ssn, or JOIN operations across tables on the hashed identifier without ever decrypting or viewing the real Social Security Number.
4. Regex Masking (Custom Pattern Masking): Applies regular expression replacement rules (e.g., masking credit card digits so only the last four are visible: XXXX-XXXX-XXXX-1234, or masking email domains: XXX@company.com).

---

Row-Level Security (RLS) in BigQuery

While column-level security protects vertical slices of a table, multi-tenant architectures and data mesh domains require restricting horizontal slices (rows) based on user identity or organizational roles. BigQuery provides Row-Level Security through row access policies.

Defining Row Access Policies

Row access policies are created using standard DDL:

CREATE OR REPLACE ROW ACCESS POLICY regional_sales_filter
ON `enterprise_sales.orders`
GRANT TO ('group:emea-analysts@company.com')
FILTER USING (region = 'EMEA');

CREATE OR REPLACE ROW ACCESS POLICY us_sales_filter
ON `enterprise_sales.orders`
GRANT TO ('group:us-analysts@company.com')
FILTER USING (region = 'US');

Key Row-Level Security Characteristics for the Exam

• Transparent Filtering: When a member of emea-analysts@company.com executes SELECT * FROM enterprise_sales.orders, BigQuery automatically and transparently rewrites the query execution plan to inject WHERE region = 'EMEA'. The user cannot bypass this predicate.
• Full-Table Fallback Role: Users without an applicable row access policy receive zero rows by default, unless they possess roles/bigquery.filteredDataViewer or admin permissions that grant access to unrestricted rows.
• Coexistence with Column-Level Security: Row access policies and policy tags operate simultaneously. A query plan filters unauthorized rows first via Row-Level Security, then masks or redacts unauthorized columns via Dynamic Data Masking.

---

Sensitive Data Discovery and Protection (Cloud DLP / SDP)

Before enforcing policy tags, data engineers must locate where sensitive PII resides across terabytes of untagged legacy data. Sensitive Data Protection (formerly Cloud DLP) provides automated inspection and de-identification capabilities:

• Automated Profiling: Dataplex integrates directly with Sensitive Data Protection to profile BigQuery tables and Cloud Storage buckets automatically, detecting over 150 built-in infoTypes (such as US_SOCIAL_SECURITY_NUMBER, CREDIT_CARD_NUMBER, PASSPORT, EMAIL_ADDRESS).
• De-Identification Techniques:
  • Masking & Redaction: Obscures characters partially or completely.
  • Bucketing / Generalization: Converts continuous values into coarse categories (e.g., ages to 10-year ranges) to prevent re-identification.
  • Cryptographic Hashing with Salt: Deterministic tokenization enabling entity correlation without storing raw identifiers.
  • Format-Preserving Encryption (FPE): Encrypts data while preserving original length and character set (e.g., a 16-digit credit card number encrypts into a ciphertext 16-digit number), allowing legacy schemas to process tokens without schema breakage.

---

Dataplex Hierarchy and Security Governance Reference

---

Automated Data Quality (Auto DQ) and Lineage Tracking

Enterprise governance requires proving that data is reliable, timely, and untampered with.

Dataplex Automated Data Quality (Auto DQ)

Dataplex Auto DQ provides a fully managed, serverless framework for defining and evaluating data quality rules directly against BigQuery tables and Cloud Storage assets without deploying Dataproc or Dataflow pipelines.

• Declarative Quality Rules: Data engineers define data quality specifications in declarative YAML or SQL syntax, including:
  • completeness_check: Validates that null percentages do not exceed a strict threshold.
  • uniqueness_check: Verifies primary key uniqueness.
  • range_check: Ensures values fall between acceptable minimum and maximum bounds (e.g., $0 \le \text{transaction_amount} \le 100,000$).
  • regex_match: Validates formatting standards (such as phone numbers, zip codes, or ISO timestamps).
  • custom_sql_expression: Executes arbitrary SQL assertions for complex business logic.
• Automated Scheduling & Alerts: Rules execute on automated schedules (hourly, daily, or post-ingestion). Results publish aggregated Data Quality Scores directly to Dataplex Catalog, triggering Cloud Monitoring alerts or Pub/Sub notifications if quality thresholds fail.

Automated End-to-End Data Lineage Tracking

Understanding how data flows through enterprise transformations is critical for root-cause error analysis, impact analysis prior to schema migrations, and regulatory compliance (such as BCBS 239 or GDPR).

Dataplex automatically tracks and visualizes Data Lineage without requiring manual instrumenting code:

• Native Integration: Tracks transformations executed by BigQuery SQL queries, Cloud Dataflow pipelines (Apache Beam), and Cloud Dataproc Spark jobs.
• Visual DAG Graph: Renders an interactive directed acyclic graph in the Google Cloud console showing the source tables/buckets, intermediate transformation processes, destination tables, and downstream Looker dashboards.
• Impact Analysis: When a table schema is modified or a pipeline fails, data engineers inspect downstream lineage nodes instantly to identify every report, model, or dashboard impacted by the change.