2.3 Data Cataloging, Governance, and Querying

Data Cataloging, Governance, and Querying

Before data can be fed into feature engineering pipelines or SageMaker model training jobs, it must be discovered, cataloged, governed, and explored. In enterprise environments, data lakes contain sensitive information (Personally Identifiable Information [PII], financial records) distributed across hundreds of tables. ML engineers must implement robust governance frameworks to control data access at granular levels, track schema evolution, and query raw data in-place without copying or moving petabytes of files.

---

1. AWS Glue Data Catalog: Centralized ML Metadata Management

The AWS Glue Data Catalog is a managed, Apache Hive Metastore-compatible metadata repository. It stores structural metadata—such as database names, table definitions, column data types, physical S3 URI locations, and partition key structures—without storing the underlying data records.

+-----------------------------------------------------------------------------+
|                     AWS GLUE DATA CATALOG INTEGRATION                       |
|                                                                             |
|   [Underlying Storage]                                                      |
|   └── Amazon S3 Data Lake (Parquet / ORC / JSON / CSV)                      |
|             ^                                                               |
|             | Metadata Reference (Schemas, Partitions, SerDes)              |
|             v                                                               |
|   [AWS GLUE DATA CATALOG] <--- (Maintained by Crawlers / ETL / APIs)        |
|             |                                                               |
|             +---> Amazon Athena (Ad-hoc SQL Exploration & CTAS)             |
|             +---> Amazon SageMaker (Data Wrangler / Feature Store / Studio) |
|             +---> Amazon EMR & AWS Glue ETL (Distributed Spark Jobs)         |
|             +---> Amazon Redshift Spectrum (Data Warehouse Queries)         |
+-----------------------------------------------------------------------------+

Key Concepts of the Glue Data Catalog:

• Databases & Tables: A database is a logical grouping of table definitions. A table contains schema definitions (column names, data types such as int, bigint, string, struct, array), the S3 physical path, and the Serializer/Deserializer (SerDe) used to parse the file format.
• Partitions: Each partition maps a specific partition key value (e.g., year=2026/month=08/) to an S3 folder location. When query engines query a partitioned Glue table, they use partition metadata to skip non-matching S3 prefixes.
• Schema Versioning: The Glue Data Catalog tracks schema evolution across time. When new columns are added or data types change, a new schema version is created, allowing historical rollback and backward compatibility validation.

---

2. AWS Glue Crawlers and Schema Drift Management

AWS Glue Crawlers connect to source data stores (Amazon S3, Amazon RDS, DynamoDB), determine the data format using built-in or custom classifiers, infer schemas, and automatically register or update table definitions in the Glue Data Catalog.

+-----------------------------------------------------------------------------+
|                      GLUE CRAWLER WORKFLOW & LOGIC                          |
|                                                                             |
|   [Amazon S3 Buckets] ---> [AWS Glue Crawler]                               |
|                                   |                                         |
|                                   +---> 1. Run Classifiers (Grok/JSON/Parquet|
|                                   +---> 2. Detect Partition Keys (S3 Paths) |
|                                   +---> 3. Compare with Catalog Schema      |
|                                   |                                         |
|                                   v                                         |
|   [Schema Drift Action] -------------------------------------------------+  |
|   ├── "Update the table definition in the data catalog" (Default)        |  |
|   ├── "Add new columns only"                                             |  |
|   └── "Ignore the change and don't update the table"                     |  |
+-----------------------------------------------------------------------------+

Classifier Hierarchy:

1. Built-in Classifiers: Glue includes native classifiers for common formats: JSON, CSV, TSV, Parquet, ORC, Avro, XML, and web server log formats.
2. Custom Classifiers: When data arrives in proprietary delimited formats or custom log structures, you define a custom Grok pattern, JSONPath, or XML path classifier. Custom classifiers are evaluated first in priority order before built-in classifiers.

Managing Schema Drift & Deleted Objects:

When upstream producer systems change data schemas (e.g., adding a device_battery_level column or removing deprecated fields), the Glue Crawler behavior is governed by configuration settings:

• When schema changes are detected:
  • Update the table definition in the data catalog: Automatically adds new columns or changes data types.
  • Add new columns only: Protects existing column definitions from accidental modification while adopting new features.
  • Ignore the change: Leaves the table definition intact.
• How crawler should handle deleted objects in the data store:
  • Mark the table as deprecated in the data catalog:
  • Delete tables and partitions from the data catalog:
  • Ignore the change (Keep existing metadata):

[!TIP] Event-Driven Crawling: Rather than running expensive scheduled crawlers that scan entire S3 buckets on a cron schedule, configure Amazon S3 Event Notifications with Amazon EventBridge to invoke an AWS Lambda function that triggers the Glue Crawler only when new partitions or files land in S3 (s3:ObjectCreated:*).

---

3. Fine-Grained Data Governance with AWS Lake Formation

Securing machine learning datasets using standard IAM policies and S3 bucket policies becomes unmanageable as organizations scale to hundreds of datasets and thousands of columns. AWS Lake Formation sits on top of the AWS Glue Data Catalog and provides centralized, fine-grained access control.

+-----------------------------------------------------------------------------+
|             AWS LAKE FORMATION FINE-GRAINED ACCESS CONTROL                  |
|                                                                             |
|   [Raw S3 Table: `customer_analytics` (Contains PII, Financials, Targets)]  |
|                                  |                                          |
|                                  v                                          |
|   [AWS LAKE FORMATION PERMISSION ENGINE]                                    |
|   ├── Tag-Based Access Control (LF-TBAC): `Confidentiality=Restricted`      |
|   ├── Column-Level Security: Mask `ssn` and `credit_card` columns           |
|   └── Row-Level Security: Filter `WHERE country = 'US' AND active = true`   |
|                                  |                                          |
|         +------------------------+------------------------+                 |
|         |                                                 |                 |
|         v                                                 v                 |
|   [Role A: ML Feature Engineer]                     [Role B: Data Auditor]  |
|   - Sees all sanitized features                     - Sees metadata only    |
|   - PII columns masked with hashes                  - No raw data access    |
|   - Filtered to authorized geographic rows          - Full audit log views  |
+-----------------------------------------------------------------------------+

Lake Formation Security Capabilities:

1. Lake Formation Tag-Based Access Control (LF-TBAC):

   • Defines access permissions based on metadata tags assigned to databases, tables, or columns (e.g., Department=Marketing, DataClassification=Public, PII=True).
   • When new tables or columns are added with matching tags, IAM roles automatically inherit appropriate access permissions without manually updating individual IAM or S3 bucket policies.

2. Column-Level Access Control & Data Masking:

   • Restricts specific IAM roles or ML user identities from accessing sensitive feature columns (e.g., social security numbers, bank account numbers, patient medical identifiers).
   • Supports Dynamic Data Masking:
     • Redact (Nullify): Replaces sensitive column values with nulls.
     • Custom Masking / Hashing: Masks strings with asterisks or cryptographic hashes.

3. Row-Level Filtering:

   • Restricts data access based on SQL WHERE filter clauses applied at query time.
   • Example: An ML training role for a European model can be granted access with a row filter WHERE region = 'EU', ensuring compliance with data residency regulations without physically duplicating data into separate S3 regional buckets.

4. Cell-Level Security:

   • Combines row-level filtering and column-level restrictions simultaneously to grant access to an exact rectangular subsection of a table.

5. Cross-Account Data Sharing:

   • Shares Data Catalog tables and underlying S3 data directly with external AWS accounts using AWS Resource Access Manager (RAM) and Lake Formation cross-account permissions without copying data.

---

4. Ad-Hoc Data Exploration & Preparation with Amazon Athena

Amazon Athena is an interactive, serverless query service that allows ML engineers to analyze data directly in Amazon S3 using standard SQL (Presto/Trino engines). It requires zero infrastructure setup and charges only for the data scanned by queries.

+-----------------------------------------------------------------------------+
|                   AMAZON ATHENA IN ML DATA PREPARATION                      |
|                                                                             |
|   1. Exploratory Data Analysis (EDA):                                       |
|   [Athena SQL Query] ---> Scans S3 In-Place ---> Returns Distribution Stats  |
|                                                                             |
|   2. Partition Projection (High-Cardinality Optimization):                  |
|   [Athena Query with Projection] ---> Skips Glue Catalog Partition Lookups  |
|                                  ---> Computes S3 paths dynamically in SQL  |
|                                                                             |
|   3. Feature Extraction & Dataset Splitting via CTAS:                       |
|   [Athena CTAS Query] ---> Reads Raw JSON/CSV                               |
|                       ---> Transforms & Filters                             |
|                       ---> Writes Snappy Parquet to S3 Training Folder      |
+-----------------------------------------------------------------------------+

Partition Projection for High-Cardinality Datasets

• The Challenge: Datasets with millions of partitions (e.g., IoT telemetry partitioned by device_id, year, month, day, hour) cause severe performance degradation. Every query forces Athena to make thousands of GetPartitions API calls to the Glue Data Catalog, resulting in query latency and AWS Glue rate-limiting errors.
• The Solution (Partition Projection):
  • Partition values and S3 directory locations are calculated directly from predefined table configuration properties (e.g., date ranges, numeric ranges, enums).
  • Athena computes the exact S3 prefix paths in memory without making metadata API calls to the Glue Data Catalog.
  • Dramatically accelerates query execution times and eliminates partition registration overhead.

-- Example: Athena Table Definition with Partition Projection
CREATE EXTERNAL TABLE telemetry_projected (
    sensor_id STRING,
    temperature DOUBLE,
    vibration DOUBLE,
    pressure DOUBLE
)
PARTITIONED BY (
    device_type STRING,
    log_date STRING
)
STORED AS PARQUET
LOCATION 's3://iot-sensor-lake/telemetry/'
TBLPROPERTIES (
    'projection.enabled' = 'true',
    'projection.device_type.type' = 'enum',
    'projection.device_type.values' = 'turbine,pump,compressor',
    'projection.log_date.type' = 'date',
    'projection.log_date.range' = '2026/01/01,NOW',
    'projection.log_date.format' = 'yyyy/MM/dd',
    'storage.location.template' = 's3://iot-sensor-lake/telemetry/device_type=${device_type}/date=${log_date}'
);

CTAS (CREATE TABLE AS SELECT) for ML Training Preparation

ML engineers use Athena CTAS queries to extract, clean, encode, and export raw training datasets into partitioned, compressed Apache Parquet formats ready for SageMaker training consumption:

-- Create optimized training split in S3 directly from Athena
CREATE TABLE sagemaker_churn_train
WITH (
    format = 'PARQUET',
    parquet_compression = 'SNAPPY',
    external_location = 's3://ml-feature-store-prod/training_splits/train/',
    partitioned_by = ARRAY['churn_label']
) AS
SELECT 
    account_id,
    tenure_months,
    total_spend,
    support_tickets,
    avg_session_length,
    CASE WHEN churned = 1 THEN 'CHURN' ELSE 'RETAIN' END AS churn_label
FROM raw_customer_events
WHERE event_date >= '2026-01-01'
  AND total_spend IS NOT NULL;

Athena Workgroups for Cost Governance

• Athena Workgroups allow ML teams to isolate query execution environments, configure separate Amazon CloudWatch query metrics, mandate query result encryption, and set Per-Query and Per-Workgroup Data Scan Limits to prevent runaway costs from unpartitioned table scans.

[!IMPORTANT] Core Governance Exam Takeaway: When an enterprise ML scenario requires granting data scientists access to specific columns and authorized customer records while masking PII (like SSNs) without duplicating or redacting underlying S3 files, the correct AWS solution is AWS Lake Formation with column-level masking and row-level filtering.