3.2 Glue Crawlers, Data Catalog & Schema Evolution

3.2 Glue Crawlers, Data Catalog & Schema Evolution

In modern data lakes built on Amazon S3, metadata management is the bridge connecting raw file storage to analytical query engines. The AWS Glue Data Catalog provides a centralized, Apache Hive-metastore-compatible repository that stores table definitions, column types, physical data locations, and partition key structures.

Query engines like Amazon Athena, Amazon Redshift Spectrum, Amazon EMR, and AWS Lake Formation rely on the Glue Data Catalog as their single source of truth for table schemas and security policies.

---

AWS Glue Crawlers Architecture & Classifier Precedence

AWS Glue Crawlers are automated processes that connect to source data stores, evaluate data samples using classifiers, infer format and schema attributes, and populate or update table definitions in the Glue Data Catalog.

Classifier Evaluation Order

When a crawler scans a data store, it tests the data against classifiers in a strict, deterministic sequence:

1. Custom Classifiers (User-Defined): Evaluated first in the order specified in the crawler configuration. Custom classifiers include:
   • Grok Classifiers: Use Grok patterns to parse unstructured or semi-structured log files (e.g., Apache access logs, syslog).
   • JSON Path Classifiers: Define explicit JSONPath expressions to parse complex JSON documents.
   • XML Classifiers: Define row tags to parse XML schemas.
   • Custom CSV Classifiers: Specify custom column headers, delimiters, and quote characters.
2. Built-In Classifiers: Evaluated only if no custom classifier matches. Built-in classifiers automatically recognize standard formats including Apache Parquet, Apache Avro, ORC, JSON, BSON, XML, CSV, and compressed variants (Gzip, Snappy, Bzip2).

Important Exam Concept: If a custom classifier and a built-in classifier both match a dataset, the custom classifier takes precedence. If you create a custom Grok classifier for JSON files, Glue will use your Grok classifier instead of the built-in JSON classifier.

---

Configuring Crawlers for Schema Evolution & Object Deletion

As upstream data sources evolve, schemas inevitably change—columns are added, renamed, or dropped, and historical data partitions accumulate. AWS Glue Crawlers provide fine-grained controls to manage schema drift and data lifecycle events.

Schema Change Behavior Matrix

When a crawler detects a mismatch between the physical storage schema and the Data Catalog table definition, its behavior is governed by the Schema Change Policy:

Object Deletion & Deprecation Policy

When files or partitions are deleted from S3, crawler behavior depends on the Object Deletion Policy:

• Mark the table as deprecated in the data catalog: Adds a property DEPRECATED to the catalog table. Recommended when historical metadata must be preserved for auditing.
• Delete tables and partitions from the data catalog: Removes metadata entries for missing files/partitions immediately. Essential for maintaining clean catalogs when data retention policies delete old S3 prefixes.
• Ignore the change: Leaves catalog definitions untouched even if underlying S3 objects no longer exist.

---

Catalog Partition Syncing Strategies: Crawlers vs ETL Direct Updates

In S3 data lakes, data is structured into Hive-style partition paths (e.g., s3://bucket/table/year=2026/month=08/day=13/). For query engines like Athena to leverage partition pruning, partition metadata must be registered in the Data Catalog.

Data engineers have three primary mechanisms to synchronize partition metadata:

Comparison of Partition Syncing Methods

Direct Catalog Update Code Pattern

Instead of running a separate Glue Crawler after an ETL job completes, configure the Glue ETL job to update the catalog schema and partitions dynamically during execution:

# Direct Catalog Update within Glue Job Write Option
glueContext.write_dynamic_frame.from_catalog(
    frame=transformed_dyf,
    database="analytics_db",
    table_name="processed_events",
    additional_options={
        "enableUpdateCatalog": True,
        "updateBehavior": "UPDATE_IN_DATABASE",
        "partitionKeys": ["year", "month", "day"]
    },
    transformation_ctx="write_catalog_dyf"
)

---

Handling Schema Evolution in Amazon Athena Queries

When schema changes occur in S3 data lakes, downstream query engines like Amazon Athena handle evolution based on the underlying file format:

1. Parquet & ORC (Schema by Name vs. Schema by Read Index):
   • Athena reads Parquet columns by name by default (parquet.column.index.access=false). Name mapping tolerates column reordering and adding or removing columns more safely than ordinal mapping. Set the property to true only when the files and table intentionally rely on column position.
   • Athena reads ORC columns by index by default (orc.column.index.access=true); set it to false when the dataset should map ORC columns by name.
2. JSON & CSV:
   • Schema evolution is handled by appending new columns to the end of the table schema in the Data Catalog. Missing fields in historical files automatically evaluate to NULL during Athena queries.

---

Boto3 Glue Crawler Automation Script

Data engineers automate crawler execution and status monitoring using the AWS SDK for Python (boto3):

import boto3
import time

glue_client = boto3.client('glue', region_name='us-east-1')

def trigger_and_wait_for_crawler(crawler_name):
    # Start Crawler
    response = glue_client.start_crawler(Name=crawler_name)
    print(f"Started crawler: {crawler_name}")
    
    # Poll Crawler Status
    while True:
        crawler = glue_client.get_crawler(Name=crawler_name)['Crawler']
        state = crawler['State']
        print(f"Crawler state: {state}")
        
        if state == 'READY':
            last_crawl = crawler.get('LastCrawl', {})
            status = last_crawl.get('Status')
            print(f"Crawl completed with status: {status}")
            break
        elif state == 'RUNNING' or state == 'STOPPING':
            time.sleep(15)
        else:
            raise Exception(f"Unexpected crawler state: {state}")

# Example invocation
# trigger_and_wait_for_crawler("s3-raw-sales-crawler")