3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing

3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing

Processing massive, continually growing enterprise datasets requires efficient incremental data processing patterns. Re-reading entire S3 data lakes or relational database tables during every batch execution wastes compute resources and increases pipeline latency.

AWS Glue provides two core mechanisms to achieve high-efficiency incremental processing: Glue Job Bookmarks for automated state tracking across job runs, and Pushdown Predicates for metadata-level partition pruning.

---

AWS Glue Job Bookmarks: State Management & Incremental Ingestion

AWS Glue Job Bookmarks track state information across consecutive runs of an ETL job. By persisting bookmark state, Glue ensures that subsequent job runs process only data ingested since the previous execution.

How Job Bookmarks Handle S3 vs. JDBC Data Sources

• Amazon S3 Sources: Glue tracks processed S3 object keys, file sizes, and last modified timestamps. When a job runs, Glue compares S3 bucket contents against the stored bookmark state, passing only newly added or modified files to the job.

  Crucial Limitation: S3 Bookmarks rely on append-only patterns or new file additions. If an existing S3 object's content is modified in-place without changing its object key or file modification metadata in a recognizable manner, the bookmark may not re-process the file.

• JDBC Data Sources: For relational databases, Glue tracks a sequential column (such as an auto-incrementing primary key ID or a last_updated timestamp). Data engineers specify the bookmark key in connection options (jobBookmarkKeys=["updated_at"], jobBookmarkKeysSortOrder="asc").

Job Bookmark Configuration Options (--job-bookmark-option)

Resetting & Rewinding Bookmarks for Data Backfills

If a pipeline encounters a downstream corruption event and historical data must be reprocessed, data engineers can rewind or reset the job bookmark state using the AWS CLI:

# Reset Glue Job Bookmark to reprocess all historical data from inception
aws glue reset-job-bookmark --job-name daily-sales-etl-job

Exam Tip: To ensure Job Bookmarks function correctly in custom PySpark scripts, you must pass the transformation_ctx parameter to create_dynamic_frame.from_catalog or create_dynamic_frame.from_options. If transformation_ctx is omitted, Glue cannot attach state markers to the reader source, causing the job to process the entire dataset every run.

---

Partition Processing & Pushdown Predicates

Partitioning S3 data lakes into hierarchical directory prefixes (e.g., s3://bucket/table/year=2026/month=08/day=13/) dramatically improves query efficiency. However, how Spark reads these partitions impacts cost and performance.

Pushdown Predicates vs. In-Memory Spark Filtering

• In-Memory Spark Filtering (df.filter(col("year") == 2026)): Spark lists all S3 objects across all partition directories, reads every file into executor memory, and then discards non-matching rows. This results in millions of expensive S3 ListObjectsV2 and GET API calls and high DPU utilization.
• Catalog Pushdown Predicates (push_down_predicate): Glue evaluates the predicate condition directly against the Glue Data Catalog metadata before Spark reads or lists files from S3 storage. Spark receives only the S3 object keys matching the target partition, bypassing non-matching S3 directories entirely.

Performance & Cost Comparison Matrix

Pushdown Predicate Implementation Example

# Pushdown predicate evaluated at Catalog metadata level
partitioned_dyf = glueContext.create_dynamic_frame.from_catalog(
    database="analytics_db",
    table_name="web_logs",
    push_down_predicate="year == '2026' and month == '08' and day == '13'",
    transformation_ctx="partitioned_dyf"
)

---

Solving the Small File Problem in AWS Glue

High-frequency ingestion pipelines (such as streaming Kinesis Firehose delivery or hourly micro-batches) often produce thousands of small files (e.g., < 5 MB) in S3. In Apache Spark, each small file creates an individual task partition, leading to severe performance bottlenecks:

• Excessive S3 ListObjectsV2 and GET request latency.
• Spark Driver out-of-memory errors caused by managing millions of file metadata objects.
• High CPU overhead spent managing task lifecycle overhead rather than data processing.

Grouping Small Files in DynamicFrames

AWS Glue provides built-in file grouping parameters (groupFiles and groupSize) within DynamicFrame reader options. Grouping automatically coalesces small S3 files into larger, memory-efficient in-memory partitions before Spark task execution:

# Group small S3 files into target 128 MB partitions
grouped_dyf = glueContext.create_dynamic_frame.from_options(
    connection_type="s3",
    connection_options={
        "paths": ["s3://my-lake-bucket/raw_logs/"],
        "groupFiles": "inGroup",
        "groupSize": "134217728"  # 128 MB in bytes (128 * 1024 * 1024)
    },
    format="json",
    transformation_ctx="grouped_dyf"
)

Output File Compaction Strategy

When writing processed data back to S3, use coalesce(N) or repartition(N) on PySpark DataFrames to ensure output Parquet files are sized optimally (typically 128 MB to 512 MB per file):

# Coalesce output partition files to avoid small file proliferation in S3
compacted_df = transformed_df.coalesce(4)
compacted_dyf = DynamicFrame.fromDF(compacted_df, glueContext, "compacted_dyf")

---

End-to-End Incremental Glue PySpark Script

The following complete Glue script combines Job Bookmarks, Pushdown Predicates, Small File Grouping, and Partitioned S3 Writing:

import sys
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.dynamicframe import DynamicFrame
from awsglue.job import Job

# 1. Parse Job Arguments
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

# 2. Ingest Incremental Data with Pushdown Predicate & File Grouping
incremental_dyf = glueContext.create_dynamic_frame.from_catalog(
    database="telemetry_db",
    table_name="device_events",
    push_down_predicate="year == '2026' and month == '08'",
    additional_options={
        "groupFiles": "inGroup",
        "groupSize": "134217728" # 128 MB grouping
    },
    transformation_ctx="incremental_dyf" # Required for Job Bookmark tracking!
)

# 3. Simple Transformation
transformed_df = incremental_dyf.toDF().dropDuplicates(["event_id"])
output_dyf = DynamicFrame.fromDF(transformed_df.coalesce(10), glueContext, "output_dyf")

# 4. Write Incremental Partitioned Parquet Data
glueContext.write_dynamic_frame.from_options(
    frame=output_dyf,
    connection_type="s3",
    connection_options={
        "path": "s3://company-analytics-lake/curated/device_events/",
        "partitionKeys": ["year", "month"]
    },
    format="parquet",
    transformation_ctx="write_output"
)

# 5. Commit Job Bookmark State
job.commit()