9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures

Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures

Data engineers designing automated analytics pipelines on AWS must rapidly diagnose and resolve runtime execution failures across serverless and managed compute engines. This section provides an authoritative technical breakdown of failure modes, diagnostic logging workflows, and memory tuning strategies for AWS Glue, Amazon EMR, and AWS Lambda.

---

AWS Glue Job Execution & Memory Troubleshooting

AWS Glue PySpark jobs execute on managed Apache Spark clusters. Failures in Glue typically stem from memory pressure on either the Spark driver or Spark executor nodes, or misconfigured state management via Glue Job Bookmarks.

1. Driver OutOfMemoryError (OOM)

The Spark driver node orchestrates execution, builds query plans, and collects metadata. A driver crash (java.lang.OutOfMemoryError: Java heap space) occurs when data or metadata returned to the driver exceeds its JVM heap size.

• Root Causes:
  • Calling .collect() or .toPandas() on large Spark DataFrames, pulling gigabytes of distributed data onto the single driver node.
  • Listing millions of S3 objects during schema inference or partition discovery without pushdown filters.
  • Using un-partitioned Glue DynamicFrames with massive single-file outputs.
• Remediation Strategies:
  • Eliminate Driver Collection: Replace .collect() with distributed S3 writes (DataFrame.write.parquet()) or use .take(n) for debugging.
  • Apply Pushdown Predicates: Filter S3 partitions at the Glue Data Catalog level before reading objects into memory:

    datasource = glueContext.create_dynamic_frame.from_catalog(
        database="analytics_db",
        table_name="raw_logs",
        push_down_predicate="year == '2026' and month == '08'"
    )
    

  • Scale Worker Types: Upgrade Glue Worker Types to allocate larger driver heaps:
    • G.1X: 1 DPU (4 vCPUs, 16 GB RAM)
    • G.2X: 2 DPUs (8 vCPUs, 32 GB RAM)
    • G.4X: 4 DPUs (16 vCPUs, 64 GB RAM)
    • G.8X: 8 DPUs (32 vCPUs, 128 GB RAM) Select memory-optimized or larger workers only after metrics identify driver or executor pressure; verify current disk and worker availability in the target Region.

2. Executor OutOfMemoryError (OOM)

Executor nodes process distributed partition data chunks. Executor OOM errors occur when individual partitions exceed worker memory or when operations cause severe data skew.

• Remediation:
  • Repartitioning Data: Use .repartition(N) or .coalesce(N) to equalize partition sizes across executors and eliminate skewed partitions.
  • Group Size & Factor Tuning: Adjust Glue DynamicFrame S3 reading parameters: groupFiles="inGroup" and groupSize="134217728" (128 MB) to bundle small files into uniform partitions.

3. Glue Job Bookmarks State Management

Glue Job Bookmarks track processed S3 objects by maintaining state across job runs.

• Common Failure Scenarios:
  • Changed Source Context: Reusing a transformation_ctx after changing its S3 source path can apply old bookmark state and skip files. S3 bookmarks otherwise use object last-modified time, so a new or modified object can be selected.
  • Duplicate Processing: Calling job.init() without job.commit() prevents successful bookmark advancement and can lead to duplicate processing on later runs.
• Resolution: Configure job parameters (--job-bookmark-option set to job-bookmark-enable, job-bookmark-disable, or job-bookmark-pause). When re-processing updated historical partitions, pause or reset the bookmark state via the AWS Management Console or AWS CLI (aws glue reset-job-bookmark --job-name <name>).

---

AWS EMR Cluster & YARN Execution Troubleshooting

Amazon EMR clusters run Apache Spark, Hive, and Presto managed by YARN (Yet Another Resource Negotiator). Troubleshooting EMR requires inspecting YARN resource allocations and container lifecycle limits.

1. YARN Container Memory Exceeded Failures

A frequent EMR failure mode is YARN forcibly terminating an executor container: Container killed by YARN for exceeding memory limits. 10.5 GB of 10.0 GB physical memory used.

• Root Cause: Spark executors require physical memory beyond the JVM heap for off-heap execution, PySpark Python process communications, and memory-mapped file operations.
• Remediation via Memory Overhead Tuning: Increase spark.executor.memoryOverhead (default is max(384MB, 0.10 * spark.executor.memory)). When using heavy PySpark UDFs, NumPy, or Apache Arrow, set memory overhead explicitly:

  spark-submit \
    --conf spark.executor.memory=8g \
    --conf spark.executor.memoryOverhead=2048m \
    --conf spark.driver.memory=8g \
    --conf spark.driver.memoryOverhead=2048m \
    etl_script.py
  

2. Spot Instance Terminations & Resiliency

EMR core and task nodes running on EC2 Spot Instances can be reclaimed by AWS with a 2-minute notification, resulting in process termination (Exit Code 137 / 143).

• Resiliency Architecture:
  • Use EMR Instance Fleets configured with the capacity-optimized allocation strategy across multiple EC2 instance families (e.g., r5.2xlarge, r5a.2xlarge, r6g.2xlarge).
  • Use the EMR release-specific YARN graceful-decommission settings for managed scale-down. Newer releases can keep shuffle data available while a node drains, but a Spot interruption can still outlast the available notice, so retain retries and durable checkpoints.

3. Diagnostic Logging & Spark History Server

To diagnose executor losses, memory leaks, and GC pauses:

1. Enable Spark Event Logging in spark-defaults.conf: spark.eventLog.enabled true spark.eventLog.dir s3://my-emr-logs-bucket/spark-events/
2. Launch a persistent Spark History Server on an EC2 instance or viewing container to inspect stage execution timelines, DAG visualizations, and garbage collection (G1GC) metric graphs.

---

AWS Lambda ETL Bottlenecks & Failure Remediation

AWS Lambda is widely used for event-driven micro-ETL (e.g., processing S3 object creation triggers). However, strict service quotas require deliberate architectural planning.

Key Service Quotas & Mitigations

• Execution Timeout (15 minutes): If an S3 file transformation exceeds 900 seconds, Lambda terminates the execution.
  • Fix: Read bounded chunks or byte ranges when processing can be safely partitioned, or offload long-running distributed work to Glue, EMR, or containers. S3 Select is closed to new customers.
• Memory Quota (10,240 MB) & /tmp Ephemeral Storage: Default /tmp space is 512 MB, configurable up to 10,240 MB (10 GB).
  • Fix: If intermediate files exceed memory, configure Lambda /tmp storage to 10 GB or mount an Amazon EFS file system for shared persistence.
• Concurrency Limits & Throttling (429 TooManyRequestsException): Unreserved concurrency spikes can exhaust the account baseline limit (1,000 per region).
  • Fix: Assign Reserved Concurrency to critical ETL Lambdas to guarantee capacity, or configure Provisioned Concurrency to eliminate cold-start latency.

---

Execution Failure Troubleshooting Matrix