13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts

13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts

Distributed data processing across elastic cloud infrastructure introduces complex failure modes. When an Azure Databricks pipeline fails or degrades, data engineers must rapidly identify whether the root cause stems from architectural antipatterns (such as collecting data to the driver), memory exhaustion, execution disk spill, data skew, or cloud infrastructure disruptions (such as Azure Spot VM evictions).

Understanding JVM memory architecture, shuffle mechanics, and error signatures is essential for resolving production incidents.

---

1. Anatomy of OutOfMemory (OOM) Errors: Driver vs. Executor

Apache Spark separates memory management into two distinct JVM processes: the Driver (master orchestrator) and Executors (worker compute engines). An OutOfMemoryError (OOM) manifest in either process has distinct root causes and remediations.

+-------------------------------------------------------------------------+
|                    DRIVER OOM VS. EXECUTOR OOM                          |
+-------------------------------------------------------------------------+
|                                                                         |
|  DRIVER OUTOFMEMORYERROR                 EXECUTOR OUTOFMEMORYERROR      |
|  * Error: java.lang.OutOfMemoryError     * Error: ExecutorLostFailure   |
|    Java heap space                       * Container killed by YARN/K8s |
|  * Location: Driver JVM                  * Location: Worker Node JVM    |
|                                                                         |
|  COMMON CAUSES:                          COMMON CAUSES:                 |
|  1. Calling df.collect() or .toPandas()  1. Giant skewed partitions     |
|  2. Broadcasting table > driver RAM      2. Low partition count (few    |
|  3. Excessive unpartitioned file            gigantic tasks per core)    |
|     metadata (millions of files)         3. Memory-heavy Python UDFs    |
|  4. Heavy SparkContext logging           4. Insufficient executor heap  |
|                                                                         |
|  REMEDIATION:                            REMEDIATION:                   |
|  - Write directly to Delta tables        - Increase shuffle partitions  |
|  - Use .take(n) or .limit(n)             - Upsize worker VM memory      |
|  - Upsize Driver VM instance             - Salt skewed join keys        |
+-------------------------------------------------------------------------+

1. Driver OutOfMemoryError Diagnosis & Fixes

When a driver runs out of memory, the entire Spark session crashes, aborting all active jobs.

• Antipattern: .collect() or .toPandas() on Large Datasets:

  # DANGEROUS: Pulls 50 million rows across network into driver JVM heap!
  all_data = df.collect()
  pandas_df = df.toPandas()
  
  # RECOMMENDED: Filter, aggregate, or sample on workers before collecting
  sample_data = df.limit(100).toPandas()
  # Or write directly to cloud storage without touching the driver:
  df.write.format("delta").mode("append").saveAsTable("silver.cleansed_events")
  

• Antipattern: Oversized Broadcast Joins: Broadcasting a DataFrame via broadcast(df) forces the driver to collect the entire dataset into driver RAM before serializing and broadcasting it to executors. If the broadcast table exceeds spark.driver.memory (or spark.sql.autoBroadcastJoinThreshold), the driver crashes with OOM.

2. Executor OutOfMemoryError Diagnosis & Fixes

When an executor exhausts its heap or off-heap memory, the cluster manager terminates the container, logging ExecutorLostFailure (exit code 137 / OOMKilled).

• Root Cause: Partition Sizing & High Concurrency: If a cluster has 8 cores per worker and only 8 partitions for a 100 GB dataset, each task attempts to process ~12.5 GB of data simultaneously within the worker's shared JVM heap.
• Remediation:
  • Increase partition count so that individual tasks process 100 MB–200 MB chunks.
  • Switch worker node types from compute-optimized VMs (Standard_F series) to memory-optimized VMs (Standard_E series, e.g., Standard_E8ds_v5).

---

2. Memory Spill vs. Disk Spill: The Hidden Performance Killer

Spark allocates executor memory into two primary pools: Execution Memory (used for shuffle, joins, sorts, and aggregations) and Storage Memory (used for cached data and broadcast variables).

When execution memory is exhausted during a memory-intensive operation (such as sorting or hashing), Spark evicts data from RAM to disk. This process is called Spill.

+-------------------------------------------------------------------------+
|                     SPARK MEMORY SPILL LIFECYCLE                        |
+-------------------------------------------------------------------------+
|                                                                         |
|  EXECUTOR JVM HEAP                                                      |
|  ┌────────────────────────────────────────────────────────┐             |
|  │ Spark Memory Pool (Execution + Storage)                │             |
|  │ [ In-Memory Data Structures (Hash Tables, Sort Buffers) ]            |
|  └───────────────────────────┬────────────────────────────┘             |
|                              │ Execution Memory Full!                   |
|                              ▼                                          |
|  SPILL (MEMORY): Deserialized Java objects in RAM (~24 GiB)             |
|                              │                                          |
|                              │ Compress & Serialize to Disk             |
|                              ▼                                          |
|  SPILL (DISK): Compressed binary bytes on local NVMe SSD (~6 GiB)       |
|                                                                         |
+-------------------------------------------------------------------------+

Spill (Memory) vs. Spill (Disk) Explained

Exam Tip: Spill (Memory) is almost always significantly larger than Spill (Disk) (often 3x to 5x larger) because in-memory JVM objects consume far more space than compressed, serialized on-disk bytes.

Eliminating Spill in Production Pipelines

1. Increase Shuffle Partition Count: Ensure partitions are small enough (< 200 MB) to fit entirely within execution memory.
2. Enable AQE Partition Coalescing & Skew Join: Dynamically handles partition sizing at runtime.
3. Upsize Worker RAM: Utilize memory-optimized Azure VM series (Standard_E family).
4. Prune Unused Columns: Drop unnecessary columns before joins and aggregations to reduce in-memory row widths.

---

3. Data Skew Diagnosis & Remediation via Salting

Data skew occurs when data is distributed unevenly across partitions due to high cardinality imbalances in join or grouping keys (for example, millions of web click events with user_id = NULL or store_id = 9999).

Remediation 1: Manual Key Salting

When Adaptive Query Execution (AQE) is disabled or unable to resolve extreme skew, data engineers implement Key Salting. Salting adds a pseudo-random integer (0 to N-1) to the skewed key of the large DataFrame and replicates the matching key on the smaller DataFrame across all N values using an explode operation.

# Manual Key Salting Implementation in PySpark
from pyspark.sql import functions as F

SALT_FACTOR = 4  # Split skewed keys into 4 distinct sub-buckets

# Step 1: Add a random salt (0..3) to the large skewed DataFrame
large_skewed_df = large_df.withColumn("salt_key", F.concat(F.col("join_key"), F.lit("_"), (F.rand() * SALT_FACTOR).cast("int")))

# Step 2: Replicate the lookup DataFrame by exploding an array of [0, 1, 2, 3]
small_replicated_df = small_df.withColumn("salt_array", F.array([F.lit(i) for i in range(SALT_FACTOR)])) \
                              .withColumn("salt_val", F.explode("salt_array")) \
                              .withColumn("salt_key", F.concat(F.col("join_key"), F.lit("_"), F.col("salt_val")))

# Step 3: Execute the join on the salted key (Distributes identical join keys across 4 tasks!)
balanced_joined_df = large_skewed_df.join(small_replicated_df, on="salt_key", how="inner") \
                                    .drop("salt_key", "salt_array", "salt_val")

+-------------------------------------------------------------------------+
|                        KEY SALTING ARCHITECTURE                         |
+-------------------------------------------------------------------------+
|                                                                         |
|  WITHOUT SALTING (100% of Key 'A' goes to Partition 0):                 |
|  Large DF: [A, A, A, A, A, A, A, A] ===> [ Task 0: 8 rows (STRAGGLER) ] |
|  Small DF: [A]                      ===> [ Task 0: 1 row ]              |
|                                                                         |
|  WITH SALTING (Key 'A' salted with random 0..3):                        |
|  Large DF: [A_0, A_1, A_2, A_3, A_0, A_1, A_2, A_3]                     |
|  Small DF: Exploded to [A_0, A_1, A_2, A_3]                             |
|                                                                         |
|  Task 0: [A_0, A_0] joins [A_0]  (2 rows) -> Perfectly Balanced!        |
|  Task 1: [A_1, A_1] joins [A_1]  (2 rows) -> Perfectly Balanced!        |
|  Task 2: [A_2, A_2] joins [A_2]  (2 rows) -> Perfectly Balanced!        |
|  Task 3: [A_3, A_3] joins [A_3]  (2 rows) -> Perfectly Balanced!        |
+-------------------------------------------------------------------------+

---

4. Azure Spot VM Evictions & Network Fetch Failures

Azure Databricks allows worker nodes to run on Azure Spot Virtual Machines to reduce compute costs by up to 80%. However, Azure can evict Spot VMs at any time when compute capacity is reclaimed.

Spot Eviction Failure Cascades & FetchFailedException

1. Executor Loss: When Azure evicts a Spot VM, the executor running on that VM terminates abruptly.
2. Shuffle Fetch Failure: If the evicted executor was hosting intermediate shuffle files on its local disk, downstream tasks running on other workers fail when attempting to read those shuffle files over the network (org.apache.spark.shuffle.FetchFailedException).
3. Automatic Stage Resubmission: Spark handles FetchFailedException natively: the DAG scheduler marks the lost shuffle data as missing and automatically resubmits the upstream stage to recompute the missing partitions.

+-------------------------------------------------------------------------+
|                SPOT EVICTION & STAGE RETRY WORKFLOW                     |
+-------------------------------------------------------------------------+
|                                                                         |
|  1. Worker VM 2 (Spot) Evicted by Azure!                                |
|  2. Downstream Task on Worker 1 attempts to read Shuffle Block from VM 2|
|  3. Network Fetch Fails -> Spark throws FetchFailedException            |
|  4. DAG Scheduler marks Stage 0 Shuffle Output as missing               |
|  5. Spark resubmits missing tasks of Stage 0 on healthy Worker 1 & 3    |
|  6. Downstream Stage 1 resumes successfully without job failure         |
+-------------------------------------------------------------------------+

Production Best Practices for Spot Compute:

• Never Run Driver Nodes on Spot VMs: If the driver VM is evicted, the entire Spark application terminates immediately and cannot recover. Azure Databricks enforces this by placing the Driver on an On-Demand VM by default.
• Use Spot Fallback: Enable Databricks Spot Fallback so that if Azure Spot capacity is exhausted, worker nodes automatically provision as On-Demand VMs to prevent job failure.
• Baseline On-Demand + Burst Spot: Configure clusters with a minimum count of On-Demand workers to guarantee baseline progress, using Spot VMs only for autoscaling burst capacity.