2.2 Memory Management, Fault Tolerance, and Garbage Collection

Efficient distributed data processing requires careful memory accounting. Prior to Spark 1.6, Spark used a static memory manager where execution and storage pools were rigidly separated by fixed boundaries, frequently causing out-of-memory errors when one pool starved while the other sat empty. Modern Apache Spark uses the Unified Memory Manager, which dynamically balances memory across execution and caching workloads alongside Spark's task recovery mechanisms.

---

1. The Unified Memory Architecture: JVM Heap Breakdown

When an executor starts, it receives a total memory allocation governed by spark.executor.memory. This JVM heap space is partitioned into four major regions:

┌────────────────────────────────────────────────────────────────────────┐
│                     Total Executor JVM Heap Space                      │
├──────────────┬───────────────────┬─────────────────────────────────────┤
│ Reserved     │ User Memory       │ Spark Memory (spark.memory.fraction)│
│ Memory       │ (1 - fraction)    ├──────────────────┬──────────────────┤
│ (300 MiB)     │ (~40% of Usable)  │ Storage Memory   │ Execution Memory │
│              │                   │ (~50% pool)      │ (~50% pool)      │
└──────────────┴───────────────────┴──────────────────┴──────────────────┘

Region 1: Reserved Memory (Fixed 300 MiB)

• Size: Exactly 300 MiB, hardcoded in Spark (UnifiedMemoryManager.scala).
• Purpose: An accounting safety margin excluded from unified execution/storage allocation in upstream Spark 3.5.7; it is not an isolated pool that prevents OOM.
• Usable Heap: Subtracted before any other pool is calculated: UsableMemory = ExecutorMemory - 300 MiB.
• Accounting example: For an 8 GiB heap, usable heap for this calculation is 8192 - 300 = 7892 MiB; at fraction 0.6, the unified pool is about 4735 MiB. This is not the entire container allocation: memory overhead, Python and native allocations must also fit.

Region 2: User Memory

• Size: Governed by 1.0 - spark.memory.fraction. With the default fraction of 0.6, User Memory receives 40% of Usable Memory.
• Purpose: Stores non-Spark data structures, user code objects, internal metadata, custom RDD transformations, and user-defined functions (UDFs).
• Characteristics: Spark does not monitor or evict objects in User Memory; if user code allocates massive hash tables or large collections, it can trigger JVM heap exhaustion.

Region 3: Spark Memory (spark.memory.fraction = 0.6)

Spark Memory receives the remaining 60% of Usable Memory. This pool is dedicated to distributed data processing and caching, split dynamically between two sub-pools:

1. Execution Memory:
   • Used for computational buffers required by shuffles, joins, sorts, and hash aggregations.
   • Short-lived working memory allocated during task execution.
   • If an operation requires more Execution Memory than is available, records are sorted and spilled to local executor disk.
2. Storage Memory:
   • Used for cached DataFrames and RDDs (df.cache(), df.persist()).
   • Stores broadcast variables distributed to executors (sparkContext.broadcast(...)).
   • Stores unrolled data blocks during serialization.
   • Governed initially by spark.memory.storageFraction = 0.5 (50% of the Spark Memory pool).

---

2. Dynamic Borrowing and Eviction Policy

The defining feature of the Unified Memory Manager is the dynamic boundary between Execution Memory and Storage Memory. Rather than maintaining hard walls, both pools borrow space from each other under the following rules:

Asymmetric Borrowing Rules

1. Borrowing Without Contention:
   • If no cached data exists in Storage Memory, Execution Memory can expand to occupy 100% of Spark Memory.
   • If no active shuffles or sorts require Execution Memory, Storage Memory can expand to occupy 100% of Spark Memory for cached DataFrames.
2. Contention (The Asymmetric Eviction Rule):
   • Execution Evicts Storage: If Execution Memory requires additional space and Storage Memory is currently borrowing execution capacity, Execution will forcefully evict Storage blocks from memory. Evicted blocks are dropped to disk (if cached with MEMORY_AND_DISK) or invalidated from memory (if cached with MEMORY_ONLY).
   • Storage Cannot Evict Execution: If Storage Memory requires space to cache a new DataFrame, but Execution Memory is currently occupying the space, Storage CANNOT evict Execution blocks. The caching operation either drops records to disk or skips caching them.

Why Is Eviction Asymmetric?

Execution blocks hold active, mid-flight computational state (such as partial aggregations or sort buffers). Evicting an active execution block would cause running tasks to fail or force expensive intermediate disk-thrashing. In contrast, cached storage blocks are immutable and can be recomputed when their required source data and dependencies remain available.

---

3. Off-Heap Memory and Project Tungsten

Traditional JVM execution stores data as Java objects on the JVM heap. A Java object layout adds representation-dependent overhead:

• Java object overhead depends on JVM layout, compressed references, alignment, and data type; many small objects can use much more memory than their payload.
• Garbage collection must traverse millions of these tiny object pointers, resulting in severe "Stop-the-World" GC pauses.

Project Tungsten Binary Format

Project Tungsten re-architected Spark's execution engine by treating memory like raw C-style memory buffers:

1. Binary Row Format: Rows are stored as compact, contiguous binary byte arrays rather than Java objects. Field access uses 64-bit word offsets, eliminating object headers and pointer dereferencing.
2. Cache-Aware Algorithms: Because rows are packed densely in contiguous memory, CPU cache lines (L1, L2, L3) are saturated with real data rather than pointer references, speeding up memory bandwidth.
3. Whole-Stage Code Generation: Synthesizes bytecode for an entire stage into a single Java function loop compiled on-the-fly via the Janino compiler, keeping active data in CPU registers.

Off-Heap Memory Configuration

Spark can allocate execution and storage memory outside the JVM heap using native memory allocators via sun.misc.Unsafe:

spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=10g

When off-heap memory is enabled, Tungsten writes binary records directly to operating system memory. This memory is completely invisible to the JVM Garbage Collector, reducing GC tracking for those buffers; heap objects, metadata, and Python workers still consume memory and can pause or fail.

---

4. Fault Tolerance Architecture in Apache Spark

Distributed systems face frequent node failures, hardware stalls, and network interruptions. Spark ensures resilience through four complementary layers of fault tolerance:

1. RDD Lineage Recomputation

• DataFrames and RDDs are immutable representations of data.
• Each DataFrame retains a lineage graph (metadata recording the exact sequence of transformations and input file sources that created it).
• If a worker node crashes and loses an in-memory partition, the DAG Scheduler inspects the lineage graph and re-executes only the missing partition on another available worker node, without restarting the job.

2. Shuffle File Persistence

• In wide transformations, map tasks write their shuffle output files to local executor disk.
• If a reducer task fails in Stage 2, it simply re-fetches the persisted shuffle files from Stage 1 executors rather than recomputing Stage 1.
• If an executor hosting Stage 1 shuffle files dies, only the specific map tasks from that failed executor are rescheduled.

3. Task Retries (spark.task.maxFailures)

• When a task throws an exception (such as a transient network timeout or connection reset), Spark does not immediately abort the job.
• The scheduler re-attempts the task up to spark.task.maxFailures times (default is 4).
• Repeated counted failures of the same task can abort the stage/job at the configured threshold. A failed job does not necessarily terminate a long-lived application; fatal driver failure is different.

4. Speculative Execution (spark.speculation)

• In heterogeneous clusters, some tasks run substantially slower than others due to degraded worker hardware, slow disks, or long GC pauses (known as stragglers).
• When spark.speculation = true, Spark monitors task progress across the stage. If a task runs significantly slower than the median task completion time, Spark launches a speculative duplicate copy on an alternate executor.
• Whichever task copy finishes first is committed; the slower copy is killed.

---

5. JVM Garbage Collection Optimization

When working with large JVM heaps (> 8 GB), garbage collection pauses are a frequent source of executor failure. If a "Stop-the-World" GC pause exceeds spark.network.timeout (default 120 seconds), the driver mistakenly assumes the executor has died, removes it from the cluster, and triggers unnecessary task recomputation.

Recommended Garbage Collector: G1GC

The Garbage-First Garbage Collector (G1GC) is one JVM option. Use the supported runtime defaults and GC logs before changing collector options; this is an illustrative configuration, not a required recipe:

spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 -XX:MaxGCPauseMillis=200

• Regional Division: G1GC breaks the JVM heap into hundreds of small equal regions (1 MB to 32 MB) rather than maintaining massive monolithic Old and Young generations.
• Density-Based Reclamation: G1GC prioritizes regions containing the highest density of dead objects (collecting the "garbage first") to meet the configured pause target (-XX:MaxGCPauseMillis=200).

Best Practices to Mitigate GC Pressure

1. Use DataFrames Instead of RDDs: DataFrames use Tungsten binary row format, reducing per-row JVM object allocation. JVM-object RDDs can create large object graphs; Python-object RDDs also consume Python worker memory, overwhelming the garbage collector.
2. Avoid Python UDFs: Python UDFs require transferring data across sockets between the JVM and Python worker subprocesses, causing extreme object allocation and serialization churn. Prefer built-in pyspark.sql.functions.
3. Tune spark.memory.fraction: Increasing the unified-memory fraction leaves less heap for other objects. Diagnose partition sizes, task concurrency, user objects, and GC before adjusting it; a larger fraction can worsen failures elsewhere.

Official reference

Memory and GC tuning — source checked October 16, 2026. The examples state their runtime assumptions.