2.3 Caching, Persistence, and StorageLevels

In Apache Spark, DataFrames are evaluated lazily. Without a reusable cache or other retained/reused results, later actions may repeat upstream computation; exact work depends on the optimized plan and available outputs. In complex analytical pipelines, iterative machine learning algorithms, or branching workflows, re-executing identical computations introduces massive CPU, I/O, and network redundancies. Caching and persistence mechanisms allow developers to store intermediate DataFrame partitions in memory or on disk for rapid downstream reuse.

---

1. Caching Fundamentals: cache() vs. persist()

Spark provides two primary methods on the DataFrame API to retain intermediate datasets across actions: cache() and persist().

# Using cache() shortcut
df_cached = df.cache()

# Using persist() with an explicit StorageLevel
from pyspark.storage import StorageLevel
df.unpersist()  # Remove an existing persistence assignment before changing its level
df_persisted = df.persist(StorageLevel.MEMORY_ONLY)

The Relationship Between cache() and persist()

df.cache() is simply an alias that calls df.persist() with a default storage level:

• In PySpark DataFrames, df.cache() defaults to StorageLevel.MEMORY_AND_DISK_DESER (in Scala/Java DataFrames and Spark SQL, this corresponds to StorageLevel.MEMORY_AND_DISK).
• In low-level RDDs, rdd.cache() historically defaulted to StorageLevel.MEMORY_ONLY.
• Because the PySpark DataFrame default MEMORY_AND_DISK_DESER permits disk storage, if the dataset exceeds available executor memory, Spark does not drop the overflow partitions; instead, it automatically serializes and spills them to local executor disks.

Lazy Evaluation of Caching

A critical concept on Databricks certification exams is the lazy evaluation of caching:

# Step 1: Tag the DataFrame for caching (NO DATA IS CACHED YET)
clean_df = raw_df.filter("age > 21").cache()

# Step 2: An action is called (DATA IS COMPUTED AND STORED IN CACHE HERE)
row_count = clean_df.count()

# Step 3: Subsequent actions reuse the cached memory buffers directly
summary_df = clean_df.groupBy("country").avg("income").collect()

When clean_df.cache() is invoked, Spark merely attaches an InMemoryRelation marker to the logical execution plan. No memory is allocated, and no files are read. Only when the first action (.count()) executes does Spark read the source data, apply the filter, and materialize the resulting partition blocks into executor storage memory.

---

2. StorageLevel Configurations and Trade-offs

The pyspark.storage.StorageLevel class provides fine-grained control over how cached partition blocks are stored. A StorageLevel encapsulates four boolean flags and an integer replication count: useDisk, useMemory, useOffHeap, deserialized, and replication.

These are Python API constants in Spark 3.5.7. Scala names ending in _SER must not be copied into PySpark code: MEMORY_AND_DISK_SER and MEMORY_ONLY_SER are not PySpark constants. DataFrame SQL caching uses cached columnar batches; these flags do not mean each SQL row becomes a Python or Java object. OFF_HEAP needs off-heap configuration and sufficient native/container capacity; it does not eliminate JVM GC. Replication improves availability but does not replace a durable source or checkpoint.

---

3. The Cache Lifecycle: Best Practices and Eviction

Caching is not free. Misusing cache can degrade performance rather than improving it.

When TO Cache

1. Iterative Algorithms: Workflows that access the same intermediate dataset repeatedly in a loop (e.g., machine learning model training loops such as Logistic Regression, K-Means, or Gradient Boosted Trees).
2. Multi-Branching Pipelines: When an expensive set of transformations (e.g., parsing raw JSON, deduplicating, and joining with dimension tables) produces a clean DataFrame that is subsequently consumed by multiple independent downstream actions or written to multiple destinations.

# Good caching candidate: clean_df is consumed by two separate branches
clean_df = raw_df.join(lookup_df, "id").filter("is_valid == true").cache()

# Branch 1: Write raw clean data to Parquet
clean_df.write.parquet("/mnt/lakehouse/clean_orders")

# Branch 2: Compute aggregation metrics
metrics_df = clean_df.groupBy("region").agg(F.sum("amount"))
metrics_df.write.parquet("/mnt/lakehouse/regional_metrics")

When NOT to Cache

• Single-Use DataFrames: If a DataFrame is read, transformed, and written to disk exactly once, calling .cache() adds pure write and read overhead without any performance benefit.
• DataFrames That Vastly Exceed Memory: A 2 TB cache on 200 GB RAM may use substantial disk, evict blocks, or perform poorly. Compare avoided recomputation against cache IO rather than assuming all such caches are beneficial or harmful.
• Caching Prior to Narrow Filters: Caching a DataFrame before filtering restricts the Catalyst optimizer from pushing filter predicates down into the underlying Parquet/Delta storage reader.

Explicit Eviction with unpersist()

When a cached DataFrame is no longer needed in subsequent stages of a pipeline, it should be eagerly removed from memory using df.unpersist():

# Free cached memory buffers
clean_df.unpersist(blocking=True)

• The blocking Argument: Setting blocking=True halts the calling thread until all executor nodes have completely evicted the partition blocks from memory and disk. By default, blocking=False initiates asynchronous eviction in the background.
• Automatic LRU Eviction: If memory becomes saturated and new data must be cached, Spark automatically evicts the oldest cached blocks using a Least Recently Used (LRU) policy. However, relying strictly on LRU eviction can lead to unpredictable memory pressure and unexpected recomputations.

---

4. Catalyst Optimizer and Project Tungsten Memory Coordination

Caching interacts directly with Spark's underlying query optimizer (Catalyst) and memory execution engine (Project Tungsten).

Caching as a Lineage Boundary

When a DataFrame is cached, Catalyst treats the cached relation as an InMemoryRelation:

• It substitutes a cached relation in the execution plan without deleting recovery lineage for downstream queries, reading pre-computed columnar buffers rather than re-evaluating upstream operators.
• In-Memory Optimization: Although the cached scan substitutes for upstream computation, Catalyst can still optimize queries that read from the cache. For example, if a query selects only two columns from a 50-column cached DataFrame, Catalyst performs column pruning on the in-memory columnar store, reading only the two required column byte streams.

Project Tungsten Columnar Storage

Under Project Tungsten, cached DataFrames are stored in a compact in-memory columnar representation:

1. Columnar In-Memory Format: Values from the same column are stored contiguously in memory, enabling hardware-level Single Instruction Multiple Data (SIMD) CPU vectorization.
2. Compact buffers: Columnar cached batches reduce per-row object overhead. They are not automatically all off heap, and native memory configuration is separate.
3. Whole-Stage Code Generation: Physical plan operators reading from cached memory buffers continue to leverage compiled bytecode loops (indicated by the * prefix in df.explain()), reducing interpretation overhead where the operators support code generation; throughput depends on the workload.

Official reference

PySpark storage levels — source checked October 16, 2026. The examples state their runtime assumptions.