11.1 Diagnosing Performance with the Spark UI & Execution Plans

Optimizing Apache Spark applications requires deep visibility into how queries are planned, distributed, and executed across a cluster. When an application experiences high latency, resource starvation, or out-of-memory errors, developers rely on two primary diagnostic tools: the Spark Web UI and execution plans generated by explain(). Mastering these diagnostic interfaces is essential for identifying bottlenecks, detecting data skew, and evaluating physical execution strategies.

---

1. Navigating the Spark Web UI

The Spark Web UI (accessible by default on port 4040 of the driver node during active execution, or through the Spark History Server post-execution) provides granular telemetry across all phases of computation. Understanding the distinct responsibilities of each tab allows developers to isolate performance issues rapidly.

The Jobs Tab

Every time an action (such as .count(), .collect(), .take(), or .write()) is invoked on a DataFrame, Spark schedules a Job. The Jobs tab provides a chronological overview of all jobs within an application:

• Job Status: Displays active, completed, and failed jobs along with submission time, total duration, and completion progress.
• Event Timeline: Visualizes the allocation of executors over time alongside job and stage boundaries. It highlights whether executor cores are sitting idle or actively executing tasks.
• Associated Stages: Lists the specific stages that comprise each job, indicating which stage is currently consuming execution time.

The Stages Tab

A job is broken into one or more Stages separated by shuffle boundaries (wide transformations such as groupBy(), join(), or distinct()). The Stages tab is the primary interface for identifying operational bottlenecks:

• Task Distribution Quantiles Table: For every stage, Spark displays statistical metrics across all completed tasks at the 0th percentile (minimum), 25th percentile, 50th percentile (median), 75th percentile, and 100th percentile (maximum). This table reports:
  • Duration: The elapsed compute time per task.
  • GC Time: Time spent paused during Java Virtual Machine (JVM) garbage collection.
  • Task Deserialization Time: Time spent converting task bytes received from the driver into runnable task objects.
  • Shuffle Read / Write Size and Records: Volume of intermediate data written to or fetched from shuffle files across cluster nodes.
  • Spill (Memory) and Spill (Disk): Volume of data evicted from memory to disk during sorting or aggregation.

The Executors Tab

The Executors tab provides a hardware-centric perspective on cluster resource utilization:

• Executor Metrics: Lists the Driver and all active worker Executors, displaying their IP addresses, assigned CPU cores, and total allocated memory.
• Task Metrics per Node: Shows active tasks, failed tasks, completed tasks, and total task time per executor.
• Garbage Collection (GC) Time vs. Compute Time: Highlights individual nodes experiencing disproportionate GC overhead.
• Storage Memory & Disk Used: Shows block-storage usage; it is not a complete measurement of free JVM, Python, native, or container memory.

The Storage Tab

When you invoke .cache() or .persist() on a DataFrame and materialize it via an action, the Storage tab displays the operational footprint of the cached data:

• Cached RDD/DataFrame Name: Identifier of the persisted dataset.
• Storage Level: Details the persistence mode (e.g., Memory Deserialized 1x Replicated or Memory Serialized 1x Replicated).
• Fraction Cached: The percentage of partition blocks successfully loaded into memory (e.g., 100% indicates complete caching; a lower percentage indicates memory exhaustion where some partitions were not evaluated, not cached, or evicted).
• Size in Memory vs. Size on Disk: Explicit byte consumption in RAM versus spilled bytes on disk.

The Environment Tab

The Environment tab lists the complete configuration profile of the active Spark session:

• JVM Runtime & System Properties: Java version, Scala version, and system environment variables.
• Spark Properties: All configured runtime settings (such as spark.sql.shuffle.partitions, spark.executor.memory, and spark.sql.adaptive.enabled).
• Classpath Entries: All loaded JAR dependencies and application libraries.

The SQL / DataFrame Tab

The SQL tab presents a high-level, graphical Directed Acyclic Graph (DAG) for every executed SQL query or DataFrame operation. Clicking an execution details each physical plan operator (e.g., FileScan parquet, Exchange, HashAggregate, SortMergeJoin) with real-time metrics annotated directly onto the graph:

• Number of output rows: Tracks row volume through each stage of the pipeline.
• Scan duration & metadata: Displays the time spent reading files and the count of files read.
• Peak Execution Memory: Highest memory footprint consumed by operators during in-memory hashing or sorting.
• Spill Metrics: Direct visual indicators showing whether individual operators spilled data to disk.

---

2. Diagnosing Common Performance Anti-Patterns in the UI

When investigating slow or failing queries, three specific diagnostic signatures appear frequently on certification exams and in production environments.

1. Identifying Data Skew (The "Straggler Task" Problem)

Data skew occurs when one or more partitions contain significantly more records than the remaining partitions. In the Stages tab, data skew manifests as a severe quantile imbalance:

In this scenario, 75% of all tasks complete in under 2.5 seconds processing ~50,000 records. However, the maximum task takes over 18 minutes because it was assigned 14.5 million records associated with a single skewed key. This straggler task holds the entire stage open, preventing downstream stages from executing.

2. Identifying JVM Garbage Collection Pressure

Spark executors run inside JVM processes. When executor memory is overwhelmed by millions of short-lived Java objects or oversized in-memory collections, the JVM initiates stop-the-world garbage collection pauses:

• Investigation heuristic: Sustained high GC time relative to task elapsed time merits inspection. Ten percent is an example trigger to investigate, not a universal failure threshold or CPU-time measurement.
• In the Executors tab, inspect the Task Time (GC Time) column. If total task time is 10 minutes and GC time is 3.5 minutes (35%), executor memory is severely constrained.
• Remediation: Increase spark.executor.memory, review the actual PySpark StorageLevel constants and reduce unnecessary cached data, reduce executor core count per node (to reduce concurrent task allocation), or avoid operations that aggregate high-cardinality collections into single rows (such as F.collect_list()).

3. Diagnosing Disk Spill

During shuffle, aggregation (HashAggregate), or sort (Sort) operations, Spark buffers data in execution memory. If the data volume exceeds available memory, Spark spills records to local executor disks:

• Spill (Memory): The size of the data in memory before it was spilled (uncompressed, deserialized).
• Spill (Disk): The size of the data after serialization and compression as written to disk.
• Because disk I/O and serialization are orders of magnitude slower than memory access, queries with large spill metrics experience severe performance penalties.
• Remediation: Increase partition count via repartition() or spark.sql.shuffle.partitions, or allocate additional memory per executor core.

---

3. Interpreting Execution Plans with df.explain()

While the Spark UI provides post-execution and runtime telemetry, inspecting execution plans allows developers to analyze query planning before and during execution.

# Basic plan output
df.explain()

# Formatted plan with structured breakdown (Spark 3.0+)
df.explain(mode="formatted")

Explain Modes in Spark 3.x

The mode argument in df.explain(mode=...) supports five options:

1. "simple": Displays the physical plan only.
2. "extended": Displays the Parsed Logical Plan, Analyzed Logical Plan, Optimized Logical Plan, and Physical Plan.
3. "codegen": Displays the physical plan alongside the generated Java source and codegen details by Whole-Stage Code Generation.
4. "cost": Displays the optimized logical plan annotated with Cost-Based Optimizer (CBO) statistics (if CBO is enabled).
5. "formatted": Splits the plan into a high-level DAG outline and a detailed per-node section detailing schema, expressions, and pushed predicates.

The Four Stages of Catalyst Plan Generation

Understanding how Spark translates code into physical execution is critical for diagnosing optimization failures:

[ Unresolved Logical Plan ]  (Parsed code, table/column names unverified)
           ↓  (Catalog Analysis)
[ Analyzed Logical Plan ]    (Types resolved, relations confirmed in metastore)
           ↓  (Logical Optimizations: Constant folding, predicate pushdown)
[ Optimized Logical Plan ]   (Minimal data movement, filters placed early)
           ↓  (Physical Planning: Strategy selection)
[ Physical Plan ]            (Concrete operators: HashAggregate, BroadcastHashJoin)

Key Physical Plan Operators to Recognize

When examining a physical plan, developers should search for specific operator nodes:

== Physical Plan ==
*(2) HashAggregate(keys=[department#12], functions=[sum(salary#14)])
+- Exchange hashpartitioning(department#12, 200), ENSURE_REQUIREMENTS, [id=#45]
   +- *(1) HashAggregate(keys=[department#12], functions=[partial_sum(salary#14)])
      +- *(1) Project [department#12, salary#14]
         +- *(1) Filter (isnotnull(salary#14) AND (salary#14 > 50000))
            +- *(1) FileScan parquet [department#12,salary#14] 
                 Batched: true, DataFilters: [isnotnull(salary#14), (salary#14 > 50000)], 
                 Format: Parquet, PartitionFilters: [], 
                 PushedFilters: [IsNotNull(salary), GreaterThan(salary,50000)]

1. FileScan parquet ... PushedFilters: [...]: Confirms predicate pushdown. Filters pushed to the file reader allow Spark to skip non-matching Parquet row groups at the storage layer without reading them into memory.
2. *(1) Prefix (Whole-Stage Code Generation): Asterisks preceding operator names (such as *(1) Filter) denote that Project Tungsten fused multiple physical operators into a single, compiled Java bytecode loop, eliminating virtual method invocation overhead.
3. Exchange hashpartitioning(...): Confirms that a shuffle boundary exists. Data is being repartitioned across the network by the hash of the specified key.
4. Join Operators: The plan explicitly identifies the chosen join algorithm: BroadcastHashJoinExec (no shuffle of the large table), SortMergeJoinExec (two-sided shuffle and sort), or ShuffledHashJoinExec.

Driver and executor logs: publish, customize, diagnose

The UI is a summary, while logs provide stack traces and failure context. Driver logs show query analysis, scheduling and result-collection failures; executor stdout/stderr show task exceptions, Python worker failures, GC symptoms and container losses. In the Executors tab follow the platform-provided stdout/stderr links where available. YARN aggregates application container logs; Kubernetes uses driver/executor pod logs. Managed platforms expose compute log downloads or delivery settings. An unavailable link is a deployment configuration issue, not proof that no executor error occurred.

Enable spark.eventLog.enabled and configure a durable spark.eventLog.dir before launch for later History Server inspection. Event logs record application events; they do not replace stdout/stderr application logs. Configure log4j2.properties for JVM logger names, levels and appenders; distribute the intended configuration to both driver and executors using supported deployment options. spark.sparkContext.setLogLevel("WARN") changes logging in a classic session, but Python logging inside a UDF runs in worker processes and must be configured there. Avoid verbose record-level logging in a large job.

For a driver failure after collect(), inspect result-size limits and JVM/Python memory. For executor OOM, inspect the failing task's partition bytes, join build side, grouping cardinality, Python/native allocations and container exit reason. For idle cores, check the number of runnable tasks, scheduler resource requests, skew, source parallelism and executor provisioning. Increasing workers cannot parallelize one huge hot-key task. Record the application ID, job/stage/task attempt and executor ID before comparing runs, and change one measured bottleneck at a time.

Official reference

Monitoring and logs — source checked October 16, 2026. The examples state their runtime assumptions.