1.3 Execution Hierarchy: Applications, Jobs, Stages, and Tasks

To optimize and parallelize large-scale computation, Apache Spark organizes execution into a disciplined four-tiered hierarchy: Application, Job, Stage, and Task. When an engineer executes a PySpark script, Spark does not simply run queries row-by-row. Instead, it compiles user code into a Directed Acyclic Graph (DAG) and decomposes the work across this execution hierarchy.

Understanding where boundaries are drawn between these four levels—and specifically what role actions, shuffles, and partitions play—is one of the most heavily tested areas on the Databricks Associate Developer exam.

---

The Four-Tier Execution Pyramid

+-------------------------------------------------------------------------+
|                        LEVEL 1: SPARK APPLICATION                       |
|    Underlying SparkContext lifecycle; sessions can share one application   |
+-------------------------------------------------------------------------+
                                     |
                                     | Triggers 1 or more
                                     v
+-------------------------------------------------------------------------+
|                           LEVEL 2: SPARK JOB                            |
|           Instantiated whenever an ACTION is called (e.g., count)       |
+-------------------------------------------------------------------------+
                                     |
                                     | Divided by Shuffle Boundaries
                                     v
+-------------------------------------------------------------------------+
|                          LEVEL 3: SPARK STAGE                           |
|       Chains pipelined narrow transformations (ShuffleMap or Result)    |
+-------------------------------------------------------------------------+
                                     |
                                     | 1 Task per Data Partition
                                     v
+-------------------------------------------------------------------------+
|                          LEVEL 4: SPARK TASK                            |
|     Smallest execution unit; runs on 1 executor core on 1 partition     |
+-------------------------------------------------------------------------+

---

Level 1: The Spark Application

A Spark Application represents the overarching operational lifecycle of a user's program. It begins with its underlying SparkContext and driver lifecycle, often initialized through a SparkSession on the Driver and ends when spark.stop() is invoked or the main process terminates.

• An application encompasses the Driver process, all allocated Executor JVMs, and the continuous execution of all user code.
• An application can run multiple sequential or concurrent Jobs throughout its lifecycle.
• Applications normally have separate executor processes. Physical worker nodes and cluster capacity can be shared; this is process isolation, not exclusive ownership of the hardware.

---

Level 2: The Spark Job

A Job is a sequence of parallel computations triggered directly by an Action.

• What Triggers a Job? Whenever an action is invoked—such as df.count(), df.collect(), df.take(10), df.show(), or df.write.parquet(...)—Spark translates the accumulated DataFrame lineage into a physical execution plan and can launch one or more jobs, depending on the physical plan and result retrieval.
• Most ordinary transformations are lazy: Calling transformations such as df.select(), df.filter(), or df.withColumn() merely appends nodes to the logical plan; zero jobs are launched until an action is evaluated.
• An application can contain dozens or hundreds of sequential Jobs. In the Spark Web UI, the "Jobs" tab lists each job with its corresponding job ID, submission time, duration, and triggering action method.

---

Level 3: The Spark Stage and Shuffle Boundaries

Each Spark Job is decomposed into one or more Stages. The division between stages is determined strictly by Shuffle Boundaries created by wide transformations.

Narrow Transformations vs. Wide Transformations

Pipelining Within a Stage

When multiple narrow transformations are chained in succession—such as df.filter(...).select(...).withColumn(...)—Spark optimizes them through pipelining. Pipelining collapses these operations into a single continuous in-memory execution loop within the same stage. Data rows are transformed without writing intermediate results to memory buffers or disk.

What Happens at a Shuffle Boundary?

When a wide transformation (such as groupBy("department").agg(...)) is encountered, data must be re-grouped across partitions based on a hash of the grouping key. Because rows belonging to a department may reside across multiple worker nodes, Spark must perform a shuffle:

1. Shuffle Map Stage: The preceding stage writes its output partitions to local executor disk files sorted by hash bucket.
2. Network Exchange: The cluster network transfers shuffle blocks from source executors to destination executors.
3. Result / Downstream Stage: The new stage reads the shuffled partition blocks and proceeds with computation.

---

Level 4: The Spark Task

A Task is the absolute smallest unit of execution in Apache Spark. It represents the application of a stage's computational logic to exactly one partition of data.

The 1:1 Mapping Rule

There is a strict 1:1 relationship between partitions and tasks:

Number of logical tasks for a stage = partitions computed by that stage; attempts can be higher

• If a DataFrame has 100 partitions during a Stage, Spark creates 100 logical tasks for that stage; failures/speculation may add attempts.
• Each Task is assigned to a single CPU core (task execution slot) on an Executor.
• If your cluster has 20 total executor cores, the 100 tasks will be processed in waves (5 waves of 20 tasks running concurrently).

Shuffle Partition Count

For stages following a wide transformation (like a join or aggregation), the number of partitions—and therefore the number of tasks—is governed by the configuration parameter:

# Default is 200 shuffle partitions in Spark SQL / DataFrame APIs
spark.conf.set("spark.sql.shuffle.partitions", "200")

The default initial shuffle count is 200 in upstream Spark 3.5.7. AQE may coalesce it. Small partitions can waste scheduling time; large partitions can spill or exhaust memory. Neither outcome follows from input bytes alone.

---

DAG Formulation: DAGScheduler and TaskScheduler

The translation of high-level code into low-level tasks is managed by two core components on the Driver:

Loading diagram...

1. DAGScheduler (High-Level Scheduler):
   • Examines the RDD/DataFrame lineage graph.
   • Identifies wide dependencies and carves the graph into a DAG of discrete stages.
   • Determines the execution sequence and creates a TaskSet containing all tasks for each ready stage.
2. TaskScheduler (Low-Level Scheduler):
   • Receives each TaskSet from the DAGScheduler.
   • Dispatches tasks to available executor cores across the cluster.
   • Prioritizes Data Locality to minimize network transfer: PROCESS_LOCAL (data in same JVM) > NODE_LOCAL (data on same host disk) > RACK_LOCAL (data on same server rack) > ANY (network transfer required).
   • Monitors task execution and retries failed tasks up to spark.task.maxFailures (default 4) before failing the stage.

---

Code Walkthrough: Tracing an Action Through the Hierarchy

For this teaching model, assume two scan partitions, AQE disabled, no prior cached/shuffle results, and the two-stage physical plan described below. File count alone does not establish scan partition count; verify the real plan and UI.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

# Application begins
spark = SparkSession.builder.appName("HierarchyDemo").getOrCreate()

# 1. Read dataset (assume this scan yields 2 partitions)
spark.conf.set("spark.sql.adaptive.enabled", "false")
spark.conf.set("spark.sql.shuffle.partitions", "200")
raw_df = spark.read.schema(
    "status_code INT, user_id STRING, url STRING, bytes_sent LONG"
).csv("/data/web_traffic.csv", header=True)

# 2. Narrow Transformations (Pipelined together in Stage 1)
clean_df = raw_df \
    .filter(col("status_code") == 200) \
    .select("user_id", "url", "bytes_sent") \
    .withColumn("mb_sent", col("bytes_sent") / (1024 * 1024))

# 3. Wide Transformation (Introduces Shuffle Boundary)
grouped_df = clean_df.groupBy("url").sum("mb_sent")

# 4. Action: Triggers JOB 0
grouped_df.write.mode("overwrite").parquet("/output/summary.parquet")

Execution Breakdown

1. Application: HierarchyDemo encompasses the entire execution.
2. Job: The .write.parquet(...) action requests execution; label the illustrated job J (its real ID depends on earlier work).
3. Stage 1 (ShuffleMapStage):
   • Contains reading the CSV, applying .filter(), .select(), and .withColumn(). All three transformations are pipelined into a single in-memory task.
   • Under the stated two-input-partition assumption, this stage has 2 logical tasks. Files can be split or combined into partitions.
   • Output rows are hashed by url and written to local shuffle files.
4. Stage 2 (ResultStage):
   • Reads the shuffled blocks across the network.
   • Computes sum("mb_sent") for each unique URL.
   • Writes the Parquet output files.
   • Governed by spark.sql.shuffle.partitions (default 200), executing exactly 200 tasks.
5. Logical Tasks in the illustrated two-stage job: 2 + 200 = 202 tasks.