1.2 Driver, Executors, and Cluster Managers

Apache Spark is a distributed computing engine engineered to process large-scale datasets across clusters of commodity or cloud-based machines. At the core of Spark's architecture is a centralized master-worker model. A single coordinator process—the Driver—collaborates with a cluster manager to launch, monitor, and assign computation to worker processes known as Executors.

Understanding how compute resources are divided, how the Driver communicates with Executors, and how various cluster managers allocate hardware is foundational knowledge for any Spark developer and a frequent focus of certification exam questions.

---

The Master-Worker Architecture

A Spark application operates as an independent set of processes running on a cluster. The architecture decouples the high-level orchestration of compute from the low-level execution of data tasks:

1. The Driver Node (Master): Translates user code into execution plans, delegates tasks, and tracks overall application health.
2. The Cluster Manager: Allocates cluster resources (CPU, RAM) across applications; it does not plan Spark SQL queries.
3. Executor Nodes (Workers): Execute assigned tasks in parallel, persist cached data partitions in memory, and report status back to the Driver.

+-------------------------------------------------------------------------+
|                              DRIVER NODE                                |
|  +-----------------------+     +------------------+                     |
|  | SparkSession / Plan   | --> | DAGScheduler     |                     |
|  +-----------------------+     +------------------+                     |
|             |                            |                              |
|             v                            v                              |
|  +-----------------------+     +------------------+                     |
|  | Spark Web UI (:4040)  |     | TaskScheduler    |                     |
|  +-----------------------+     +------------------+                     |
+-------------------------------------------------------------------------+
                                     | Tasks & Heartbeats
       +-----------------------------+-----------------------------+
       |                                                           |
       v                                                           v
+------------------------------+            +------------------------------+
|     EXECUTOR 1 (Worker)      |            |     EXECUTOR 2 (Worker)      |
|  +------------------------+  |            |  +------------------------+  |
|  | Task Execution Slots   |  |            |  | Task Execution Slots   |  |
|  | [Core 1]  [Core 2]     |  |            |  | [Core 1]  [Core 2]     |  |
|  +------------------------+  |            |  +------------------------+  |
|  +------------------------+  |            |  +------------------------+  |
|  | BlockManager (Cache)   |  |            |  | BlockManager (Cache)   |  |
|  +------------------------+  |            |  +------------------------+  |
+------------------------------+            +------------------------------+

---

The Driver Node: The Orchestration Center

The Driver is the brain of the Spark application. When you launch a PySpark script or interactive notebook, the Driver process is instantiated to run your main() method.

Primary Driver Responsibilities

• Maintaining Application State and Session: The Driver instantiates and maintains the SparkSession (or SparkContext in legacy code), which serves as the unified entry point for reading data, executing SQL queries, and configuring runtime properties.
• Formulating the Execution Plan (DAG): When transformations are chained, the Driver does not immediately compute rows. Instead, it records them into a logical plan and compiles them into a Directed Acyclic Graph (DAG) of stages and tasks.
• Task Scheduling and Dispatch: The Driver hosts two critical internal schedulers:
  • DAGScheduler: Divides the RDD dependency graph produced by the chosen physical plan into stages at shuffle boundaries.
  • TaskScheduler: Accepts TaskSets from the DAGScheduler and dispatches individual tasks to Executors based on data locality.
• Serving the Spark Web UI: By default, the Driver binds to port 4040 (e.g., http://<driver-ip>:4040) to serve an interactive Web UI displaying active and completed jobs, stages, storage memory usage, environment variables, and executor health.
• Collecting Results: When an action such as collect() or take() is executed, the Driver coordinates the retrieval of partition results from Executors over the network and brings them into its local memory.

Single Point of Failure (SPOF)

In standard cluster environments, the Driver represents a single point of failure. If the Driver process runs out of memory or crashes, all associated Executors are terminated by the cluster manager, and the Spark application immediately fails.

---

Executor Nodes: Distributed Execution Engines

Executors are worker processes running within their own Java Virtual Machine (JVM) instances across the cluster's worker nodes. Executors belong to one application and can remain for its lifetime. Dynamic allocation, when enabled, can add or remove them as demand changes.

Primary Executor Responsibilities

• Running Assigned Tasks: An Executor receives serialized task instructions from the Driver's TaskScheduler, deserializes the code, and executes it against local data partitions across multi-threaded task slots.
• Managing In-Memory Storage (BlockManager): Each Executor includes an internal BlockManager that caches DataFrames/RDDs in local memory or spills them to disk when df.cache() or df.persist() is invoked.
• Managing Shuffle Output Files: When wide transformations require data repartitioning, Executors write intermediate shuffle partition files to their local disk storage so downstream stages can fetch them.
• Transmitting Heartbeats: Executors communicate back to the Driver via periodic heartbeats (default spark.executor.heartbeatInterval is 10 seconds), reporting task completion metrics, memory utilization, and liveness. If the Driver does not receive a heartbeat within spark.network.timeout (default 120 seconds), the Executor is marked dead and its tasks are rescheduled.

---

Worker Nodes vs. Executors: Resource Allocation

A critical distinction on the exam is the difference between a Worker Node and an Executor:

• Worker Node: A physical computer or virtual machine (e.g., an AWS EC2 instance or Azure VM) attached to the cluster.
• Executor: A distinct JVM process running inside a Worker Node.

A single Worker Node can host multiple Executor JVMs depending on hardware configuration. Compute capacity is determined by three key configuration settings:

---

Driver vs. Executor Comparison

---

Cluster Managers and Deployment Modes

Apache Spark relies on an external Cluster Manager to acquire physical machines and allocate JVM containers. Common cluster managers and managed deployment environments include:

1. Standalone: Spark's built-in cluster manager. Simple to deploy, featuring a dedicated Standalone Master and Worker daemons.
2. Apache YARN: The standard Hadoop resource manager consisting of a centralized ResourceManager and per-node NodeManager daemons.
3. Kubernetes (K8s): Cloud-native container orchestration. The Driver runs in a K8s Driver Pod, which directly invokes the Kubernetes API server to spawn Executor Pods on demand.
4. Databricks Managed Compute: Fully managed enterprise compute architecture. Databricks automates cloud VM provisioning, runtime patching, and fine-grained autoscaling across All-Purpose Clusters (interactive collaborative development) and Job Clusters (automated production ETL pipelines).

Deployment Modes: Client vs. Cluster

When submitting a Spark application via spark-submit, the --deploy-mode flag determines where the Driver process physically executes:

# Submitting in cluster mode
spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --executor-memory 4g \
  --executor-cores 4 \
  my_etl_script.py

---

Initializing SparkSession in PySpark

In modern Spark (versions 2.0 through 3.x+), the SparkSession provides a single unified entry point to DataFrame and SQL functionality, replacing the legacy SparkContext, SQLContext, and HiveContext.

from pyspark.sql import SparkSession

# Instantiate the unified SparkSession entry point on the Driver
spark = SparkSession.builder \
    .appName("ProductionDataPipeline") \
    .config("spark.executor.instances", "4") \
    .config("spark.executor.cores", "4") \
    .config("spark.executor.memory", "8g") \
    .config("spark.driver.memory", "4g") \
    .config("spark.sql.shuffle.partitions", "200") \
    .getOrCreate()

# Accessing underlying SparkContext if low-level RDD APIs are required
sc = spark.sparkContext
print(f"Spark App ID: {sc.applicationId}")
print(f"Spark Master: {sc.master}")

These requests apply when a new application is launched and the deployment supports them. An existing session may be reused; deployment properties cannot resize an already running driver. Set driver heap and executor resources in spark-submit or the platform compute configuration before launch. With four four-core executors, default task CPU requests permit up to 16 concurrent tasks.