2.1 Narrow vs Wide Dependencies and the Shuffle Mechanism

In Apache Spark, computation is expressed as a Directed Acyclic Graph (DAG) of transformations applied to distributed data collections (DataFrames and RDDs). Understanding how Spark executes this graph requires examining the fundamental mathematical relationship between parent partitions and child partitions. This dependency classification dictates how Spark schedules tasks, allocates memory, and manages physical data movement across cluster nodes.

---

1. Narrow Dependencies: Intra-Stage Pipelining

A narrow dependency occurs when each partition of the parent DataFrame or RDD is consumed by at most one partition of the child DataFrame or RDD. This relationship is typically one-to-one (one parent partition produces exactly one child partition) or occasionally many-to-one (such as coalesce reducing partition count without adding a shuffle without a network shuffle).

Parent Partition 0 ───────> Child Partition 0
Parent Partition 1 ───────> Child Partition 1
Parent Partition 2 ───────> Child Partition 2

Execution Characteristics of Narrow Dependencies

1. In-Memory Pipelining: Because all required data resides within a single partition on a single node, Spark chains multiple narrow operations together in memory. Spark executes these operations as a single pipeline without a shuffle between those operations; operator behavior and storage reads still determine I/O.
2. Whole-Stage Code Generation: Spark's Catalyst engine collapses chained narrow transformations into a single optimized Java bytecode loop using Janino compilation. For example, a select(), a filter(), and a withColumn() execute in CPU registers and L1/L2 cache within a single processing loop.
3. No Shuffle Introduced: The operation adds no shuffle exchange. Reading source or cached data can still involve a network, and narrow operators can have other memory or disk costs.
4. Pipelining: Compatible narrow operators can run in the same stage; not every operator supports whole-stage code generation.

Common Narrow Transformations

• select(...) / selectExpr(...): Projection of existing or computed columns.
• filter(...) / where(...): Row-level predicate evaluation.
• withColumn(...) / withColumnRenamed(...): Adding, transforming, or renaming columns.
• drop(...): Removing specified columns.
• map(...) / flatMap(...): Row-level transformations in RDD and Dataset APIs.
• union(...): Appending two DataFrames with identical schemas (each parent partition maps directly to a child partition without rearranging rows).
• coalesce(n) (when n < current_partitions): Combines existing partitions on the same worker nodes to reduce partition count without triggering a full network shuffle.

# Example of pipelined narrow operations within a single stage
df_filtered = (
    spark.read.parquet("dbfs:/data/transactions.parquet")  # Stage 0: Read
    .filter("amount > 100.0")                              # Stage 0: Narrow
    .select("transaction_id", "customer_id", "amount")      # Stage 0: Narrow
    .withColumn("amount_usd", col("amount") * 1.08)        # Stage 0: Narrow
)
# All above operations pipeline into a single task per input partition.

---

2. Wide Dependencies: Cluster-Wide Shuffles

A wide dependency (also called a shuffle dependency) occurs when multiple child partitions depend on data distributed across multiple parent partitions. This is an all-to-all or many-to-many relationship.

Parent Partition 0 ───┬───> Child Partition 0
                     ├───> Child Partition 1
Parent Partition 1 ───┼───> Child Partition 0
                     └───> Child Partition 1

Why Wide Dependencies Require a Shuffle

Operations like aggregation, grouping, and joining require records sharing the same key to be co-located within the same physical partition so they can be processed together. Because records with the same key may reside on any parent partition across any cluster worker node, Spark must redistribute the data across the cluster.

This physical data redistribution is called a Shuffle. A shuffle is the most resource-intensive operation in Apache Spark because it involves:

• Network serialization and socket transmission between worker nodes.
• Disk I/O (writing intermediate shuffle files on map nodes and reading them on reduce nodes).
• Memory allocation for sort and hash aggregation buffers.

Stage Boundaries in the Execution DAG

Whenever Spark encounters a wide dependency, the DAG Scheduler breaks the execution plan into distinct stages:

• Parent Stage (Map Stage): Executes all upstream narrow transformations up to the wide dependency, then writes shuffle files to local disk.
• Stage Boundary: The execution barrier where the map stage must finish writing all shuffle blocks before the child stage can begin processing.
• Child Stage (Reduce Stage): Fetches the shuffle blocks across the network from the map executors, merges the data by key, and executes downstream transformations.

Common Wide Transformations

• groupBy(...) / groupByKey() / reduceByKey(...): Key-based aggregations.
• join(...): Full sort-merge joins and shuffle hash joins between two distributed datasets.
• distinct(): Deduplication requiring an all-to-all key comparison.
• repartition(n): Full cluster redistribution to increase or rebalance partitions.
• cube(...) / rollup(...): Multi-dimensional aggregations.
• orderBy(...) / sort(...): Global sorting across all partitions.

---

3. The Physical Shuffle Mechanism: Write and Read

The shuffle process operates in two distinct phases: Shuffle Write on the producer side and Shuffle Read on the consumer side.

Phase 1: Shuffle Write (Map Side)

During the map stage, each task processes a slice of input data and produces key-value pairs designated for downstream partitions:

1. In-Memory Buffering: Records are passed into an internal partition buffer in execution memory (PartitionedAppendOnlyMap or ShuffleExternalSorter).
2. Hash Partitioning: For each record, Spark applies a hash function (typically MurmurHash3) to the partition key modulo the target number of shuffle partitions: target_partition = hash(key) % num_partitions.
3. Spilling to Disk: If the records exceed available execution memory, Spark sorts the records in memory by partition ID and key, serializes them, and writes a temporary spill file to the executor's local scratch disk.
4. Index and Data File Consolidation: Once the map task finishes, all in-memory buffers and intermediate spill files are merged into a consolidated data file plus an index in the standard sort-based layout (checksum files and other shuffle implementations can add or change files) on the executor's local storage:
   • Data File (shuffle_<shuffleId>_<mapId>_0.data): Contains all rows written by this map task, arranged contiguously in blocks sorted by target child partition ID.
   • Index File (shuffle_<shuffleId>_<mapId>_0.index): Stores the precise byte offsets and lengths of each partition block within the data file.

Map Task 0 Output on Local Disk:
┌────────────────────────────────────────────────────────┐
│ shuffle_0_0_0.data                                     │
│ [ Partition 0 Bytes ] [ Partition 1 Bytes ] [ Part 2 ] │
└────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────┐
│ shuffle_0_0_0.index                                    │
│ Offset 0: 0-1024B | Offset 1: 1024-3072B | ...         │
└────────────────────────────────────────────────────────┘

This two-file consolidation design prevents executors from creating millions of individual files on the operating system filesystem, which would quickly exhaust filesystem inodes.

Phase 2: Shuffle Read (Reduce Side)

In the reduce stage, each child task is responsible for a single target partition ID (e.g., Partition 4):

1. Block Location Lookup: The reduce task requests block locations from the Driver's MapOutputTrackerMaster.
2. Network Fetching: The reduce task contacts each map executor (or the node's standalone External Shuffle Service) over Netty-based RPC. It requests only its designated byte segment using the byte offsets in the index file.
3. Deserialization and Merging: As remote byte streams arrive, they are deserialized and loaded into execution memory. Depending on the operation, Spark merges values into an aggregation map or performs an external sort-merge.

---

4. Shuffle Bottlenecks and Performance Impacts

Because shuffles combine almost every physical hardware resource, bottlenecks can manifest in several areas:

---

5. Shuffle Partition Configuration: spark.sql.shuffle.partitions

The configuration property spark.sql.shuffle.partitions governs the number of partitions created when shuffling data for DataFrame queries (joins, aggregations, distinct). Its default value is 200.

The Impact of 200 Partitions on Small vs. Large Data

1. Small Datasets (< 1 GB):
   • 200 partitions creates hundreds of microscopic partitions (often a few kilobytes each).
   • Scheduling 200 tasks incurs more driver scheduling, task launching, and metadata overhead than the actual calculation time.
   • Tasks finish in single-digit milliseconds, yet the job takes several seconds.
2. Very Large Datasets (> 500 GB):
   • Dividing 500 GB across 200 partitions yields 2.5 GB per partition.
   • Because individual partition sizes exceed standard executor task memory slots, tasks must repeatedly spill to disk, causing severe I/O degradation and potential OOM crashes.

Sizing Rules of Thumb

• Illustrative Sizing Starting Point: Test 100 MB to 200 MB per partition, then measure task memory, skew, runtime, and concurrency; this is not an official universal optimum.
• Calculation Formula:

$$\text{Initial partition estimate} = \left\lceil\frac{\text{estimated shuffle bytes}}{150\text{ MB}}\right\rceil$$

• Cluster Parallelism: Consider enough partitions for available executor task slots (e.g., 2x to 4x total cores) so all CPU cores remain fully utilized across task waves.

Adaptive Query Execution (AQE)

In modern Spark (Spark 3.0+), Adaptive Query Execution dynamically adjusts shuffle partitions at runtime based on completed map-stage metrics:

# Enabling AQE for automatic partition coalescing post-shuffle
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "134217728")  # 128 MB

When enabled, Spark initially plans for spark.sql.shuffle.partitions, but after the map stage writes shuffle files, AQE inspects actual output sizes and automatically coalesces small adjacent shuffle partitions into single tasks.

---

6. Narrow vs. Wide Transformations Summary