10.1 Broadcast Variables, Accumulators, and Closures
Key Takeaways
Changing a normal Python variable inside worker tasks does not update the driver variable.
SparkContext.broadcast creates a read-only shared value for tasks; functions.broadcast is a distinct SQL join hint.
Only the driver reads an accumulator value; task code adds updates rather than using it as shared mutable state.
Accumulator updates inside transformations can be repeated after recomputation and are unsuitable for business-critical exactly-once totals.
Why ordinary variables are not shared
A distributed task runs in an executor process, often on a different machine from the driver. Spark serializes the function and referenced closure values needed by that task. Mutating a Python list or counter inside a worker does not mutate the original object in the driver's Python interpreter. Local-mode experiments can hide assumptions about placement, so reason from process boundaries rather than from one observed laptop run.
Suppose count starts at zero on the driver and a function increments it for each row. Task copies can each increment their own value while the driver's count stays zero. Even if one could send updates through an external service, retries could repeat them. A shared mutable global is not a valid replacement for DataFrame.agg or a deliberately designed stateful streaming operator.
Read-only broadcast variables
A broadcast variable distributes a read-only value for reuse by tasks rather than repeatedly embedding a large lookup in task closures. In classic PySpark, create it through SparkContext.broadcast and read its value in worker code.
sc = spark.sparkContext
rates = sc.broadcast({"standard": 1.0, "reduced": 0.5})
rows = sc.parallelize([("standard", 10.0), ("reduced", 10.0)], 2)
priced = rows.map(lambda row: row[1] * rates.value[row[0]])
# For this small example only:
print(sorted(priced.collect())) # [5.0, 10.0]
This example uses the classic RDD API to make the variable behavior visible. It is not supported through Spark Connect's SparkContext surface. A DataFrame built-in expression or a small relational lookup join may be a better production solution. A broadcast dictionary remains a local copy on each consuming process, so it must fit the relevant JVM/Python memory. Broadcasting a huge object is not a way to increase available memory.
Treat the broadcast contents as immutable. Modifying rates.value inside a worker does not consistently update the value on other workers or the driver and violates the intended contract. To change lookup policy, create a new broadcast and ensure tasks reference it deliberately. After use, unpersist() removes executor cached copies so they can be resent if needed; destroy() removes broadcast data and metadata and makes further use invalid. Do not destroy a value while tasks still need it.
Broadcast variable versus broadcast join hint
| API | Object returned/purpose | Consumer |
|---|---|---|
| spark.sparkContext.broadcast(python_value) | Broadcast wrapper with .value | Classic task closure |
| pyspark.sql.functions.broadcast(dataframe) | DataFrame with a broadcast strategy hint | SQL optimizer and supported join plan |
| df.cache() | Persisted distributed relation | Later DataFrame computations |
The join hint does not give the user a dictionary with .value. The shared-variable API does not automatically turn a large DataFrame join into a BroadcastHashJoin. A cache also differs: its rows remain partitioned, rather than every task receiving the whole relation. Choosing among these tools depends on whether the input is a Python lookup object, a relational join build side, or a reused distributed intermediate.
Accumulators collect task updates
An accumulator supports associative, commutative updates such as summation, with the driver reading the final value. In classic PySpark, a numeric example is sc.accumulator(0). Tasks can add; they should not read its current value as if it were synchronized global state.
invalid_count = sc.accumulator(0)
def inspect_value(value):
if value < 0:
invalid_count.add(1)
sc.parallelize([3, -1, 4, -2], 2).foreach(inspect_value)
print(invalid_count.value) # 2 for this completed action example
foreach is an action, so it requests execution before the driver reads the value. Merely placing accumulator.add inside a lazy transformation is not enough to run it. A cached relation can also change whether upstream task code runs on a later action. The counter therefore reflects executed task updates under the API's semantics, not a universally stable description of the source data.
Retries, recomputation and reliable counts
Spark documents action-only accumulator updates as applied once per task despite task retries. Updates inside transformations can be applied more than once if stages or tasks are recomputed. A sequence of separate actions can intentionally run the function again. Consequently, do not use an accumulator as an authoritative payment count, inventory ledger, or streaming customer balance.
For a business count of invalid DataFrame rows, express the validation as columns and aggregate the resulting dataset: df.filter(F.col("amount") < 0).count(). For a grouped summary, use groupBy/agg and persist the intended result through a suitable destination. For state across streaming batches, use a stateful operator and checkpointed state. These choices make the computation part of the distributed data plan rather than an incidental side effect of task execution.
Common scenario checks
If a question asks why a driver list is unchanged, inspect closure copying. If it asks how to reuse a small read-only Python lookup, consider a broadcast variable and its memory cost. If it asks for a diagnostic counter from tasks, an accumulator may fit, with retry limitations. If it asks to send an email or charge a card once per record, none of these variables supplies exactly-once external side effects.
Confirm the API environment before answering: Spark Connect omits SparkContext, so these classic shared-variable snippets require classic Spark. See Spark shared variables and closures.
A Python list defined on the driver is appended to inside distributed worker tasks. What should a developer expect?
All workers synchronously update the original driver list.
The driver receives each append exactly once automatically.
The list becomes a Spark table.
Task closure copies do not provide shared mutation of the original driver list.
Which statement distinguishes a broadcast variable from functions.broadcast(df)?
The variable distributes a read-only value to classic tasks; the function annotates a DataFrame for a SQL join strategy.
Both return a dictionary with .value.
The join hint updates all worker variables.
The variable guarantees exactly-once accumulator updates.
Sections you finish are checked off in the contents.