12.3 Event-Time Processing, Watermarking, and Windowing

In real-world streaming architectures, data records rarely arrive at a compute cluster in clean, sequential order. Network partitions, intermittent mobile device connectivity, and varying queuing delays mean events frequently arrive out of order, delayed by minutes or even days. If a stream processing system evaluated records based solely on the server clock time when the record was received, results would be non-deterministic, varying with every cluster restart or network spike.

To solve this challenge, Structured Streaming bases stateful calculations on Event-Time rather than processing time, using Windowing to group data into discrete temporal buckets and Watermarks to drop excessively late records while bounding state store memory consumption.

---

The Three Notions of Time

Understanding the distinction between time domains is a foundational concept on the Databricks Associate Developer exam:

1. Event-Time: The timestamp embedded within the data record at the moment the event physically occurred at the originating source (e.g., an IoT sensor measuring temperature, a user clicking a link on a smartphone, or a point-of-sale terminal swiping a credit card).
2. Ingestion-Time: The timestamp assigned to a record when it is initially ingested into a centralized messaging broker (e.g., the Kafka log append timestamp).
3. Processing-Time: The local wall-clock timestamp of the Spark executor node at the exact moment the task thread processes the micro-batch record.

+-------------------------------------------------------------------------+
|                         EVENT-TIME VS PROCESSING-TIME                   |
+-------------------------------------------------------------------------+

   10:00:00 AM               10:05:00 AM               10:14:00 AM
  [Event Occurs] --------> [Broker Ingest] --------> [Spark Processes]
         |                         |                         |
         v                         v                         v
    EVENT-TIME               INGESTION-TIME            PROCESSING-TIME
  (Inside payload)           (Kafka Broker)           (Executor Clock)

---

Windowing on Event-Time

Structured Streaming enables temporal aggregations using the pyspark.sql.functions.window() function. Windowing slices continuous streams into discrete time buckets against an event-time timestamp column.

1. Tumbling Windows (Non-Overlapping)

Tumbling windows are contiguous, fixed-duration time intervals that do not overlap. Every event belongs to exactly one window.

• Syntax: F.window(timeColumn, windowDuration)
• Example: A 10-minute tumbling window divides time into intervals: [10:00, 10:10), [10:10, 10:20), [10:20, 10:30).

from pyspark.sql import functions as F

# Tumbling 10-minute window aggregation
tumbling_counts_df = sensor_stream_df \
    .groupBy(
        F.window(F.col("event_time"), "10 minutes"),
        F.col("device_type")
    ) \
    .count()

2. Sliding Windows (Overlapping)

Sliding windows have a fixed duration but slide forward by a user-specified interval that is smaller than the window duration. Because the slide interval is smaller than the duration, windows overlap, and an individual event falls into multiple windows.

• Syntax: F.window(timeColumn, windowDuration, slideDuration)
• Example: A 10-minute window with a 5-minute slide interval produces windows: [10:00, 10:10), [10:05, 10:15), [10:10, 10:20).
• An event occurring at 10:07 falls into both [10:00, 10:10) and [10:05, 10:15).

# Sliding 10-minute window advancing every 5 minutes
sliding_counts_df = sensor_stream_df \
    .groupBy(
        F.window(F.col("event_time"), "10 minutes", "5 minutes"),
        F.col("device_type")
    ) \
    .agg(F.avg("temperature").alias("avg_temp"))

Schema of the Resulting Window Column

When F.window(...) is invoked in a groupBy(), Spark produces a struct column named window containing two TimestampType fields:

• window.start: The inclusive start timestamp of the window bucket.
• window.end: The exclusive end timestamp of the window bucket.

# Selecting window boundaries explicitly
formatted_df = sliding_counts_df.select(
    F.col("window.start").alias("window_start"),
    F.col("window.end").alias("window_end"),
    F.col("device_type"),
    F.col("avg_temp")
)

---

Handling Late Data with Watermarking

The State Store Memory Problem

In a streaming aggregation, Spark maintains intermediate running totals in an in-memory State Store across executors. Without a mechanism to finalize past windows, Spark would have to retain state for every historical time window dating back to the dawn of the application, because a late-arriving event might arrive tomorrow that belongs to last month's window. Over time, executor memory would inevitably be exhausted, crashing the stream with an OOM error.

What is a Watermark?

A Watermark is a moving temporal boundary that tracks how far behind the latest observed event-time Spark is willing to accept late data. It is defined using .withWatermark(timeColumn, delayThreshold).

$$\text{Watermark} = \max(\text{event\_time seen so far}) - \text{delayThreshold}$$

Latest observed event time: 12:45
Delay threshold: 15 minutes
Candidate watermark: 12:30
An event outside the delay threshold is not guaranteed inclusion.
For window aggregation, inspect the assigned window end and output mode.

How Watermarking Operates in Micro-Batches

1. Threshold Calculation: During micro-batch $N$, Spark scans all newly ingested records and finds the maximum event timestamp. It subtracts the delayThreshold from this maximum to determine the candidate watermark.
2. Enforcement in Subsequent Batches: The new watermark is established at the completion of micro-batch $N$ and strictly applied during micro-batch $N+1$.
3. Operator-specific late-data handling: A watermark is not an unconditional row filter. Window aggregation uses its window-based state/finalization rules; records beyond the threshold may or may not be included. The guarantee is one-sided: data within the delay threshold is not dropped as too late under the documented aggregation conditions.
4. State Store Cleanup: Once watermark >= window.end, Spark finalizes that window under the query's lateness policy. The engine removes the window's intermediate aggregate from the State Store, reclaiming executor memory.

# Defining a 10-minute delay watermark on event_time
watermarked_stream = sensor_stream_df \
    .withWatermark("event_time", "10 minutes") \
    .groupBy(
        F.window(F.col("event_time"), "1 hour"),
        F.col("sensor_id")
    ) \
    .count()

---

Output Modes and Watermarking Interaction

The output mode fundamentally alters when aggregated window results are delivered to the sink:

• In Update Mode: On every micro-batch, Spark emits updated aggregate values for any window whose data changed during that batch. Once the watermark passes window.end, the window is purged from the State Store, but no final record is emitted because the downstream system already received the latest updates.
• In Append Mode: Append emits finalized result rows that will not be revised by this query; physical exactly-once delivery depends on the sink. Therefore, in Append mode, Spark will not emit a window's aggregate row until the watermark passes the window end time (watermark >= window.end). Until that threshold is crossed, late data could arrive to alter the count, so emitting early would violate Append semantics.

---

Streaming Joins

Structured Streaming supports joining streaming DataFrames with static DataFrames or with other streaming DataFrames.

1. Stream-Static Joins

• Semantics: Joining a real-time streaming DataFrame with a static reference DataFrame (e.g., enriching an IoT telemetry stream with a static customer/device metadata table).
• State Management: Stateless on the stream side. Spark joins incoming stream partitions directly against the static DataFrame.
• Watermarking: Not required for stream-static inner joins.
• Caveat: Do not assume the static side is a permanently frozen snapshot or a guaranteed live dimension. Source type, cache, file listing and transaction-snapshot behavior determine refresh; define the required consistency and test the chosen source/runtime.

2. Stream-Stream Joins

• Semantics: Joining two independent streaming DataFrames (e.g., joining an ad_impressions stream with an ad_clicks stream to calculate click-through attribution).
• The State Retention Challenge: Both streams are continuously arriving. When an impression arrives, its matching click might not arrive for several minutes. Spark must buffer both streams in its state store to find matches.
• State-bounding pattern and outer-join requirements:
  1. Watermarks on Both Streams: Watermarks on both inputs plus an appropriate time range can bound both state stores. An inner join can run without them but may retain unlimited history.
  2. Time-Range Join Condition: To bound state, the join predicate should specify a temporal boundary linking the event timestamps of both streams (e.g., click.time >= impression.time AND click.time <= impression.time + INTERVAL 1 HOUR).
  3. Outer Join Constraints: In stream-stream left outer, right outer, or full outer joins, the required watermark side(s) and time bounds depend on which side can produce unmatched rows; consult the supported join matrix. These bounds are needed so Spark knows when a row will never receive a match and can safely emit a NULL-padded outer row.

Official reference

Watermark semantics — source checked October 16, 2026. The examples state their runtime assumptions.