7.5 Streaming Watermarking, Triggers, & Output Modes

7.5 Streaming Watermarking, Triggers, & Output Modes

DP-750 Exam Focus: Master stateful stream processing semantics in Spark Structured Streaming. Understand how Watermarks (withWatermark()) handle late-arriving event-time data and bound state store memory, configure Stateful Stream-Stream Joins, evaluate Streaming Triggers (especially Trigger.AvailableNow()), and select appropriate Output Modes (Append, Complete, Update) based on aggregation and sink requirements.

---

1. Watermarking Mechanics & Late Data Handling

In real-time distributed data pipelines, network latency, device disconnections, and clock skews cause events to arrive out of order. If a streaming query performs stateful operations (such as tumbling window aggregations or stream-stream joins), Spark must buffer historical records in state memory (RocksDB) to ensure accurate calculations.

Without a mechanism to discard old records, the internal state store would grow infinitely over time, eventually causing worker memory exhaustion.

Watermarking defines an event-time threshold that instructs the streaming engine how long to wait for late-arriving data before finalizing window results and dropping older records from state memory.

+-----------------------------------------------------------------------------------+
|                             WATERMARK TIMELINE ENGINE                             |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|  Event Time ->  12:00      12:10      12:20      12:30      12:40      12:50    |
|                  |          |          |          |          |          |     |
|                  +----------+----------+----------+----------+----------+     |
|                                                              |                    |
|                                                   Max Event Time Seen = 12:45     |
|                                                              |                    |
|                                          [ Watermark Delay: 15 Minutes ]          |
|                                                              v                    |
|  Watermark Horizon = (12:45 - 15 min) ====================> 12:30                 |
|                                                                                   |
|  [ Arriving Event: 12:35 ]  --->  ACCEPTED (Event Time >= Watermark)              |
|  [ Arriving Event: 12:25 ]  --->  DROPPED  (Event Time < Watermark: Too Late)     |
+-----------------------------------------------------------------------------------+

The withWatermark() API

To define a watermark, call withWatermark() on a streaming DataFrame, specifying:

1. timeColumn: A timestamp column representing the event time generated by the source producer.
2. delayThreshold: The time duration string (e.g., "10 minutes", "2 hours") representing the maximum acceptable delay.

$\text{Watermark Threshold} = \max(\text{EventTime seen by engine}) - \text{DelayThreshold}$

from pyspark.sql.functions import col, window, count, avg

# Tumbling 10-minute window aggregation with a 15-minute watermark
df_windowed_metrics = (df_parsed
    .withWatermark("reading_time", "15 minutes")
    .groupBy(
        window(col("reading_time"), "10 minutes"),
        col("device_id")
    )
    .agg(
        count("*").alias("total_readings"),
        avg("temperature").alias("avg_temperature")
    ))

How Watermarking Operates

• In-Memory Retention: Any record with an event time greater than or equal to the current watermark is evaluated and updates intermediate window state in RocksDB.
• Late Data Rejection: Any record with an event time strictly less than the current watermark is classified as "too late" and is dropped immediately without updating state.
• State Store Eviction: Once the watermark moves past the end time of a time window (e.g., window 12:00 - 12:10 when the watermark reaches 12:11), the intermediate state for that window is purged from RocksDB, freeing cluster memory.

---

2. Stateful Stream-Stream Joins

Joining two independent real-time data streams (e.g., joining an impressions stream with a clicks stream) is a complex stateful operation. Because matching events from both streams arrive asynchronously with unpredictable delays, Spark must buffer unmatched records from both streams in the state store.

To prevent state memory from expanding indefinitely, both input streams must define watermarks and a time-range join condition.

                      STATEFUL STREAM-STREAM JOIN MATRIX

  Stream A: Impressions (Watermark: 2 Hours)
  Stream B: Clicks      (Watermark: 3 Hours)

  Join Condition:
  impressions.ad_id = clicks.ad_id AND
  click_time BETWEEN impression_time AND impression_time + INTERVAL 1 HOUR

  +-------------------------------------------------------------------+
  | State Store Buffer (RocksDB)                                      |
  | - Retains impression records for up to 1 hour waiting for click   |
  | - Evicts impression once click_time window passes watermark       |
  +-------------------------------------------------------------------+

# Stateful Stream-Stream Inner Join with Time Bounds and Watermarks
# 1. Define watermark on Impressions stream
impressions = (df_impressions
    .withWatermark("impression_time", "2 hours"))

# 2. Define watermark on Clicks stream
clicks = (df_clicks
    .withWatermark("click_time", "3 hours"))

# 3. Join with equality key AND event-time interval condition
joined_stream = impressions.join(
    clicks,
    expr("""
        impressions.ad_id = clicks.ad_id AND
        click_time >= impression_time AND
        click_time <= impression_time + interval 1 hour
    """),
    joinType="inner" # Inner, left_outer, right_outer supported
)

Exam Rule for Stream-Stream Joins: For outer joins (e.g., left_outer), watermarks and time-range join constraints are mandatory. Spark can only emit an unmatched left row (with nulls for the right) once the right watermark has advanced past the join time boundary.

---

3. Streaming Triggers: Continuous, Micro-Batch, & AvailableNow

The Trigger defines the timing of streaming computation—dictating whether Spark processes records in continuous sub-millisecond loops, periodic micro-batches, or one-time incremental batch sweeps.

The Power of Trigger.AvailableNow for DP-750

Trigger.AvailableNow (introduced in Apache Spark 3.3 / DBR 10.4 LTS) is the foundational trigger for cost-effective Lakehouse architecture:

• Batch Cost, Streaming Semantics: Allows data engineers to orchestrate streaming pipelines on a schedule (e.g., once every hour or night via Databricks Jobs), running on on-demand ephemeral compute clusters.
• Rate-Limiting Bounded Execution: Unlike legacy Trigger.Once (which pulled all backlogged data into one monolithic batch, frequently crashing worker nodes with OOM errors), Trigger.AvailableNow honors maxFilesPerTrigger or maxOffsetsPerTrigger, splitting backlogs into safe, manageable micro-batches before stopping.

---

4. Structured Streaming Output Modes

The Output Mode defines what data is written to the downstream sink (e.g., Delta table, Kafka topic, or console) during each micro-batch.

+-----------------------------------------------------------------------------------+
|                         OUTPUT MODES SEMANTIC BEHAVIOR                            |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|  [ OutputMode("append") ]                                                         |
|  - Default mode.                                                                  |
|  - Only newly finalized rows are emitted to the sink.                             |
|  - With watermarked window aggregations: emits row ONLY when window closes.       |
|  - Mandatory for file/Delta sinks in stateless streaming.                         |
|                                                                                   |
|  [ OutputMode("complete") ]                                                       |
|  - Entire updated result table is rewritten to sink on every micro-batch.         |
|  - Requires an aggregation query.                                                 |
|  - NOT supported on standard Delta file sinks without full overwrite.             |
|                                                                                   |
|  [ OutputMode("update") ]                                                         |
|  - Only rows that were modified or newly added in the current batch are emitted.  |
|  - If query has no aggregations, behaves identically to 'append'.                 |
|                                                                                   |
+-----------------------------------------------------------------------------------+

Detailed Output Mode Compatibility Matrix

Exam Trap: Attempting to write a streaming aggregation query in Append mode without defining a watermark throws an AnalysisException: Append output mode not supported when there are streaming aggregations on streaming DataFrames/DataSets without watermark. Spark cannot know when an aggregation window is final unless a watermark is declared.