1.3 Streaming Workloads with Structured Streaming

1.3 Streaming Workloads with Structured Streaming

Structured Streaming is the core engine for building fault-tolerant, scalable, end-to-end streaming applications in Databricks. The Professional exam assesses your ability to configure triggers, manage state with watermarks, and select appropriate output modes for different sink constraints. This section provides an exhaustive overview of the internal mechanisms that make Structured Streaming robust for production data engineering.

Structured Streaming Mechanics

Under the hood, Structured Streaming treats a live data stream as a table that is being continuously appended. By default, it operates using a micro-batch processing engine. The engine keeps track of offsets, ensuring exactly-once processing guarantees even when the application crashes or restarts.

Micro-Batch vs Continuous Processing

In default micro-batch mode, Spark groups incoming data into small batches, processes them using the Catalyst Optimizer, and writes out the results. This provides exactly-once fault tolerance and high throughput, with latencies typically around 100 milliseconds.

For applications requiring sub-millisecond latency, Spark introduced Continuous Processing mode. Instead of batching, it launches long-running tasks that continuously read, process, and write data. However, Continuous Processing has strict limitations: it currently only supports map-like operations (select, project, map) and guarantees at-least-once (not exactly-once) semantics. It does not support complex aggregations or stateful processing, which limits its use cases in complex data engineering pipelines.

Trigger Configurations

Triggers define how frequently Spark executes the micro-batches. Proper trigger configuration is critical for balancing latency, cost, and throughput.

• Trigger.ProcessingTime('X seconds'): Fires a micro-batch at regular intervals. This is the most common trigger for continuous stream processing.
• Trigger.Once(): Processes a single micro-batch of all available data and then stops the query. This is useful for running daily batch jobs using streaming checkpointing logic, but can cause OutOfMemory errors if the backlog of data is too large.
• Trigger.AvailableNow(): Introduced in Spark 3.3, this replaces Trigger.Once(). Instead of processing all data in a single massive batch, AvailableNow processes all available data across multiple discrete micro-batches until the backlog is cleared, and then shuts down the cluster. This is the recommended approach for cost-effective, incremental batch processing because it inherently avoids memory overloads while providing the same job-terminating behavior.

# Example of using AvailableNow
df.writeStream \
  .format("delta") \
  .trigger(availableNow=True) \
  .option("checkpointLocation", "/path/to/checkpoint") \
  .start("/path/to/table")

State Management and Watermarking

When performing stateful operations like windowed aggregations or stream-stream joins, Spark must maintain intermediate state in memory (typically backed by RocksDB in Databricks). If the stream runs indefinitely, this state will grow unbounded, eventually crashing the application.

Understanding Watermarks

A watermark is a threshold defined by the withWatermark(event_time_column, 'delay_threshold') function. It tells Spark how late data is allowed to arrive relative to the maximum event time seen so far.

For example, if the max event time seen is 12:00 PM and the watermark delay is 10 minutes, the watermark threshold is 11:50 AM. If a record arrives with an event time of 11:49 AM, it is considered late and is dropped.

Crucially, the watermark allows Spark to safely clear old state from memory. Once the watermark surpasses a specific time window, Spark knows no more data will be accepted for that window, outputs the final aggregate result, and purges the state.

from pyspark.sql.functions import window

# Watermarking example
streaming_df.withWatermark("event_time", "10 minutes") \
  .groupBy(window("event_time", "5 minutes")) \
  .count()

Output Modes

Output modes dictate what gets written to the external sink during each micro-batch. The choice of output mode is strictly constrained by the operations performed in the query and the capabilities of the sink.

• Append Mode: The default mode. Only new rows appended to the result table are written to the sink. This is strictly required when the downstream sink does not support updates (like writing to raw Parquet files). If you are performing stateful aggregations, Append mode requires a watermark; Spark will only output the aggregation once the watermark passes the window, guaranteeing it will not change.
• Update Mode: Only rows that were updated in the result table during the current micro-batch are written to the sink. This is commonly used for dashboards tracking rolling counts, writing to Delta tables that support upserts, or publishing to key-value stores.
• Complete Mode: The entire updated result table is written to the sink after every micro-batch. This is only supported for queries with aggregations and is typically used when writing to small, easily overwritten tables or dashboards.

Stream-Stream Joins

Joining two streaming dataframes is one of the most complex operations in Structured Streaming. Because data arrives independently on both streams, Spark must buffer records as state to wait for matching records from the other stream.

To prevent state from growing infinitely, stream-stream joins strictly require watermarking on both streams and a time-range constraint in the join condition.

For example, joining an 'ad_clicks' stream to an 'ad_impressions' stream requires a condition stating that the click must occur within 1 hour of the impression. This bounds the state memory requirement, allowing Spark to discard old impressions that are older than the 1-hour window.

# Stream-Stream Join Example
impressions_with_watermark = impressions.withWatermark("impression_time", "2 hours")
clicks_with_watermark = clicks.withWatermark("click_time", "3 hours")

# Join with time constraints
impressions_with_watermark.join(
  clicks_with_watermark,
  expr("""
    click_ad_id = impression_ad_id AND
    click_time >= impression_time AND
    click_time <= impression_time + interval 1 hour
  """)
)

Without these constraints, the job will fail with an AnalysisException, enforcing safe operational boundaries for your streaming architecture.