2.2 Custom & Stateful Streaming Operations

Structured Streaming in PySpark allows for near real-time data processing. While stateless transformations (like map, filter, or standard selects) process each micro-batch independently, many complex applications require stateful processing. Stateful streaming tracks data across micro-batches for operations like aggregations, deduplication, stream-stream joins, and complex event processing.

Stateful Stream Processing Mechanics

When a stream performs a stateful operation (e.g., counting events per user over a tumbling window), Spark must remember previous data. This "memory" is stored in a State Store. During each micro-batch, Spark reads the previous state from the store, updates it with new incoming data, and writes the updated state back to the store.

Because state can grow indefinitely, stateful streams are highly susceptible to Out-Of-Memory (OOM) errors. Spark uses watermarks to bound state size. A watermark defines how late data can be before it is ignored. Once the watermark passes a certain time window, Spark drops the state for that window, freeing up resources. The watermark is calculated as the maximum event time seen so far minus the specified watermark delay.

State Store Configuration

By default, Spark uses an HDFS-backed state store (technically, in-memory state that is checkpointed to distributed storage). The default provider keeps state entirely in the executor's JVM memory. This is fast but risky for workloads with massive state (e.g., tracking millions of unique user sessions over a 30-day window). Memory constraints often lead to garbage collection pauses and application crashes.

For large-scale production workloads, the RocksDB state store is highly recommended. RocksDB maintains state locally on the executor's disk (spilling out of memory) while continuing to checkpoint to distributed storage for fault tolerance. RocksDB is an embeddable persistent key-value store optimized for fast storage environments like SSDs.

To enable RocksDB, configure the Spark session:

spark.conf.set(
    "spark.sql.streaming.stateStore.providerClass",
    "org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider"
)

This single configuration change can stabilize streaming applications that frequently crash due to memory pressure from large state. RocksDB handles state management much more effectively by utilizing disk space when memory limits are approached. It inherently supports versioning and snapshots, allowing Spark to reliably recover state during micro-batch failures.

Stream-Stream Joins

Joining two streaming DataFrames is one of the most complex operations in Structured Streaming. Because data on both sides is continuously arriving, Spark must buffer records from both streams in state until matching records arrive. If unconstrained, the state for both streams would grow indefinitely.

To prevent the state from growing infinitely, a stream-stream join must have three constraints:

1. Watermark on Stream A: Defines how late records from Stream A can arrive.
2. Watermark on Stream B: Defines how late records from Stream B can arrive.
3. Time Range Constraint in the Join Condition: Specifies the temporal relationship between the two streams (e.g., an ad click must happen within 1 hour of the ad impression).

Example:

impressions_with_watermark = impressions_df \
    .withWatermark("impressionTime", "2 hours")

clicks_with_watermark = clicks_df \
    .withWatermark("clickTime", "3 hours")

joined_df = impressions_with_watermark.join(
    clicks_with_watermark,
    expr("""
      clickAdId = impressionAdId AND
      clickTime >= impressionTime AND
      clickTime <= impressionTime + interval 1 hour
    """)
)

In this example, Spark knows exactly how long to hold an impression in state. If an impression arrives at 12:00 PM, a matching click must arrive between 12:00 PM and 1:00 PM. Considering the watermarks, Spark can safely discard the impression state after a definitive boundary, preventing unbounded state growth. The exact state cleanup delay is dynamically computed using the watermarks from both streams and the time range constraint.

Custom Stateful Processing: flatMapGroupsWithState

While SQL aggregations and joins are declarative, some use cases require custom business logic that doesn't fit standard SQL functions. For instance, you might need to track a complex user journey, implement custom sessionization, or emit alerts when a specific sequence of events occurs.

For these scenarios, PySpark provides applyInPandasWithState (the Python equivalent of the Scala mapGroupsWithState / flatMapGroupsWithState APIs).

This API allows you to define a Python function that takes:

1. A grouping key (e.g., user_id)
2. An iterator of new rows for that key
3. A state object that can be read, updated, or cleared

def process_user_events(key, pdf_iter, state):
    # Read previous state
    current_state = state.get()
    if current_state is None:
        current_state = {"login_count": 0, "last_event": None}
        
    # Process new events in pandas dataframe iterator
    for pdf in pdf_iter:
        current_state["login_count"] += len(pdf)
        
    # Update state
    state.update(current_state)
    
    # Yield output
    import pandas as pd
    yield pd.DataFrame({"user_id": [key[0]], "total_logins": [current_state["login_count"]]})

Crucially, this API allows you to configure Timeouts. You can set a processing time timeout or event time timeout. If no new data arrives for a key within the timeout period, your function is invoked one last time with an empty iterator, giving you a chance to emit a final record (e.g., "session closed") and clear the state using state.remove(). Properly clearing state is mandatory to avoid OOM errors in custom stateful streaming. The flexibility of this API allows data engineers to build sophisticated state machines directly within the streaming pipeline, managing complex workflows in real time. This level of granular control is essential when building enterprise-grade real-time systems where standard aggregations fall short and custom session management, deduplication, and anomaly detection are required. This level of granular control is essential when building enterprise-grade real-time systems where standard aggregations fall short and custom session management, deduplication, and anomaly detection are required. This level of granular control is essential when building enterprise-grade real-time systems where standard aggregations fall short and custom session management, deduplication, and anomaly detection are required.