6.3 Windowing Strategies in Streaming Dataflow: Fixed, Sliding, and Session Windows

6.3 Windowing Strategies in Streaming Dataflow: Fixed, Sliding, and Session Windows

Exam Focus: The Google Cloud Professional Data Engineer exam rigorously tests your ability to select the correct windowing strategy based on business requirements. You must understand the difference between event time and processing time, avoid memory exhaustion caused by misconfigured sliding windows, understand the dynamic merging mechanics of session windows, and anticipate how the default trigger handles window boundaries.

In stream data processing, an unbounded collection represents an infinite, continuously arriving sequence of records. Because an unbounded stream has no end, global operations—such as calculating a total sum, finding the minimum value, or counting records—cannot be performed across the entire stream at once without subdividing the data into finite, discrete temporal chunks. This temporal partitioning process is known as Windowing.

---

1. Time Semantics: Event Time vs. Processing Time

To understand windowing, data engineers must distinguish between two fundamental temporal domains:

+─────────────────────────────────────────────────────────────────────────────+
|                        TIME DOMAINS IN STREAMING                            |
+─────────────────────────────────────────────────────────────────────────────+
|                                                                             |
|   EVENT TIME: 12:01:05 PM                      PROCESSING TIME: 12:04:30 PM |
|   (Occurred on mobile device)                  (Arrived at Dataflow worker) |
|               │                                              ▲              |
|               │                                              │              |
|               └────────────[ Network Delay / Skew ]──────────┘              |
|                                (3m 25s Time Lag)                            |
+─────────────────────────────────────────────────────────────────────────────+

1. Event Time: The timestamp assigned to an event when it physically occurred on the originating client device, sensor, or operational database. This timestamp is embedded directly in the payload or message metadata.
2. Processing Time: The wall-clock time of the compute infrastructure (the Dataflow worker instance) at the precise instant the element is processed by an execution transform.

Why Event Time is Mandatory for Accurate Analytics

In real-world networks, events never arrive instantaneously or in perfect chronological order. Network latency, intermittent cellular connectivity, mobile device offline caching, and regional network partitions introduce unpredictable time skew between event time and processing time.

If an analytics pipeline calculates 5-minute sales metrics using processing time, a mobile transaction executed at 12:01 PM that is delayed by intermittent connectivity until 12:15 PM would be incorrectly attributed to the 12:15 PM window. Processing time yields non-deterministic results that change whenever a pipeline is paused, re-run, or backfilled. Event time provides deterministic, reproducible analytics regardless of network delays.

Timestamp Assignment in Beam

• Sources such as PubsubIO.readMessagesWithAttributes() automatically extract the Pub/Sub message publish timestamp as the Beam event timestamp.
• If the event timestamp is stored in a custom payload attribute (e.g., event_timestamp), developers use the WithTimestamps transform to assign the custom event timestamp to each record before windowing.

// Assigning custom event time from a JSON payload
PCollection<TelemetryEvent> timestampedEvents = rawEvents.apply(
    "AssignEventTime",
    WithTimestamps.of((TelemetryEvent event) -> Instant.ofEpochMilli(event.getDeviceTimestamp()))
);

---

2. Windowing Fundamentals on Unbounded Collections

In Apache Beam, every element in a PCollection is associated with an implicit timestamp and assigned to one or more windows.

The GlobalWindow (Default)

• By default, all elements in any newly created PCollection belong to a single, all-encompassing GlobalWindow spanning $[-\infty, +\infty)$.
• In a bounded (batch) pipeline, performing an aggregation on the GlobalWindow succeeds because the runner processes all data to completion and emits the final result when the batch source reaches end-of-file.
• In an unbounded (streaming) pipeline, performing a grouping or aggregation (such as GroupByKey or CombinePerKey) on the default GlobalWindow without configuring custom triggers will never emit any output. Because the stream never terminates, the global window never closes.
• To perform aggregations on an unbounded stream, developers must apply a non-global windowing strategy: Fixed Windows, Sliding Windows, or Session Windows.

---

3. Fixed (Tumbling) Windows

Fixed windows (also referred to as tumbling windows) segment the stream into continuous, non-overlapping, uniform time intervals.

Stream Timeline (Event Time) ─────────────────────────────────────────────────>
[ Window 1: 12:00 - 12:05 ) [ Window 2: 12:05 - 12:10 ) [ Window 3: 12:10 - 12:15 )
   • Event A (12:02)           • Event C (12:06)           • Event E (12:11)
   • Event B (12:04)           • Event D (12:09)           • Event F (12:14)

Architectural Characteristics

• Non-Overlapping: Windows are contiguous. An event belongs to exactly one window based on its event timestamp $t$.
• Formula: A window of duration $D$ starting at boundary $W_0$ contains all events with timestamp $t$ satisfying: $W_{\text{start}} \le t < W_{\text{start}} + D$.
• State Lifecycle: When the watermark advances past the window end, the window fires its aggregation result and its state is reclaimed from memory.
• Typical Use Cases: Hourly billing rollups, 5-minute average website latency dashboards, daily summaries, regular periodic metrics.

// Implementing a 5-minute Fixed Window in Apache Beam
PCollection<KV<String, Long>> fiveMinuteCounts = inputEvents
    .apply("FixedWindows", Window.into(FixedWindows.of(Duration.standardMinutes(5))))
    .apply("CountPerKey", Count.perKey());

---

4. Sliding (Hopping) Windows

Sliding windows (also known as hopping windows) represent overlapping, fixed-duration time intervals defined by two parameters: Window Duration and Slide Period.

Duration = 10 Minutes, Slide Period = 5 Minutes
Window 1: [ 12:00 ─────────────────────> 12:10 )
Window 2:           [ 12:05 ─────────────────────> 12:15 )
Window 3:                     [ 12:10 ─────────────────────> 12:20 )
                            ▲
                      Event X (12:07)
             (Belongs to BOTH Window 1 AND Window 2!)

Architectural Characteristics

• Overlapping Intervals: Because the slide period is smaller than the window duration ($S < D$), consecutive windows overlap in time.
• Multi-Window Assignment: Each incoming element is assigned to multiple concurrent windows simultaneously.
• Duplication Multiplier: The exact number of overlapping windows to which each element is assigned is given by:

$N = \frac{\text{Window Duration}}{\text{Slide Period}}$

The Memory Explosion Anti-Pattern

A frequent production mistake on streaming Dataflow pipelines is setting an excessively small slide period relative to the window duration.

• The Trap: An engineer specifies a 1-hour window sliding every 1 second ($D = 3600\text{s}, S = 1\text{s}$). In this scenario, $N = 3600 / 1 = 3,600$.
• Impact: Every single incoming event is duplicated across 3,600 separate in-memory window buffers. If the stream processes 10,000 events/sec, the pipeline attempts to write and aggregate 36,000,000 window states per second, rapidly crashing worker VMs or Streaming Engine backend storage with catastrophic Out-of-Memory exceptions.
• Best Practice: Keep the ratio $D / S$ modest (typically between $2$ and $20$, such as a 10-minute window sliding every 1 minute).

Typical Use Cases

• Moving averages (e.g., 15-minute moving average temperature updated every 1 minute).
• Rolling anomaly detection (e.g., alert if error rate over the last 30 minutes exceeds 5%, evaluated every 2 minutes).
• Trend smoothing in financial trading tickers.

// Implementing a 10-minute Sliding Window updated every 1 minute
PCollection<KV<String, Double>> movingAverage = sensorReadings
    .apply("SlidingWindows", Window.into(
        SlidingWindows.of(Duration.standardMinutes(10))
                      .every(Duration.standardMinutes(1))
    ))
    .apply("AveragePerSensor", Mean.perKey());

---

5. Session Windows

Session windows are dynamic, data-driven windows defined by periods of continuous activity separated by a specified Gap Duration of inactivity.

User 1: [ Event ]──[ Event ]──[ Event ] ──(Inactivity > 30m)──> [ Window Closes: Length 45m ]
User 2: [ Event ] ────────────────────────(Inactivity > 30m)──> [ Window Closes: Length 0m ]

Architectural Characteristics

• Strictly Scoped Per Key: Session windows do not apply globally across a dataset. They are scoped strictly per key (e.g., user_id or session_token). Different keys have completely independent window start times, durations, and end boundaries.
• Data-Driven Boundaries: Unlike fixed or sliding windows, session windows have no fixed calendar start or end points. They expand and contract based on incoming event intervals.
• Dynamic Window Merging Mechanics:
  1. When an event arrives at timestamp $t$, Beam creates an initial ephemeral window $[t, t + \text{gap})$.
  2. When a subsequent event arrives for the same key at timestamp $t_2$ (where $t_2 < t + \text{gap}$), its window $[t_2, t_2 + \text{gap})$ overlaps with the existing window.
  3. Apache Beam's windowing engine dynamically merges the two overlapping windows into a single expanded window spanning $[t, t_2 + \text{gap})$.
  4. Merging continues dynamically until a period of inactivity greater than or equal to the gap duration elapses, at which point the session window finalizes.

Typical Use Cases

• Tracking e-commerce user browsing sessions (e.g., group all user clicks until 30 minutes of inactivity, then calculate total session spend or abandoned carts).
• Mobile gaming play sessions.
• IoT connected vehicle trips (from engine start until parked for > 15 minutes).

// Implementing Session Windows with a 30-minute Gap Duration
PCollection<KV<String, Long>> sessionCounts = userClicks
    .apply("SessionWindows", Window.into(Sessions.withGapDuration(Duration.standardMinutes(30))))
    .apply("CountClicksPerSession", Count.perKey());

---

6. Single-Key vs. Global Aggregations & Trigger Interactions

Understanding how windowing interacts with keys and triggers is vital for exam scenario questions.

Keyed vs. Non-Keyed Window Aggregations

• Keyed PCollections (PCollection<KV<K, V>>): When windowing is applied to a keyed PCollection, window state and aggregations are managed independently per key. One key can fire its window while another key continues accumulating state. This enables horizontal parallelization across all cluster workers.
• Non-Keyed PCollections (PCollection<V>): Applying a global aggregation (such as Count.globally()) across a windowed stream forces all elements for that window across the entire stream onto a single worker node, creating an extreme performance bottleneck.

Default Trigger Behavior and Window Firing

• By default, Apache Beam attaches the Default Trigger to any windowed PCollection:

$\text{Default Trigger} = \text{Repeatedly}\left(\text{AfterWatermark}.\text{pastEndOfWindow}()\right)$

• Trigger Firing: The default trigger fires exactly once when the system watermark passes the end of the window ($W_{\text{end}}$). At this instant, the aggregated pane is emitted downstream to sinks.
• Late Data Handling under Default Trigger: If an event arrives with an event timestamp belonging to an already-closed window (because its event timestamp is earlier than the current watermark), the default trigger permanently drops the late event.
• Allowed Lateness: To retain and process late-arriving events, engineers must explicitly configure .withAllowedLateness(Duration) and specify accumulating or discarding pane accumulation modes.

---

7. Comparative Windowing Matrix & Production Scenarios

Realistic Exam Scenarios & Architecture Pitfalls