11.2 Stream and Batch Processing: Dataflow & Dataproc

Stream and Batch Processing: Dataflow & Dataproc

Architectural Selection Principle: Selecting between Cloud Dataflow and Cloud Dataproc depends primarily on the software ecosystem, operational model, and pipeline semantics. Choose Cloud Dataflow for new, cloud-native pipelines requiring serverless operation, unified batch/stream processing via Apache Beam, advanced windowing, and zero infrastructure management. Choose Cloud Dataproc when migrating existing open-source big data ecosystems (Apache Spark, Hadoop, Hive, Presto, Flink), executing ephemeral job-scoped Spark workloads, or utilizing customized JVM configurations on cost-effective Spot VMs.

---

Cloud Dataflow: Architecture & Apache Beam Mechanics

Cloud Dataflow is a fully managed, serverless distributed execution framework for executing Apache Beam pipelines (written in Java, Python, Go, or SQL). Dataflow abstracts away all underlying compute infrastructure, handling provisioning, horizontal scaling, graph optimization, and fault tolerance automatically.

+-----------------------------------------------------------------------------------------+
|                             CLOUD DATAFLOW EXECUTION MODEL                              |
+-----------------------------------------------------------------------------------------+
|  PIPELINE DEFINITION  |  Apache Beam SDK (Java / Python / Go / Beam SQL)                |
|                       |  - Unified abstractions: PCollection (Data) & PTransform (Logic)|
+-----------------------+-----------------------------------------------------------------+
|  GRAPH OPTIMIZATION   |  Execution Graph Optimization & Pipeline Fusion                 |
|                       |  - Fuses adjacent DoFns into single stages to eliminate I/O     |
+-----------------------+-----------------------------------------------------------------+
|  DISTRIBUTED RUNTIME  |  Dynamic Work Rebalancing + Streaming Engine                    |
|                       |  - State & Shuffle offloaded to managed Google backend          |
+-----------------------+-----------------------------------------------------------------+
|  AUTOSCALING          |  Horizontal Worker Scaling + Dataflow Prime Vertical Scaling    |
+-----------------------------------------------------------------------------------------+

Core Apache Beam Abstractions

• Pipeline: Encapsulates the entire data processing workflow from data ingestion to transformation and output sinking.
• PCollection (Parallel Collection): Represents a distributed, multi-element dataset. A PCollection can be bounded (finite, fixed-size dataset for batch processing) or unbounded (continuous, infinite data stream for real-time processing).
• PTransform (Parallel Transform): Represents an operation that takes one or more PCollections as input, performs transformation logic (e.g., ParDo, GroupByKey, Combine, Filter), and outputs one or more PCollections.
• Pipeline Fusion: Before executing a pipeline, Dataflow optimizes the execution graph by fusing adjacent ParDo operations into single execution steps. This eliminates unnecessary intermediate data serialization, disk writes, and network roundtrips between worker VMs.
• Dynamic Work Rebalancing: In batch pipelines, uneven data distribution can cause certain worker nodes to become "stragglers," holding up pipeline completion. Dataflow continuously monitors worker progress in real time and dynamically splits remaining unprocessed work from lagging workers, redistributing it across idle workers to minimize overall execution time.

DYNAMIC WORK REBALANCING (STRAGGLER MITIGATION)

Worker 1 (Fast):   [ Finished Chunk A ] ---> [ IDLE ] <==================+
Worker 2 (Lagging): [ Processing Chunk B .............................. ] |
                                                    |
                        Dataflow Runtime Splits Work Automatically        |
                                                    v                     |
Worker 2 (Relieved):[ Processing Chunk B1 ]                               |
Worker 1 (Assisting):                     [ Takes Over Chunk B2 ] <-------+

Dataflow Streaming Engine & Exactly-Once Semantics

• Streaming Engine Architecture: Traditional stream runners store intermediate window state and shuffle data directly on worker VM disks. Dataflow Streaming Engine decouples state storage and shuffle processing from worker VMs, moving state into a dedicated, highly optimized Google Cloud backend service. This drastically reduces worker VM CPU and memory overhead, accelerates autoscaling responsiveness, and provides smoother throughput.
• End-to-End Exactly-Once Processing: Dataflow ensures that every record in an unbounded stream is processed exactly once by coordinating checkpoints, tracking message IDs, and integrating with transactional sinks (such as BigQuery Storage Write API with exactly-once stream semantics, Cloud Spanner, and Cloud Storage with metadata tracking).

---

Windowing, Watermarks & Handling Late Data

Real-time stream processing must resolve the fundamental disconnect between Event Time (when the event actually occurred on the client device) and Processing Time (when the event reaches the processing engine over the network).

EVENT TIME VS. PROCESSING TIME & WATERMARKS

Event Time Clock:    12:00 ---- 12:05 ---- 12:10 ---- 12:15 ---- 12:20
                       |          |          |          |
                       | (Network Delays)    |          |
                       v          v          v          v
Processing Time:     12:02 ---- 12:09 ---- 12:14 ---- 12:26
                                  ^
                   [ Watermark Advances to 12:05 ]
                   (System assumes all <= 12:05 events have arrived)

1. Windowing Strategies

Windows partition unbounded streams into finite temporal chunks for aggregation.

2. Watermarks, Allowed Lateness & Panes

• Watermarks: A watermark is a dynamically computed temporal threshold that represents the pipeline's confidence that all data up to event time $T$ has arrived. Once the watermark passes the end of a window, the window is evaluated and emitted.
• Allowed Lateness (withAllowedLateness): Mobile devices and IoT sensors frequently experience offline network disconnections, delivering data hours after the watermark has passed. Allowed lateness specifies how long Dataflow keeps window state open after the watermark has closed the window.
• Accumulating vs. Discarding Panes: When late data arrives for an already-emitted window:
  • Accumulating Mode: Dataflow re-emits the complete window total including both original and newly arrived late records (e.g., emits 100, then emits 105).
  • Discarding Mode: Dataflow emits only the incremental delta brought by the late data (e.g., emits 100, then emits +5).

WINDOW EMISSION & LATE DATA TIMELINE

Window [12:00 - 12:05) 
  |--- Normal Events Arrive ---> Watermark passes 12:05 ---> [ Pane 1 Emitted: Total = 100 ]
                                                                      |
  |--- (Window closed, but within Allowed Lateness = 10m) ------------+
  |--- Late Event (EventTime 12:03) arrives at ProcessingTime 12:12
  |                                                                   |
  +---> [ Accumulating Mode ] ----------------------------------------> [ Pane 2 Emitted: Total = 105 ]
  +---> [ Discarding Mode ]   ----------------------------------------> [ Pane 2 Emitted: Delta = +5 ]

---

Dataflow Prime: Next-Generation Serverless Optimization

Dataflow Prime enhances standard Cloud Dataflow with intelligent resource management and automated infrastructure optimization.

• Resource-Based Autoscaling: Traditional Dataflow scales by adding identical worker VMs (homogeneous scaling). Dataflow Prime provisions compute resources (CPU vs. Memory) dynamically per pipeline step. If a transform is memory-heavy (e.g., large cache lookup) but another is CPU-heavy (e.g., encryption), Prime assigns right-sized resources to individual stages without over-provisioning the entire cluster.
• Vertical Autoscaling for Memory: Automatically detects impending Out-Of-Memory (OOM) conditions during large aggregations and dynamically increases worker memory allocations on the fly, preventing costly pipeline crashes and retries.
• Right-Fitting Recommendations: Analyzes historical pipeline metrics in Cloud Logging/Monitoring to provide actionable machine configuration recommendations.

---

Cloud Dataproc: Architecture & Cluster Topologies

Cloud Dataproc is Google Cloud's fully managed service for open-source big data processing engines, including Apache Spark, Apache Hadoop, Hive, Presto/Trino, Pig, and Flink.

+-----------------------------------------------------------------------------------------+
|                             CLOUD DATAPROC CLUSTER TOPOLOGY                             |
+-----------------------------------------------------------------------------------------+
|  MASTER / PRIMARY WORKERS (Standard Persistent VMs)                                     |
|  - Runs HDFS NameNode, YARN ResourceManager, Spark Driver, Hive Metastore               |
|  - State & cluster metadata maintained; NEVER run on Spot VMs                           |
+-----------------------------------------------------------------------------------------+
|  PRIMARY WORKERS (Standard VMs)                                                         |
|  - Runs YARN NodeManager & HDFS DataNode; maintains block storage                       |
+-----------------------------------------------------------------------------------------+
|  SECONDARY WORKERS (Spot / Preemptible VMs)                                             |
|  - Compute-only (YARN NodeManagers); DO NOT store HDFS data                             |
|  - Scaled up/down aggressively; up to 80% cost reduction; zero HDFS corruption on loss |
+-----------------------------------------------------------------------------------------+
|  PERSISTENCE TIER: Cloud Storage (gs://data-lake-bucket) via Cloud Storage Connector    |
+-----------------------------------------------------------------------------------------+

Modern Dataproc Deployment Models

1. Ephemeral (Job-Scoped) Clusters:
   • Rather than maintaining an expensive 24/7 static Hadoop cluster, architects deploy an ephemeral cluster via Cloud SDK/Airflow, execute a single Spark/Hive job, and immediately delete the cluster upon job completion.
   • Rapid Provisioning: Dataproc clusters initialize in under 90 seconds, making ephemeral job execution operationally efficient.
   • Decoupled Storage: Persistent cluster storage is completely replaced by Google Cloud Storage (GCS) using the open-source Cloud Storage Connector (gs:// connector instead of hdfs://). Cluster termination causes zero data loss.
2. Primary vs. Secondary (Spot/Preemptible) Workers:
   • Primary Workers: Run both YARN NodeManagers and HDFS DataNodes. Primary workers must be standard Compute Engine instances to preserve cluster stability and HDFS replication integrity.
   • Secondary Workers (Compute-Only): Run only YARN NodeManagers and execute stateless Spark executors. They do not participate in HDFS storage. Secondary workers can be configured as Spot VMs (or Preemptible VMs). If Google Cloud reclaims Spot VMs, Dataproc reschedules lost executor tasks onto remaining nodes without risking HDFS data corruption or cluster failure.
3. Dataproc Serverless:
   • Enables developers to submit Spark batch workloads (PySpark, Spark SQL, Spark R, Java/Scala) directly without creating, configuring, sizing, or maintaining any cluster infrastructure.
   • Google Cloud provisions and auto-tunes Spark runtime containers on demand, billing strictly for the fractional CPU cores and memory utilized per second of job execution.
4. Dataproc on GKE:
   • Deploys Spark workloads directly onto existing Google Kubernetes Engine (GKE) clusters, unifying containerized microservices and big data batch processing on a single shared compute plane.

---

Cloud Dataflow vs. Cloud Dataproc Decision Matrix

---

Concrete Architectural Scenario: Connected Vehicle IoT Fleet Analytics

Scenario Profile

• Client: Automotive OEM collecting real-time telemetry (speed, engine temperature, GPS) from 2 million connected vehicles.
• Requirements:
  1. Ingest 500,000 telemetry messages/sec from Cloud Pub/Sub.
  2. Compute 1-minute moving average speeds and detect engine overheating alerts within 3 seconds of event occurrence.
  3. Handle delayed telemetry uploads from vehicles traveling through cellular dead zones (up to 30 minutes late).
  4. Train a daily fleet maintenance prediction model using an existing complex Apache Spark MLlib library.

[ 2M Connected Vehicles ] ---> [ Cloud Pub/Sub Topic ]
                                     |
                                     v
              [ Cloud Dataflow Pipeline (Streaming Engine) ]
              - Sliding Windows (1-min duration, 10s slide)
              - Watermarks + Allowed Lateness (30 minutes)
              - Dynamic Work Rebalancing & Exactly-Once Ingestion
                                     |
                  +------------------+------------------+
                  |                                     |
                  v                                     v
    [ BigQuery Real-Time Dashboard ]       [ Cloud Storage Data Lake (Parquet) ]
    (Sub-3s Overheating Anomaly Alerts)                 |
                                                        v
                                         [ Ephemeral Dataproc Cluster ]
                                         - Provisioned via Cloud Composer (Airflow)
                                         - Primary: 2 Standard VMs | Secondary: 50 Spot VMs
                                         - Runs Daily Spark MLlib Training Job
                                         - Deletes Cluster Immediately on Completion (<90s init)

Architecture Blueprint

1. Real-Time Stream Processing: Cloud Dataflow reads unbounded streams from Cloud Pub/Sub. Telemetry is grouped into Sliding Windows (1-minute window, 10-second slide) to calculate moving averages. To accommodate intermittent cellular connectivity, Dataflow sets allowedLateness(Duration.standardMinutes(30)) with accumulating panes.
2. Serverless Streaming Engine: Dataflow Streaming Engine offloads pipeline state, ensuring real-time anomaly detection alerts reach BigQuery and operations teams in under 3 seconds.
3. Batch Machine Learning Pipeline: Raw telemetry is persisted in Parquet format on Cloud Storage. A Cloud Composer (Managed Airflow) DAG provisions an ephemeral Cloud Dataproc cluster daily. The cluster uses 2 Standard Master/Worker VMs and 50 Spot VM Secondary Workers to execute the existing Spark MLlib model at minimal cost, terminating the cluster immediately upon completion.

---

[!IMPORTANT] Exam Watch:

1. If an exam scenario describes migrating an existing Apache Spark, Hadoop, Hive, or Presto codebase to Google Cloud with minimal engineering effort and the lowest operational disruption, Cloud Dataproc is the correct answer.
2. To minimize Dataproc batch processing costs, configure ephemeral clusters storing persistent data in Cloud Storage (gs://) and utilizing Secondary Workers on Spot / Preemptible VMs.
3. If the scenario demands a serverless, unified batch and stream pipeline with complex event-time windowing, watermarks, late data handling, or Dynamic Work Rebalancing, choose Cloud Dataflow (Apache Beam).