4.2 Batch vs Streaming Processing Trade-Offs and Architecture Selection

4.2 Batch vs Streaming Processing Trade-Offs and Architecture Selection

Exam Focus: The Professional Data Engineer exam rigorously tests your ability to evaluate the latency vs. cost curve, select between free BigQuery batch load jobs and the BigQuery Storage Write API, architect resilient ingestion buffers using Cloud Pub/Sub, and select the optimal execution framework among Dataflow Batch (with FlexRS), Dataflow Streaming (with Streaming Engine), and Cloud Data Fusion.

A foundational responsibility of the data engineer is matching analytical processing paradigms to business requirements. Choosing between batch, micro-batch, and continuous streaming is never merely a technical preference; it is an economic and operational commitment. Processing data in real time introduces non-linear cost escalations, complex state management, and stringent failure recovery requirements. A data architect must systematically justify why a workload requires continuous streaming rather than cost-effective scheduled batch execution.

---

1. Batch vs. Micro-Batch vs. Continuous Streaming: Evaluation Framework

Data pipelines process information across a spectrum of latency SLAs, processing semantics, and resource consumption profiles:

+-----------------------------------------------------------------------------------------+
|                         THE LATENCY VS. COST SPECTRUM                                   |
+-----------------------------------------------------------------------------------------+
| BATCH PROCESSING               MICRO-BATCH PROCESSING         CONTINUOUS STREAMING       |
| (Daily / Hourly Cadence)       (Seconds to Minutes Cadence)   (Sub-Second / Real-Time)   |
| - Latency: Hours to Days       - Latency: 10s to 5 mins       - Latency: Milliseconds    |
| - Ephemeral Compute Clusters   - Short-lived Tasks/Cloud Run  - Always-on Compute Engine |
| - 100% Resource Duty Cycle     - Periodic Buffer Ingestion    - Stateful Windows & Watermarks
| - Free BigQuery Load Jobs      - Storage Write API (Buffered) - Storage Write API (Committed)
| - Lowest Cost Profile ($)      - Moderate Cost Profile ($$)   - Highest Cost Profile ($$$)|
+-----------------------------------------------------------------------------------------+

The Three Processing Paradigms

1. Batch Processing (Bounded Datasets):
   • Data Semantics: Operates on complete, bounded datasets that are finalized and static at the time of execution (e.g., historical sales logs for the previous business day).
   • Execution Model: High-throughput, distributed table scans and shuffles. Compute resources are provisioned on demand, run to completion at maximum CPU utilization, and terminate immediately.
   • Failure Model: Deterministic replay. If a worker fails, the batch framework simply re-executes the failed task partition.
2. Micro-Batch Processing (Time-Sliced Windows):
   • Data Semantics: Accumulates unbounded continuous data into discrete time-bounded chunks (e.g., buffering incoming events for 60 seconds or accumulating 50 MB files before processing).
   • Execution Model: Short, recurring jobs triggered by Cloud Scheduler, Cloud Functions, or micro-batch engines (like Spark Streaming).
   • Trade-Off: Provides near-real-time freshness without the operational complexity and cost of maintaining continuous, always-on streaming state.
3. Continuous Streaming Processing (Unbounded Event Streams):
   • Data Semantics: Treats data as an infinite, continuous stream of events arriving at arbitrary times. Events may arrive out of order, delayed by network partitions, or duplicated.
   • Execution Model: Long-running, persistent compute workers (e.g., Apache Beam pipelines on Cloud Dataflow). Requires managing complex streaming constructs: event time vs. processing time, tumbling/sliding/session windows, triggers, and watermarks.
   • Failure Model: Stateful checkpointing. Engines continuously commit state snapshots to persistent storage to guarantee exactly-once processing upon worker failure.

The Non-Linear Latency vs. Cost Escalation Curve

As business requirements push latency targets down from hours to sub-second real time, infrastructure costs escalate exponentially:

• Compute Efficiency: Batch compute achieves near 100% duty cycle efficiency during execution. Streaming compute must maintain idle headroom capacity (typically running at 30–50% average CPU utilization) to absorb sudden event spikes without lagging.
• Pricing Structures: Batch loading into BigQuery from Cloud Storage is completely free of compute charges. In contrast, streaming data into BigQuery incurs dedicated ingestion fees per gigabyte processed.
• State Storage Overhead: Stateful windowing (such as computing a 30-day running average per customer in real time) requires storing gigabytes or terabytes of volatile pipeline state in memory or external persistent caches.

---

2. Compute Resource Sizing and Cost Optimization Models

Optimizing cloud data engineering spend requires matching compute topologies to the elasticity profile of the workload.

Batch Compute Optimization: Ephemeral Clusters, Spot VMs, and FlexRS

• Ephemeral Dataproc Clusters: Never maintain a permanent, running Apache Spark or Hadoop cluster for batch processing. Deploy ephemeral Dataproc clusters via Cloud Composer or workflow templates that spin up, execute a specific Spark job, and immediately delete the cluster upon completion. Cluster creation overhead is typically under 90 seconds.
• Spot VMs / Preemptible VMs: Ephemeral batch nodes can leverage Google Cloud Spot VMs, which offer a 60% to 91% discount compared to standard compute pricing. In Cloud Dataproc, secondary worker nodes can be configured as 100% Spot instances. If Google reclaims a secondary worker VM, the Spark master redistributes the lost task partitions to remaining nodes without job failure.
• Dataflow Flexible Resource Scheduling (FlexRS): For non-time-sensitive batch jobs that can complete within a flexible 6-hour execution window (such as overnight billing calculations), Dataflow FlexRS reduces compute costs by up to 40%. FlexRS orchestrates a cost-optimized combination of preemptible and standard VMs, delaying job startup until excess Google compute capacity becomes available:

  # Launch a Dataflow batch pipeline using FlexRS for cost reduction
  gcloud dataflow flex-template run overnight-transaction-aggregator \
      --template-file-gcs-location=gs://templates/aggregator.json \
      --region=us-central1 \
      --parameters flexrsGoal=COST_OPTIMIZED,workerMachineType=n2-standard-4
  

Streaming Compute Optimization: Autoscaling Headroom and Streaming Engine

• Baseline Provisioning vs. Autoscaling Lag: In streaming pipelines, CPU-based autoscaling cannot react instantaneously to sudden 10x traffic surges; spinning up new worker VMs takes 2 to 4 minutes, during which backlogs accumulate and watermarks stall. Architects must configure baseline worker allocations (--numWorkers) sized to comfortably handle ordinary peak hours, allowing autoscaling (--maxNumWorkers) to handle unexpected surges.
• Dataflow Streaming Engine: In legacy Dataflow worker architectures, pipeline execution state, window aggregation buffers, and shuffle operations were executed entirely on local persistent disks attached to the worker VMs. This introduced severe CPU and disk I/O bottlenecks. Streaming Engine decouples state storage and shuffle from worker VMs, offloading state execution to a dedicated, highly optimized Google Cloud backend service. This drastically reduces worker VM vCPU and memory sizing requirements, leading to smoother autoscaling and lower overall pipeline costs.

---

3. BigQuery Ingestion Architecture: Free Batch Loads vs. Storage Write API

BigQuery decouples compute (query slots) from storage (Capacitor columnar storage). Understanding how data enters BigQuery is one of the most frequently tested topics on the exam.

+-----------------------------------------------------------------------------------------+
|                         BIGQUERY INGESTION ARCHITECTURES                                |
+-----------------------------------------------------------------------------------------+
| PATH A: FREE BATCH LOAD JOBS (From Cloud Storage)                                       |
| Raw Files (Avro/Parquet) ──> Cloud Storage ──(bq load / Load Job)──> BigQuery Storage    |
| * Compute Cost: 100% FREE (Uses Google Shared Slot Pool)                                |
| * Quota: 1,500 loads/table/day; Latency: Minutes                                        |
+-----------------------------------------------------------------------------------------+
| PATH B: STORAGE WRITE API (Unified High-Performance Streaming)                          |
| Streaming Producers ──(gRPC Streams via Protobuf)──> BigQuery Capacitor Storage         |
| * Compute Cost: $0.025 per GB (2 TB Free/month)                                         |
| * Features: Exactly-once, Committed/Pending/Default Streams, Sub-second latency         |
+-----------------------------------------------------------------------------------------+
| PATH C: LEGACY STREAMING API (tabledata.insertAll - DEPRECATED)                         |
| Legacy Client ──(HTTP JSON Post)──> Streaming Buffer ──> Capacitor Storage              |
| * Compute Cost: $0.05 per GB (Double the cost of Storage Write API!)                    |
| * Limitations: At-least-once, best-effort deduplication, no multi-stream transactions   |
+-----------------------------------------------------------------------------------------+

BigQuery Free Batch Load Jobs

• Shared Multi-Tenant Compute: Loading data into BigQuery tables from Cloud Storage (or local files) using load jobs (bq load CLI, BigQuery Load Job API, or Cloud Console) is completely free of BigQuery compute charges. Google absorbs the compute cost using its shared multi-tenant slot pool.
• File Format Performance:
  1. Avro (Optimal): Apache Avro is the fastest and most efficient format for loading data into BigQuery. Because Avro is a row-oriented binary format with an embedded JSON schema, BigQuery reads blocks in parallel across hundreds of workers simultaneously without schema inference overhead.
  2. Parquet / ORC (Good): Columnar formats load efficiently but require parsing column footers across files, making them slightly slower to load than Avro (though optimal for querying directly in GCS via BigLake).
  3. CSV / JSON (Slowest): Plaintext formats require expensive character parsing, delimiter escaping, and type coercion. In uncompressed CSV/JSON, BigQuery can parallelize reads; however, gzipped CSV/JSON cannot be parallelized (a single thread must read the entire compressed file sequentially), resulting in severely degraded load performance.
• Quotas and Constraints: Standard tables allow up to 1,500 load jobs per table per day (averaging approximately one load per minute) and 100,000 load jobs per project per day. Attempting to execute batch loads every 5 seconds will rapidly exhaust table quotas.

Legacy Streaming API (tabledata.insertAll)

• Cost: Costs $0.010 per 200 MB ($0.05 per gigabyte).
• Limitations: Uses HTTP POST requests containing JSON strings. Guarantees only at-least-once delivery. Deduplication relies on the client passing an insertId, which BigQuery tracks in memory on a best-effort basis for only 1 minute. Architectural Guidance: Deprecated for new designs. Migrate all workloads to the Storage Write API.

Modern Standard: BigQuery Storage Write API

The Storage Write API is a unified, high-performance streaming ingestion API built on gRPC and Protocol Buffers (Protobuf) that streams data directly into BigQuery storage nodes.

• Cost: $0.025 per gigabyte (50% cheaper than legacy streaming), with the first 2 TB per month free.
• Architectural Stream Modes:
  1. Committed Stream: Provides exactly-once processing semantics. Records written to the stream are committed immediately and become instantly queryable. Ideal for mission-critical transactional event feeds where duplicate records cannot be tolerated.
  2. Default Stream: Provides at-least-once delivery with the lowest latency and highest throughput. Does not support manual commit operations; acts as a direct replacement for legacy streaming inserts at half the price.
  3. Pending Stream: Implements two-phase commit (2PC) semantics for batch and micro-batch pipelines. Workers write millions of records into multiple parallel pending streams in an uncommitted state. Once all workers finish, a single CommitWriteStream API call atomically commits all streams simultaneously. If any worker fails, the uncommitted streams are discarded, preventing partial data writes.
  4. Buffered Stream: Flushes records in micro-batches; provides commit offsets to guarantee exactly-once processing across pipeline restarts.

---

4. Ingestion Shock Absorbers: Absorbing Spiky and Bursty Workloads

Directly exposing ingestion compute workers or database endpoints to incoming client traffic during flash sales, breaking news events, or IoT telemetry bursts is a common architectural anti-pattern that leads to database connection exhaustion and dropped records.

Cloud Pub/Sub as an Elastic Ingestion Buffer

Google Cloud Pub/Sub serves as a fully managed, globally distributed asynchronous messaging backbone that decouples message producers from downstream processing pipelines:

• Infinite Horizontal Elasticity: Automatically scales to absorb hundreds of thousands of events per second without pre-provisioning or partition rebalancing.
• Durable Message Storage: Retains unacknowledged messages for up to 7 days by default (configurable up to 31 days). If a downstream Dataflow pipeline crashes or BigQuery encounters quota throttling, messages safely queue in Pub/Sub with zero data loss.
• Seek and Replayability: If a production pipeline deployment introduces a transformation bug that writes corrupted records to storage, engineers can roll back the pipeline code and use Pub/Sub's Seek feature to rewind the subscription's acknowledgement state to a specific timestamp, re-ingesting the historical message stream cleanly.

Serverless Pub/Sub to BigQuery Direct Subscriptions

Historically, landing streaming events from Pub/Sub into BigQuery required deploying and maintaining an Apache Beam streaming pipeline on Cloud Dataflow simply to read Pub/Sub messages and call BigQuery APIs. This introduced substantial VM compute costs, worker management overhead, and monitoring complexity.

• Pub/Sub BigQuery Subscriptions: Google Cloud now allows Pub/Sub topics to stream messages directly into BigQuery tables without deploying Dataflow or writing custom consumer code.
• Under the Hood: Google Cloud manages serverless workers that read Pub/Sub messages and write directly to BigQuery using the Storage Write API.
• Dead-Letter Topics: If a message payload fails schema validation or contains malformed JSON, it is automatically routed to a designated dead-letter topic for debugging, preventing pipeline stalls.
• When to Use: Use Pub/Sub BigQuery Subscriptions whenever streaming events require direct landing into BigQuery without complex in-flight enrichment, windowed aggregations, or cross-stream joins.

---

5. Technology Selection Matrix: Dataflow vs. Cloud Data Fusion vs. Dataproc

Choosing the appropriate compute engine for data ingestion and transformation requires balancing developer skillsets, pipeline complexity, and operational total cost of ownership (TCO):

+-----------------------------------------------------------------------------------------+
|                       PIPELINE ENGINE SELECTION LOGIC                                   |
+-----------------------------------------------------------------------------------------+
|                                                                                         |
|  Do you have existing Apache Spark / Hadoop / Flink codebases?                          |
|  ├── YES ──> CLOUD DATAPROC (Ephemeral Clusters / Dataproc Serverless)                 |
|  │                                                                                      |
|  └── NO ──> Does your team require a Visual Drag-and-Drop GUI (No-Code/Low-Code)?       |
|              ├── YES ──> CLOUD DATA FUSION (CDAP-based visual ETL; enterprise analysts) |
|              │                                                                          |
|              └── NO ──> Do you require unified Batch/Streaming with sub-second latency, |
|                         complex windowing, or auto-scaling Apache Beam pipelines?       |
|                          ├── YES ──> CLOUD DATAFLOW (Dataflow Batch with FlexRS or      |
|                          │           Dataflow Streaming with Streaming Engine)          |
|                          │                                                              |
|                          └── NO ──> Direct SQL transforms on raw data?                  |
|                                     └──> BIGQUERY ELT (dbt, Scheduled Queries, Procedures)
+-----------------------------------------------------------------------------------------+

1. Cloud Dataflow

• Underlying Engine: Fully managed execution runner for Apache Beam (supporting Java, Python, and Go).
• Core Strengths: Unified programming model for both batch and streaming. Serverless operations with automated dynamic work rebalancing (eliminating stragglers). Seamless integration with Streaming Engine and FlexRS.
• Best For: Complex streaming event processing, tumbling/sliding/session windows, out-of-order event handling with watermarks, and high-scale transformations.

2. Cloud Data Fusion

• Underlying Engine: Fully managed, visual data integration service built on the open-source CDAP (Cask Data Application Platform) ecosystem.
• Core Strengths: Graphical drag-and-drop web UI (Pipeline Studio), pre-built library of over 150+ connectors to transactional databases, SaaS platforms, and mainframes. Built-in data profiling and wrangling.
• Cost Profile: High fixed baseline cost. The Cloud Data Fusion instance runs 24/7 (Basic or Enterprise tier), and each pipeline execution spins up an underlying Cloud Dataproc cluster to execute the CDAP pipeline.
• Best For: Enterprise ETL developers, business analysts, and compliance teams migrating legacy ETL workflows (Informatica, Talend) who mandate visual pipeline development without writing custom code.

3. Cloud Dataproc

• Underlying Engine: Managed Apache Spark, Apache Hadoop, Presto, and Flink clusters.
• Core Strengths: Fast cluster provisioning (<90 seconds), seamless integration with Google Cloud Storage via the Cloud Storage Connector (gs:// replacing hdfs://).
• Best For: Migrating existing, mature on-premises Spark and Hadoop codebases to the cloud with zero code rewrites.

---

6. Concrete Exam Scenarios & Architecture Pitfalls