5.2 Choosing a Preprocessing Tool: BigQuery, Dataflow, Spark or Python

The exam guide lists choosing the right tool for data preprocessing based on scale and complexity, with examples: BigQuery (SQL), Dataflow, Apache Spark, and in-memory Python frameworks. Questions usually give you three signals: data size, batch or streaming, and what code or skills the team already has.

The Options at a Glance

Decision Rules

1. Is the data structured and already in BigQuery? Start with BigQuery SQL, or BigQuery DataFrames if the team prefers pandas syntax. Moving terabytes out of BigQuery to preprocess them elsewhere is a common wrong answer.
2. Is the data streaming, or must the same logic run on batch and stream? Use Dataflow. Apache Beam lets one pipeline definition handle both, which helps keep features consistent between training (batch backfill) and real-time serving.
3. Does the organization have a large existing Spark codebase? Use Managed Service for Apache Spark and reuse the code instead of rewriting it.
4. Is the data small enough for memory and the logic heavily custom? Use in-memory Python in a notebook or a single training job.
5. Do you need distributed Python across both data processing and training? Consider Ray on Agent Platform (Chapter 16).

Dataflow ML Details Worth Knowing

• MLTransform wraps several preprocessing operations in one class: computing vocabularies, scaling, and generating embeddings with Agent Platform or Hugging Face models. It needs the Apache Beam Python SDK 2.53.0 or later and default windowing. Transforms built on TensorFlow Transform (TFT) run in batch pipelines.
• Full-pass transforms (such as scaling with the dataset mean and standard deviation) compute statistics once and save artifacts that can be reapplied later. That's how you avoid recomputing different statistics at serving time.
• RunInference runs local models (PyTorch, scikit-learn, TensorFlow) or calls a remote Agent Platform endpoint from inside a pipeline, for batch or streaming scoring.

Worked Scenarios

Keeping Preprocessing Consistent

Choosing a tool isn't only about speed. The exam links preprocessing to training-serving skew:

• Put transformations where both training and serving can reuse them: BigQuery ML TRANSFORM, saved transformation artifacts, a shared feature pipeline feeding Feature Store, or preprocessing packaged in the serving container (Chapter 13).
• Avoid writing the same feature logic twice in two languages, such as SQL for training and Java for serving.
• Version preprocessing code alongside the model and record it in pipeline metadata.

Common Wrong Answers

• "Export BigQuery to CSV, then preprocess with pandas" for terabyte-scale data. It is slow, fails on memory, and adds a security risk from copied data.
• "Rewrite existing Spark jobs in Beam" when the requirement is minimal change. Reuse Spark on the managed Spark service.
• "Use scheduled SQL for real-time features" when the scenario needs seconds-level freshness. That calls for a streaming pipeline.
• "Run a long-lived self-managed cluster" when a serverless option meets the need. The exam favors managed, Google Cloud-native services when they fit.

Cost and Operations Considerations

• Serverless first: BigQuery and Dataflow scale automatically and bill for what you use. Serverless Spark removes cluster management.
• Ephemeral clusters: if you run Spark clusters, create them per job and delete them afterward, and consider Spot VMs for fault-tolerant work.
• Data locality: process data in the region where it's stored.
• Orchestrate preprocessing as a pipeline step (Agent Platform Pipelines components exist for BigQuery, Dataflow, and Managed Spark jobs) so runs are repeatable and tracked.