3.6 Ingesting Structured and Unstructured Data into Training Pipelines

3.6 Ingesting Structured and Unstructured Data into Training Pipelines

Blueprint reference: Section 3.2, "Ingesting structured and unstructured data from various sources into training pipelines."

Ingestion is where a large share of production ML defects originate, because it is the point at which data changes shape, arrives twice, or arrives late. The exam tests both the service selection and the correctness concerns.

Choosing the Ingestion Path

Why Dataflow Keeps Appearing

The reason Dataflow dominates this area on the exam is a specific property: the same Apache Beam pipeline runs in batch and in streaming mode. That matters for ML because the transformation applied to historical training data and the transformation applied to live serving data can be the same code, which eliminates a whole class of training-serving skew.

Beam gives you:

• Unified batch and streaming semantics, including windowing.
• Watermarks and triggers for handling late and out-of-order data explicitly.
• Autoscaling and dynamic work rebalancing so a straggling shard does not stall the job.
• Exactly-once processing semantics within the pipeline.

When a scenario says "the same preprocessing must apply to the nightly training extract and to the live stream," Dataflow is the intended answer.

Reading BigQuery into Training Efficiently

Training loops that pull from BigQuery through the standard query API are slow and can hit result-size limits. The BigQuery Storage Read API streams table data directly to readers as Arrow record batches over multiple parallel streams, which maps naturally onto multiple training workers each taking a stream.

# Each worker reads its own stream; no export to Cloud Storage required
from google.cloud import bigquery_storage
client = bigquery_storage.BigQueryReadClient()
session = client.create_read_session(
    parent=f"projects/{project}",
    read_session={"table": table_path,
                  "data_format": bigquery_storage.DataFormat.ARROW},
    max_stream_count=num_workers,
)

The alternative — export to Parquet or TFRecord in Cloud Storage, then read the files — is preferable when the same extract will be read many times across many runs, because the export cost is paid once and sequential file reads are extremely fast.

Correctness Concerns

Idempotency and deduplication. Pub/Sub delivers at least once. A publisher retry, a subscriber redelivery, or a pipeline restart will produce duplicate messages. Duplicates in training data over-weight the duplicated examples and inflate any evaluation that shares them. The standard defences:

• A stable message-level identifier carried in an attribute, deduplicated in a window.
• Idempotent writes keyed on a natural business key, so a repeat write overwrites rather than appends.
• The BigQuery Storage Write API with stream offsets for exactly-once ingestion.

Late and out-of-order events. In a streaming pipeline, an event's processing time is not its event time. Aggregating on arrival produces aggregates that quietly omit late data. Beam's watermarks track how complete the event-time view is, windows define the aggregation boundaries, and triggers and allowed lateness define what happens when data arrives after the window closes. For ML, late data matters because a feature computed as "transactions in the last hour" must be reproducible offline; if the streaming version silently dropped late events and the batch version included them, the two disagree.

Schema evolution. Sources add and rename fields. Defences:

• Use formats that carry a schema: Avro (designed for evolution), Parquet, or Protocol Buffers.
• Validate incoming data against an expected schema at the pipeline boundary and route violations to a dead-letter destination rather than failing the job or, worse, silently coercing.
• Version the schema and record which version each training run consumed.

Dead-letter handling. Any record that cannot be parsed or validated should be written to a dead-letter table or bucket with the error and the raw payload. A pipeline that drops bad records silently is indistinguishable from one that is working, until a model degrades for reasons nobody can trace.

PII at ingestion. Inspect and de-identify at the boundary rather than after the data has landed in several places; this is where Sensitive Data Protection belongs in the flow.

Unstructured Ingestion

For images, audio, video, and documents, the pattern is consistent: land the bytes in Cloud Storage, land the metadata in BigQuery. Ingestion then means writing the object and inserting a metadata row transactionally enough that the catalogue does not diverge from the bucket. Object tables give the SQL view over what actually exists, which is the reconciliation tool when they do diverge.

Exam Traps

• Ignoring at-least-once delivery. Duplicates will occur; dedupe explicitly.
• Aggregating on processing time when event time is what matters.
• Failing the whole job on one bad record instead of dead-lettering it.
• Separate batch and streaming transformation code. Guarantees skew; use one Beam pipeline.
• Querying BigQuery row by row from a training loop instead of using the Storage Read API or an export.