3.5 Organizing Training Data on Cloud Storage and BigQuery
Key Takeaways
- Managed datasets attach labels, splits, and annotation metadata to training data and carry lineage into the resulting model.
- Training data must be versioned or snapshotted, because a model trained on "yesterday's table" cannot be reproduced once that table changes.
- Speech and video add their own layout requirements: consistent sampling rates and codecs, sharded segment files, and annotation files that reference time offsets.
- Cloud Storage layout should be shallow and shard-count aligned to worker count, with data co-located in the training region.
- BigQuery is the source of truth for tabular training data; export a snapshot to Parquet or TFRecord when a training loop needs high-throughput sequential reads.
3.5 Organizing Training Data on Cloud Storage and BigQuery
Blueprint reference: Section 3.2, "Organizing training data (e.g., tabular, text, speech, images, and videos) on Google Cloud (e.g., Cloud Storage and BigQuery)."
Chapter 2 covered organizing data for exploration. This bullet is about organizing it for training, and the additional requirements are reproducibility, labelling, and per-modality layout.
Managed Datasets
A managed dataset on the Agent Platform is a first-class resource that binds together the data location, the annotation schema, the labels, and the train/validation/test split. Its advantages over pointing a training job at a bucket path:
- Split management. The split is recorded on the dataset rather than reimplemented in each training script, so every run uses the same partition.
- Annotation storage. Labels live with the dataset instead of in an ad hoc sidecar file.
- Labelling integration. Unlabelled items can be routed to a labelling workflow and the results land back in the dataset.
- Lineage. Models trained from a managed dataset record that provenance, which is what an audit asks for.
Custom training does not require a managed dataset — a script can read Cloud Storage or BigQuery directly — but AutoML does, and using one is the difference between "we think this model was trained on the March extract" and knowing it.
Reproducibility: Snapshot or Version the Data
The most common reproducibility failure has nothing to do with seeds. A model is trained from SELECT * FROM events, the table receives new rows the next day, and the run can never be reproduced.
Fixes, in increasing order of rigour:
| Technique | How | Notes |
|---|---|---|
| Recorded query with a time bound | WHERE event_ts < '2026-09-01' | Cheap; relies on the table being append-only |
| BigQuery snapshot table | CREATE SNAPSHOT TABLE | Point-in-time, storage-efficient, immutable |
| Time travel | FOR SYSTEM_TIME AS OF | Limited retention window |
| Materialized training extract | Write the exact rows to a dated table or Cloud Storage prefix | Most durable; also serves as the training input |
| Object versioning on the bucket | Bucket-level versioning | Protects against overwrite and deletion |
Record the dataset identifier or snapshot URI as a parameter of the training run in Experiments, so the artifact and its data are linked.
Per-Modality Layout
Tabular. BigQuery is the source of truth. For training loops that need high-throughput sequential reads, export a snapshot to Parquet (cross-framework) or TFRecord (TensorFlow), or read directly through the BigQuery Storage Read API, which streams Arrow batches in parallel and avoids materializing an export.
Text. JSONL with one example per line is the common interchange format; large corpora shard into files of roughly 100 MB or more. Keep a BigQuery metadata table with document identifier, source, language, length, and split so that filtering is a query rather than a directory walk.
Images. Original files in Cloud Storage; a shallow prefix layout; a metadata/annotation table or JSONL manifest holding URI, label, bounding boxes or masks, and split. For training, consolidate into sharded TFRecord or WebDataset shards to remove per-file request overhead.
Speech / audio. Two extra constraints appear here and are easy to overlook:
- Consistent sampling rate and encoding. Mixing 8 kHz telephony audio with 44.1 kHz studio recordings without resampling produces a model that fails on whichever domain is underrepresented. Normalize at ingestion, and record the original rate in metadata.
- Segment-level annotations. Transcripts and labels reference time offsets within a file. Store them alongside the audio URI with start and end times, and keep the segmentation deterministic so a re-run reproduces the same examples.
Long recordings are usually pre-segmented into utterance-level clips and packed into shards; leaving hour-long files whole forces every worker to read far more than it uses.
Video. The heaviest modality and the one where layout decisions matter most:
- Decode once. Extracting frames on every epoch is enormously wasteful. Pre-extract frames or clip segments at the required rate and store those.
- Clip-level shards. Store fixed-length clips as records rather than whole films, so a worker reads only what it trains on.
- Codec and resolution consistency. Normalize to one codec and resolution at ingestion.
- Annotations with time ranges. Action labels and object tracks reference frame or time ranges; keep them in a queryable table joined by video identifier.
Layout Rules That Apply to Everything
- Shallow prefixes. Cloud Storage has a flat namespace; deep hierarchies make listing slow.
- Shard count aligned to worker count. Aim for a multiple of the worker count so shards divide evenly, with enough shards to interleave for randomness.
- Shard size in the hundreds of megabytes. Small enough to parallelize, large enough that request overhead is negligible.
- Region co-location. Bucket, BigQuery dataset, and training job in the same region. Cross-region reads add latency and egress cost to every step.
- Separate raw from processed. Keep an immutable raw zone and a derived processed zone so transformations can be re-run.
Exam Traps
- Training from a live table without a snapshot, then being unable to reproduce the run.
- Mixed audio sampling rates left unnormalized.
- Decoding video frames every epoch instead of pre-extracting.
- Whole-file records where segment-level records are needed.
- A bucket in a different region from the training job.
A speech team trains on a corpus mixing 8 kHz call-centre recordings with 44.1 kHz studio audio, storing files as uploaded. Word error rate is acceptable on studio audio and poor on call recordings. What is the most likely organizational cause?
An auditor asks the team to reproduce a model trained six weeks ago. The training script ran SELECT * FROM analytics.events with no time bound, and the table receives new rows continuously. What should have been done?
A video action-recognition team stores 12,000 full-length videos as single MP4 files and decodes frames on the fly each epoch. Training is extremely slow and GPU utilization is low. What is the most effective reorganization?
A team wants labels, train/validation/test splits, and annotation metadata to be managed as one resource, with the resulting model recording which data it was trained from. What should they use?