9.1 Organizing & Ingesting Training Data from Cloud Storage & BigQuery
Key Takeaways
- Serverless training jobs can read and write Cloud Storage buckets as local paths under the /gcs root through Cloud Storage FUSE.
- When a custom training job uses a managed dataset, Agent Platform sets AIP_DATA_FORMAT to jsonl, csv, or bigquery and passes split locations in AIP_TRAINING_DATA_URI, AIP_VALIDATION_DATA_URI, and AIP_TEST_DATA_URI.
- Sharded binary formats such as TFRecord, or columnar formats such as Parquet, load faster at scale than millions of small individual files.
- By default, a serverless training job can access Cloud Storage buckets in its own project through the Custom Code Service Agent; a custom service account narrows that access.
- Organize unstructured training data with import manifests (JSON Lines or CSV) that map Cloud Storage file URIs to labels.
Section 3.2 of the exam guide starts with organizing training data (tabular, text, speech, images, and videos) on Google Cloud (for example, Cloud Storage and BigQuery) and ingesting structured and unstructured data from various sources into training pipelines. Chapter 5 covered exploration and preprocessing. This section is about getting prepared data into training jobs efficiently and repeatably.
Organizing Training Data by Type
| Data type | Recommended layout | Label storage |
|---|---|---|
| Tabular | BigQuery tables partitioned by date, one row per example. Or Parquet/CSV exports in Cloud Storage for frameworks that read files | Label column in the same table |
| Text | Short text in BigQuery columns. Long documents as files in Cloud Storage. Tuning data as JSON Lines | Label column, or JSONL fields (for example, prompt and response) |
| Speech / audio | Audio files (WAV, FLAC) in Cloud Storage, organized by source and date | Manifest (JSONL or CSV) mapping gs:// URIs to transcripts or labels |
| Images | Image files in Cloud Storage | Import file (JSONL or CSV) mapping URIs to labels or bounding boxes |
| Video | Video files in Cloud Storage. Long videos split into segments when needed | Manifest with URIs, time segments, and labels |
Principles:
- Keep raw, processed, and training-ready data in separate, versioned locations, such as
gs://bucket/datasets/claims/v12/train/. - Never overwrite a dataset version a model was trained on. Lineage and audits depend on it (Chapter 6).
- Put data in the same region as training compute.
- Keep manifests (import files) as the source of truth for which files belong to which split and label.
File Formats for Scale
| Format | Strengths | Typical use |
|---|---|---|
| TFRecord | Sequential binary records, easy sharding, fast with tf.data | TensorFlow and JAX training on images or serialized examples |
| Parquet | Columnar, compressed, schema-aware | Large tabular data read with Spark, pandas, or PyArrow |
| Avro | Row-based with schema, good for streaming exports | Data pipeline interchange |
| CSV | Simple and human-readable | Small datasets and AutoML imports. Slower and schema-less at scale |
| JSON Lines | One JSON object per line | Gemini tuning and batch inference inputs, import manifests |
Sharding: split large datasets into many medium-sized shards, not one giant file and not millions of tiny ones, so distributed workers read in parallel. Shuffle across shards and within buffers.
Reading Data into Training Code
From Cloud Storage
- Cloud Storage FUSE: serverless training jobs see buckets under
/gcs/. For example,gs://example-bucket/data.csvis read as/gcs/example-bucket/data.csv. Data is streamed, not downloaded to every replica, which speeds startup and suits large files and distributed training. - Framework loaders:
tf.data.TFRecordDatasetwith parallel reads andprefetch, PyTorchDataLoaderwith multiple workers over FUSE paths, or WebDataset-style shard readers. - NFS shares for workloads that need POSIX semantics or very low-latency shared file access.
From BigQuery
- Read directly with the BigQuery Storage Read API through client libraries and framework connectors. This streams rows in parallel without exporting to CSV first.
- For very large one-off training sets, export a query result to sharded Parquet or Avro in Cloud Storage, then train from files.
- Use BigQuery DataFrames during development to sample and inspect before training.
From managed datasets
When a custom training job uses a managed dataset, Agent Platform exports the splits and sets:
| Variable | Meaning |
|---|---|
AIP_DATA_FORMAT | jsonl, csv, or bigquery |
AIP_TRAINING_DATA_URI | Training split location (Cloud Storage wildcard URI or BigQuery table) |
AIP_VALIDATION_DATA_URI | Validation split location |
AIP_TEST_DATA_URI | Test split location |
Code that reads these variables works with any dataset version, with no hard-coded paths.
Ingesting from Various Sources
| Source | Ingestion path into training-ready storage |
|---|---|
| Operational databases (Cloud SQL, AlloyDB, Spanner) | Scheduled exports or federated queries into BigQuery, then training tables |
| Event streams (Pub/Sub) | Dataflow streaming into BigQuery (tables) or Cloud Storage (files) |
| On-premises files | Storage Transfer Service, or gcloud storage for smaller loads, into Cloud Storage |
| SaaS and marketing platforms | BigQuery Data Transfer Service into BigQuery |
| Other clouds | Storage Transfer Service, or BigQuery external tables over open formats |
| Documents and media | Upload to Cloud Storage, extract with Document AI or Speech-to-Text, store outputs in BigQuery |
Make ingestion a pipeline step (BigQuery, Dataflow, or Managed Spark components in Agent Platform Pipelines) so every training run records which data snapshot it used.
Access and Security for Training Data
- By default, a serverless training job reads Cloud Storage buckets in the same project through the Custom Code Service Agent. Assign a custom service account to limit the job to exactly the buckets and tables it needs.
- Grant the Agent Platform Service Agent access to buckets or tables in other projects that managed datasets use.
- Keep training data inside VPC Service Controls perimeters when exfiltration is a concern (Chapter 18).
Diagnosing Slow Input Pipelines
If GPUs sit idle between steps, the input pipeline is the bottleneck. Common fixes:
- Replace many small files with sharded TFRecord or Parquet.
- Read in parallel (interleave shards, multiple loader workers).
- Prefetch batches so the accelerator never waits.
- Cache decoded data when it fits in memory or local SSD.
- Move heavy preprocessing out of the training loop into an offline Dataflow or BigQuery step.
- Use the TensorBoard profiler to confirm where time goes.
A PyTorch job on Agent Platform serverless training must read 4 TB of image shards from gs://vision-data/train/. What is the simplest way to access them as files without downloading everything to each replica?
A custom training job uses an Agent Platform managed tabular dataset backed by BigQuery. How should the training code find the training split?
GPU utilization during training stays around 30%, with long pauses between steps. The data is 20 million small JPEG files in Cloud Storage. Which change is most likely to help?