5.1 Organizing & Exploring Tabular, Text & Image Data

The exam guide opens Section 2 with organizing and exploring different data types (tabular, text, and images) for efficient experimenting, training, and serving. In practice, most "the model is bad" problems come from data: the wrong storage layout, unreadable formats, leakage, or training data that doesn't look like production data.

Where Each Data Type Belongs

Keep storage in the same region as training and serving compute. Cross-region reads add latency and egress cost, and they can break data residency rules.

Managed Datasets and Versions

A managed dataset on Agent Platform is required for AutoML and optional for custom training. Google lists these benefits:

• Central organization and governance of data assets across projects and experiments.
• Labeling: create annotation sets and labeling tasks inside the dataset.
• Lineage: automatically links data to the models trained on it, which helps reproducibility and audits.
• Fair comparison: AutoML and custom models trained on the same dataset can be compared directly.
• Statistics and visualizations for exploratory analysis.
• Automatic splits by fraction, filter, predefined column, or timestamp.
• Dataset versions so you can track changes and roll back.

For custom training, the job receives data locations through environment variables such as AIP_TRAINING_DATA_URI, AIP_VALIDATION_DATA_URI, and AIP_TEST_DATA_URI.

Getting Data into Training Efficiently

Many small files (millions of tiny JPEGs) slow training because each file needs a separate request. Pack them into larger sharded files such as TFRecord for sequential reads.

Discovering Data with Knowledge Catalog

Knowledge Catalog (renamed from Dataplex Universal Catalog on April 10, 2026) is the metadata layer for finding and understanding data across projects and regions. It's integrated with Agent Platform, so managed datasets can be searched and enriched with aspects (similar to the tags in the deprecated Data Catalog). Use it to answer "which table is the approved source for customer churn labels?" before you train on an unofficial copy.

Exploratory Data Analysis (EDA) Checklist

Tabular data

• Target distribution: class balance for classification, skew and outliers for regression.
• Missing values per column and whether missingness itself predicts the label.
• Cardinality of categorical columns. A very high-cardinality ID can let the model memorize individual rows.
• Leakage: columns recorded after the outcome, such as cancellation_reason when predicting churn.
• Duplicates and rows that appear in both training and test sets.
• Time coverage: does training cover the seasons and conditions that production will see?

Text data

• Length distribution in characters and tokens, which drives model context and cost.
• Language mix and encoding problems.
• Label consistency, since different annotators often label the same text differently.
• Sensitive data such as names, account numbers, and emails (Section 5.4).

Image data

• Images per label (AutoML image classification recommends about 1,000 per label, minimum 10).
• Resolution, aspect ratio, lighting, and camera differences between training and production.
• Near-duplicate images that inflate evaluation metrics.
• A "none of the above" class if production images can fall outside every label.

Tools for Exploration

• BigQuery SQL for profiling (COUNT, APPROX_QUANTILES, COUNTIF(col IS NULL)), directly on full tables.
• BigQuery DataFrames (bigframes) for pandas-style exploration that runs in BigQuery, so terabyte tables never load into notebook memory.
• Colab Enterprise or Workbench notebooks for visualization and iterative analysis (Chapter 6).
• TensorFlow Data Validation-style statistics: BigQuery ML's ML.TFDV_DESCRIBE computes statistics you can later compare with ML.TFDV_VALIDATE to catch schema drift (Chapter 15).

Worked Scenario: A Multimodal Claims Dataset

An insurer wants a model that predicts claim severity from structured policy data, adjuster notes, and damage photos.

• Policy and claim attributes go in BigQuery, partitioned by claim date and clustered by region for efficient training queries.
• Photos stay in Cloud Storage. A BigQuery object table exposes their URIs and metadata, so each photo can be joined to its claim in SQL.
• Adjuster notes are short text, so they live in a BigQuery column next to the claim, after PII redaction (Section 5.4).
• A managed dataset version freezes the exact training snapshot, and lineage ties it to the resulting model for audits.
• EDA shows 70% of photos come from one smartphone app launched last year. The team checks that older claims (scanned prints) are still represented, so production doesn't meet an unfamiliar image style.

Designing for Serving from Day One

Exploration decisions carry into production. If a feature will be needed online, confirm it's available at prediction time and from a low-latency source. Record how each feature is computed so training and serving use the same definition, which is the job of the Feature Store and shared transformations (Sections 5.3 and 15.3).