9.3 Kubeflow on GKE, AutoML & Tabular Workflows as Training Choices

The exam guide's training bullet names four SDK families: Agent Platform custom training, Kubeflow on GKE, Agent Platform AutoML, and Tabular Workflows. Questions test whether you can pick the one that matches the team's skills, control needs, and appetite for running infrastructure.

Kubeflow on Google Kubernetes Engine (GKE)

Kubeflow is an open-source ML toolkit for Kubernetes. On GKE, its training component, Kubeflow Trainer, runs distributed training and fine-tuning as Kubernetes resources. The newer TrainJob API replaces the earlier Training Operator's framework-specific jobs such as PyTorchJob and TFJob. It supports frameworks including PyTorch, JAX, DeepSpeed, Hugging Face, and XGBoost, sets up distributed communication (for example, the environment torchrun needs), and can gang-schedule multi-node jobs.

When Kubeflow on GKE is the right answer

• The organization already runs GKE with platform engineers, shared GPU node pools, and Kubernetes-native tooling and policies.
• Portability matters: the same training definitions must run on-premises, on other clouds, and on Google Cloud.
• The team needs fine-grained Kubernetes control: custom schedulers, sidecars, node affinity, or packing inference and training on shared accelerator pools.
• Existing investment in Kubeflow Pipelines and Trainer manifests is large.

When it's the wrong answer

• The scenario stresses minimal operational overhead or a team without Kubernetes skills. Agent Platform serverless training provisions and releases compute for you.
• Nothing requires portability or Kubernetes-specific control.

AutoML as a Training SDK

AutoML (Chapter 3) trains image and tabular models through training pipelines created from the console, the API, or the Python SDK (for example, AutoMLTabularTrainingJob). Organizing AutoML training well means:

• Using managed datasets with versioned data.
• Choosing the split strategy (random, manual column, or chronological).
• Setting the optimization objective and budget in node hours.
• Registering the resulting model in Model Registry and comparing its evaluation with custom models trained on the same dataset.

AutoML gives no control over individual steps. That's the gap Tabular Workflows fill.

Tabular Workflows

Tabular Workflows are fully managed, scalable pipelines for end-to-end ML on tabular data. Each workflow is a managed instance of Agent Platform Pipelines (which runs Kubeflow pipelines serverlessly), defined through the Google Cloud Pipeline Components (GCPC) SDK.

What Tabular Workflow for End-to-End AutoML adds over AutoML

You control every step instead of the whole pipeline at once:

• Data splitting
• Feature engineering
• Architecture search, which you can limit or skip for stability and speed
• Model training, with hardware you select for training and search
• Model ensembling, including changing ensemble size
• Model distillation, to shrink the model and lower serving latency

It supports multi-terabyte datasets with up to 1,000 columns. Every component appears in the pipeline graph with transformed tables, evaluated architectures, logs, and parameters. That transparency plain AutoML doesn't provide.

Versioning: each workflow version ships with the GCPC SDK. Pinning the SDK version pins the workflow version, so upgrades don't silently change training behavior.

template_path, parameter_values = tabular_utils.get_automl_tabular_pipeline_and_parameters(
    ...,
    optimization_objective="minimize-log-loss",
    data_source_bigquery_table_path="bq://proj.ds.training",
    target_column="churned",
    ...)
job = pipeline_jobs.PipelineJob(template_path=template_path,
                                parameter_values=parameter_values, ...)
job.run()

Running KFP Pipelines Without Running Kubeflow

Teams often mix these up. Kubeflow Pipelines (KFP) is the SDK and pipeline format. Kubeflow on GKE is a self-managed deployment of Kubeflow components on your own cluster. Agent Platform Pipelines runs KFP-format pipelines serverlessly. A team can write pipelines with the KFP SDK, run them on Agent Platform Pipelines, and never operate a Kubeflow cluster. Choose self-managed Kubeflow only when a specific Kubernetes or portability requirement calls for it.

Choosing Among All Training SDKs

Worked Scenario

A retailer's AutoML churn model performs well, but the serving team needs latency under 20 ms and the data science lead wants to see and control feature engineering. The dataset is 3 TB with 600 columns. Plain AutoML can't expose steps or distill the model. The best fit is Tabular Workflow for End-to-End AutoML, with architecture search narrowed and distillation enabled, run as a pipeline with the GCPC SDK version pinned.