2.8 Choosing the Development and Experimentation Environment
Key Takeaways
- Experiments on Agent Platform tracks parameters, metrics, and artifacts across runs and is the lightweight choice while the workflow is still changing.
- Agent Platform Pipelines is the right environment once the workflow is stable, multi-step, and must be reproducible or scheduled.
- Kubeflow Pipelines on GKE is chosen when the team needs cluster-level control, existing GKE investment, or portability across clouds.
- Framework matters: TFX is the natural pipeline authoring layer for TensorFlow workflows, while KFP is framework-agnostic and preferred for PyTorch, scikit-learn, and mixed stacks.
- The escalation path is notebook plus Experiments, then a training job, then a pipeline — moving too early adds ceremony, moving too late blocks reproducibility.
2.8 Choosing the Development and Experimentation Environment
Blueprint reference: Section 2.3, "Choosing the appropriate Google Cloud environment for development and experimentation (e.g., Experiments on Gemini Enterprise Agent Platform, Gemini Enterprise Agent Platform Pipelines, and Kubeflow Pipelines) given the framework."
Note the last three words. The official bullet ties this choice to the framework in use, which is a hint that the exam will present a stack and ask which environment fits it.
The Three Environments
| Environment | What it is | Choose it when |
|---|---|---|
| Experiments on Agent Platform | A tracking layer that records parameters, metrics, and artifacts per run, comparable in the console | The workflow is still changing; you need to compare many quick attempts without building infrastructure |
| Agent Platform Pipelines | Serverless managed execution of KFP or TFX pipelines, per-run billing, no cluster | The workflow is stable and multi-step, must be reproducible, scheduled, or triggered, and you do not want to operate infrastructure |
| Kubeflow Pipelines on GKE | Self-managed Kubeflow on a cluster you run | You need cluster-level control, have existing GKE investment or on-premises parity requirements, or need portability across environments |
The decision is mostly about operational burden versus control. Agent Platform Pipelines removes the cluster entirely and charges per pipeline run plus the resources each step consumes; there is nothing to patch, upgrade, or size when idle. Self-managed Kubeflow on GKE keeps the cluster — and its cost — running continuously, and you own upgrades, but you get full control over the environment, custom operators, and the ability to run the same pipeline definition elsewhere.
That difference produces a reliable exam signal: "we do not want to manage infrastructure" or "we have no platform team" points to Agent Platform Pipelines; "we must run the same pipeline on-premises as well" or "we already operate GKE and need custom cluster configuration" points to Kubeflow on GKE.
Experiments Is Not an Alternative to Pipelines
A common misreading treats these as three mutually exclusive options. In practice Experiments is a tracking layer that composes with both: a notebook logs to it, a training job logs to it, and a pipeline step logs to it. What it is not is an orchestrator — it has no scheduling, no DAG, and no dependency management.
So the real question is usually "is this workload still exploratory, or is it a workflow?"
Exploration → Notebook + Experiments
Repeatable training → Custom training job + Experiments
Multi-step workflow → Pipeline (Agent Platform or Kubeflow) + Experiments + ML Metadata
Framework Drives the Authoring Layer
Agent Platform Pipelines executes pipelines authored in either of two SDKs, and the framework in use should decide which.
| Authoring SDK | Best fit | Why |
|---|---|---|
| KFP (Kubeflow Pipelines SDK) | PyTorch, scikit-learn, XGBoost, JAX, mixed-language, or any containerized step | Framework-agnostic; a component is just a container with typed inputs and outputs |
| TFX | TensorFlow end-to-end workflows | Opinionated standard components for ingestion, validation, transform, training, evaluation, and pushing, with built-in data and model validation |
TFX brings a great deal for free when the stack is TensorFlow — schema inference and anomaly detection through data validation, a transform step whose graph is embedded into the served model, and a model-blessing evaluator. Applied to a PyTorch workflow, most of that machinery is inapplicable and the abstraction becomes overhead. The exam expects the mapping: TensorFlow shop → TFX; heterogeneous or PyTorch shop → KFP.
Both run on the same managed Agent Platform Pipelines service, so the SDK choice does not change where the pipeline executes.
Cost Shapes
- Experiments — effectively free tracking; you pay for whatever compute produced the runs.
- Agent Platform Pipelines — a small per-run charge plus the compute each step consumes, and nothing at all between runs.
- Kubeflow on GKE — the cluster's node pool cost continuously, whether or not pipelines are running, plus the operational cost of maintaining it.
For a team running a handful of pipelines a week, a permanently-running GKE cluster is the wrong economic shape. For a team running hundreds of jobs an hour on a cluster that is already there for other reasons, it may be the right one.
Ray as a Fourth Option
For distributed Python workloads that are not naturally a DAG — hyperparameter sweeps, reinforcement learning, large-scale batch inference in Python — Ray on Agent Platform provides a managed Ray cluster. Unlike a custom training job, which releases its resources at completion, a Ray cluster persists until deleted, which makes it well suited to interactive distributed development and poorly suited to being forgotten about. It is covered in Chapter 5.
Exam Traps
- Treating Experiments as an orchestrator. It tracks; it does not schedule.
- Choosing self-managed Kubeflow "for flexibility" when the scenario complains about operational burden.
- Using TFX for a PyTorch pipeline. KFP is the framework-agnostic choice.
- Building a pipeline on day one for a workflow that is still changing daily.
- Ignoring idle cluster cost when comparing GKE-hosted Kubeflow to managed pipelines.
A three-person data science team with no platform engineers needs to run a stable five-step training workflow on a weekly schedule, using PyTorch and scikit-learn components. They explicitly do not want to operate infrastructure. What should they use?
A large enterprise runs an on-premises Kubernetes estate and requires that the same ML pipeline definition execute both on-premises and on Google Cloud, with cluster-level control over custom operators and node configuration. Which environment fits?
A TensorFlow-based team wants their pipeline to automatically infer a data schema, detect anomalies in incoming training data, embed preprocessing into the served model, and block promotion of a model that fails an evaluation threshold. Which authoring SDK gives them the most of this out of the box?
A data scientist is trying twenty variations of a feature set and model configuration over three days, changing the workflow structure almost daily. What is the appropriate environment right now?