15.2 Agent Platform Pipelines with KFP: Components, Templates & Caching

The exam guide lists building and orchestrating pipelines using managed or unmanaged services and from templates or custom solutions (for example, Agent Platform Pipelines, Managed Service for Apache Airflow, and Ray on Gemini Enterprise Agent Platform). This section covers the managed ML-native option, Agent Platform Pipelines (formerly Vertex AI Pipelines). Chapter 16 covers Managed Airflow and Ray.

What a Pipeline Is

An ML pipeline is a directed acyclic graph (DAG) of containerized tasks connected by input-output dependencies. You define it in Python with the Kubeflow Pipelines (KFP) SDK v2+ or the TFX SDK v0.30+, compile it to a YAML intermediate representation, and run it serverlessly. Tasks run in parallel by default unless one depends on another's outputs.

Components

Useful GCPC operators

A Minimal Training Pipeline

from kfp import dsl, compiler
from google_cloud_pipeline_components.v1.custom_job import CustomTrainingJobOp
from google_cloud_pipeline_components.v1.model import ModelUploadOp

@dsl.component(base_image="python:3.11", packages_to_install=["google-cloud-bigquery"])
def check_row_count(table: str, min_rows: int) -> bool:
    from google.cloud import bigquery
    n = list(bigquery.Client().query(f"SELECT COUNT(*) n FROM `{table}`").result())[0].n
    return n >= min_rows

@dsl.pipeline(name="churn-training")
def pipeline(project: str, table: str):
    ok = check_row_count(table=table, min_rows=100000)
    with dsl.If(ok.output == True):
        train = CustomTrainingJobOp(project=project, display_name="churn-train",
                                    worker_pool_specs=[...])
        ...

compiler.Compiler().compile(pipeline, "churn_pipeline.yaml")

Control flow includes dsl.If (conditions), dsl.ParallelFor (fan-out), and dsl.ExitHandler (always run a final step, such as a notification).

Running Pipelines

job = aiplatform.PipelineJob(
    display_name="churn-training",
    template_path="churn_pipeline.yaml",
    pipeline_root="gs://ml-pipelines/churn",
    parameter_values={"project": "p", "table": "ds.features"},
    enable_caching=True,
    failure_policy="fast")
job.submit(experiment="churn-experiments")

• pipeline_root is the Cloud Storage location for artifacts.
• Run as a service account with least-privilege access to data and services.
• Tasks can request machine types and accelerators, use persistent resources, read Secret Manager secrets, and run over Private Service Connect networking.
• Adding the run to an experiment makes runs comparable (Chapter 6).

Execution Caching

Before running a step, the service checks ML Metadata for a previous execution with the same cache key:

• The step's inputs (parameter values and input artifact IDs)
• Its output definitions
• The component specification (image, command, arguments, environment variables)

Only pipelines with the same pipeline name share the cache. On a match, the step is skipped and its outputs reused. Cached results have no TTL. They last until the metadata entry is deleted.

• Turn caching off for a task with task.set_caching_options(False), or for the whole run with enable_caching=False.
• Components must be deterministic. A step that reads "latest data" through a fixed query string may be wrongly skipped, because its inputs look identical. Pass a date or snapshot parameter, or turn off caching for that step.

Failure Handling

Templates and Reuse

A pipeline template publishes a compiled pipeline for reuse. Upload templates with the KFP registry client to an Artifact Registry Kubeflow Pipelines repository, with versions and tags. Teams then create runs from a template without seeing the source. The Template Gallery also offers Google-authored templates and components, such as AutoML for tabular classification and regression.

Scheduling and Triggers

Lineage, Logs, and Cost

• Every run records ML Metadata lineage automatically: artifacts, executions, and parameters.
• Task logs go to Cloud Logging. The console links each step to its underlying job.
• Pipeline costs include the orchestration charge per run plus the resources each task uses. Labels help attribute costs.