5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage

Key Takeaways

Lightweight Python Function components (@dsl.component) allow rapid authoring by packaging standalone Python code and installing dynamic pip packages into a base image at runtime.
Containerized Components (@dsl.container_component) provide complete environment isolation and performance optimization by executing pre-built custom Docker container images with compiled C++, CUDA, or proprietary dependencies.
KFP v2 enforces a strict architectural boundary between Parameters (small primitive values passed via command-line arguments and logged in task specs) and Artifacts (large, complex objects like Datasets and Models stored in Cloud Storage and passed by URI reference).
Vertex ML Metadata (MLMD) tracks the four fundamental relational entities of ML systems: Contexts, Executions, Artifacts, and Events, enabling full auditability, reproducibility, and visual lineage tracking.
Vertex AI Pipelines step caching hashes the component specification, container image, input parameter values, and upstream input artifact IDs to skip execution of unchanged DAG nodes, dramatically reducing training costs and cycle times.

Last updated: September 2026

5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage

Building enterprise-grade ML pipelines requires rigorous modularity, robust artifact handling, and comprehensive auditability. In production environments, an ML pipeline is not merely a script that runs; it is a software system where every data transformation, trained weight matrix, evaluation metric, and deployment action must be tracked, reproducible, and debuggable.

The Kubeflow Pipelines (KFP) v2 SDK paired with Vertex ML Metadata (MLMD) provides the foundation for component reusability, artifact immutability, and end-to-end data provenance across the Google Cloud AI ecosystem.

1. KFP v2 Component Architectures: Lightweight vs. Containerized

When developing reusable pipeline tasks in KFP v2, ML engineers choose between two primary component authoring paradigms: Lightweight Python Function-based Components and Custom Containerized Components.

+---------------------------------------------------------------------------------------------------------+
|                                   KFP v2 COMPONENT AUTHORING SPECTRUM                                   |
+------------------------------------+------------------------------------+-------------------------------+
| ATTRIBUTE                          | LIGHTWEIGHT FUNCTION COMPONENT     | CUSTOM CONTAINERIZED COMPONENT|
+------------------------------------+------------------------------------+-------------------------------+
| Decorator Syntax                   | `@dsl.component`                   | `@dsl.container_component`    |
| Implementation                     | Self-contained Python function     | Custom Dockerfile & entrypoint|
| Environment Setup                  | `packages_to_install` at runtime   | Pre-built, baked container    |
| Startup Latency                    | Higher (downloads pip packages)    | Minimal (instant startup)     |
| Complex Dependencies               | Limited to pip wheels              | Full OS/C++/CUDA/binary access|
| Maintenance Overhead               | Very low; no Docker build pipeline | Moderate; requires CI builds  |
| Best Suited For                    | Rapid prototyping, standard Python | Production, heavy C++, CUDA   |
+------------------------------------+------------------------------------+-------------------------------+

1. Lightweight Python Function-Based Components

Lightweight components allow developers to define pipeline tasks directly in Python without manually writing Dockerfiles, building container images, or maintaining container registries. The KFP SDK automatically serializes the Python function, embeds it into a generated task spec, and executes it inside the specified base_image.

from kfp.dsl import component, Input, Output, Dataset, Metrics, Model

@component(
    base_image="python:3.10-slim",
    packages_to_install=["pandas==2.1.0", "scikit-learn==1.3.0", "joblib==1.3.2"]
)
def evaluate_model(
    test_dataset: Input[Dataset],
    model_artifact: Input[Model],
    metrics: Output[Metrics],
    threshold: float = 0.85
) -> bool:
    import pandas as pd
    import joblib
    from sklearn.metrics import accuracy_score, precision_score, recall_score
    
    # Read input dataset via GCS local mount path
    df = pd.read_parquet(test_dataset.path)
    X_test, y_test = df.drop(columns=["target"]), df["target"]
    
    # Load model weights
    clf = joblib.load(f"{model_artifact.path}/model.joblib")
    predictions = clf.predict(X_test)
    
    acc = accuracy_score(y_test, predictions)
    prec = precision_score(y_test, predictions)
    rec = recall_score(y_test, predictions)
    
    # Log scalar metrics directly into Vertex ML Metadata
    metrics.log_metric("accuracy", float(acc))
    metrics.log_metric("precision", float(prec))
    metrics.log_metric("recall", float(rec))
    
    return bool(acc >= threshold)

2. Custom Containerized Components

For enterprise production pipelines requiring strict environment immutability, complex non-Python binaries (e.g., OpenCV with custom C++ extensions, CUDA 12.2 drivers, proprietary C++ libraries), or sub-second cold start times, Containerized Components provide full control. The container image is pre-built, vulnerability-scanned, and pushed to Google Artifact Registry.

from kfp.dsl import container_component, ContainerSpec, Input, Output, Dataset, Model

@container_component
def run_distributed_cpp_preprocessor(
    raw_data: Input[Dataset],
    processed_data: Output[Dataset],
    num_shards: int = 16
):
    return ContainerSpec(
        image="us-central1-docker.pkg.dev/my-gcp-project/ml-repo/cpp-preprocessor:v1.2.0",
        command=["/usr/local/bin/preprocess_engine"],
        args=[
            "--input_path", raw_data.path,
            "--output_path", processed_data.path,
            "--shards", str(num_shards)
        ]
    )

2. Artifacts vs. Parameters: Data Passing Mechanics

In KFP v2, data passing between DAG nodes is strictly typed and categorized into two distinct primitives: Parameters and Artifacts.

+---------------------------------------------------------------------------------------------------------+
|                                     PARAMETERS VS. ARTIFACTS IN KFP v2                                  |
+------------------------------------+------------------------------------+-------------------------------+
| DIMENSION                          | PARAMETER                          | ARTIFACT                      |
+------------------------------------+------------------------------------+-------------------------------+
| Data Scale                         | Small primitives (< kilobyte)      | Arbitrary size (MB, GB, TB)   |
| Data Types                         | `int`, `float`, `str`, `bool`, `dict` | `Dataset`, `Model`, `Metrics` |
| Storage Location                   | Vertex ML Metadata DB / JSON Spec  | Cloud Storage (gs:// bucket)  |
| Passing Mechanism                  | Command-line CLI flag / Env Var    | Cloud Storage URI path pointer|
| Lineage Representation             | Stored in Execution node metadata  | Stored as distinct MLMD Node  |
| UI Visualization                   | Table of key-value pairs           | Interactive graphs, confusion |
|                                    |                                    | matrices, markdown, tables    |
+------------------------------------+------------------------------------+-------------------------------+

Standard KFP v2 Artifact Types

Dataset (Input[Dataset], Output[Dataset]): Represents raw, processed, or partitioned tabular/image data on Cloud Storage.
Model (Input[Model], Output[Model]): Represents serialized model weight directories (SavedModel, joblib, PyTorch .pt, ONNX).
Metrics (Output[Metrics]): Scalar numeric key-value pairs (accuracy, RMSE, loss) logged via metrics.log_metric(), rendered in comparison tables.
ClassificationMetrics (Output[ClassificationMetrics]): Specialized artifact supporting rich visual curves in the Vertex AI Console (Confusion Matrices, ROC curves, Precision-Recall curves) via log_confusion_matrix() and log_roc_curve().
HTML & Markdown (Output[HTML], Output[Markdown]): Rich visual diagnostic reports rendered directly in the Vertex AI Pipelines execution dashboard.

[ Component A (Data Splitter) ]
       | 
       | Produces: Output[Dataset] (Writes dataframe to gs://my-bucket/artifacts/.../data.csv)
       v
[ Cloud Storage Artifact Store ] <====== Immutable GCS URI Pointer
       |
       | Consumes: Input[Dataset] (Reads dataframe from GCS mount)
       v
[ Component B (Model Trainer) ]

3. Vertex ML Metadata (MLMD): Provenance and Lineage

Vertex ML Metadata (MLMD) is a managed metadata repository that automatically records every action, parameter, and output generated across Vertex AI Pipelines, Custom Training Jobs, and Model Registry deployments.

+---------------------------------------------------------------------------------------------------------+
|                                 VERTEX ML METADATA (MLMD) CORE GRAPH ENTITIES                           |
+---------------------------------------------------------------------------------------------------------+
|                                                                                                         |
|   +-------------------------------------------------------------------------------------------------+   |
|   | CONTEXT (e.g., PipelineRun: 'churn-pipeline-20260901-0830', Experiment: 'churn-hyperopt-v2')    |   |
|   +-------------------------------------------------------------------------------------------------+   |
|          |                                                                                              |
|          | Groups & Scopes                                                                              |
|          v                                                                                              |
|   +-------------------+              EVENT (Input)             +-------------------+                    |
|   | ARTIFACT (Data)   | -------------------------------------> | EXECUTION (Task)  |                    |
|   | Type: Dataset     |                                        | Type: Trainer     |                    |
|   | URI: gs://.../raw |                                        | State: COMPLETE   |                    |
|   +-------------------+                                        +-------------------+                    |
|                                                                          |                              |
|                                                                          | EVENT (Output)               |
|                                                                          v                              |
|                                                                +-------------------+                    |
|                                                                | ARTIFACT (Model)  |                    |
|                                                                | Type: Model       |                    |
|                                                                | URI: gs://.../mod |                    |
|                                                                +-------------------+                    |
+---------------------------------------------------------------------------------------------------------+

The Four Core MLMD Entities

Artifacts: Concrete entities or data objects created or consumed by an ML workflow (e.g., datasets, model binaries, evaluation metric JSONs, vocabulary files). Each artifact possesses a unique URI in Cloud Storage and a schema type.
Executions: Individual execution steps or computational tasks within a workflow (e.g., a data preprocessing container run, a distributed training job, a batch prediction scoring run). Executions record runtime parameters, start/end timestamps, container images, and completion status.
Events: Directed relational edges that connect Executions and Artifacts. An Event denotes whether an artifact was an INPUT consumed by an execution or an OUTPUT produced by an execution.
Contexts: Logical groupings that cluster executions and artifacts together (e.g., a specific PipelineRun, an Experiment, or an ExperimentRun).

Why Lineage Matters for Enterprise Governance

Reproducibility & Debugging: If a production model begins returning anomalous predictions, ML engineers can query MLMD to trace the exact lineage: identifying the specific training execution ID, the exact Git commit SHA embedded in metadata, the hyperparameter dictionary, and the specific raw GCS data shards consumed during training.
Regulatory Auditability: In regulated industries (finance, healthcare), MLMD provides immutable proof of data provenance, ensuring that models were trained strictly on compliant, audited data partitions.

4. Vertex AI Pipelines Execution Caching

One of the most powerful cost- and time-saving features of Vertex AI Pipelines is Execution Caching.

+---------------------------------------------------------------------------------------------------------+
|                                      EXECUTION CACHE FINGERPRINTING                                      |
+---------------------------------------------------------------------------------------------------------+
|                                                                                                         |
|   COMPONENT CACHE HASH = HASH(                                                                          |
|       1. Component Specification (Python code, commands, entrypoints)                                   |
|       2. Container Image URI & Digest (e.g., us-docker.pkg.dev/.../preprocessor@sha256:abc...)          |
|       3. Input Parameter Values (e.g., learning_rate=0.01, max_depth=5)                                 |
|       4. Upstream Input Artifact IDs & Checksums (e.g., dataset_artifact_id_89412)                      |
|   )                                                                                                     |
|                                                                                                         |
|   [ Cache Hit Check ]                                                                                   |
|      * If Hash Matches Previous Execution in MLMD: REUSE OUTPUT ARTIFACTS (Execution Time: 0s, Cost: $0)|
|      * If Hash Differs (Code changed, data updated, params altered): EXECUTE NEW CONTAINER             |
+---------------------------------------------------------------------------------------------------------+

Caching Mechanics and Best Practices

Cache Key Calculation: Vertex AI computes a deterministic fingerprint for each task based on four factors: the component specification/code, the container image URI/digest, all input parameter values, and the upstream input artifact IDs.
Cache Hits: When a component is submitted with identical inputs and code, Vertex AI skips container execution entirely, instantly linking the previously generated output artifacts into the current pipeline run. A 4-hour feature extraction step on 100 GB of data executes in 0 seconds.
Configuring Caching in Code:

# Enable or disable caching at the individual component level
train_task = train_model(dataset=ingest_task.output)
train_task.set_caching_options(enable_caching=True)  # Or False to force re-execution

# Configure pipeline-level caching during submission
job = aiplatform.PipelineJob(
    display_name="churn-training-job",
    template_path="churn_pipeline.yaml",
    enable_caching=True  # Enables caching across all tasks in DAG
)
job.submit()

[!IMPORTANT] When to Disable Caching: Always disable caching (enable_caching=False) for components that read non-deterministic external data sources directly (e.g., querying CURRENT_DATE() from BigQuery, fetching real-time API feeds, or downloading from static GCS URIs that get overwritten in-place without new artifact IDs).

Loading diagram...

Vertex ML Metadata (MLMD) Lineage and Event Tracking Graph

Test Your Knowledge

An ML engineering team is writing a KFP v2 pipeline task that requires a specialized C++ optical character recognition (OCR) binary, specific NVIDIA CUDA 12.2 GPU drivers, and system-level Ubuntu library dependencies. The component takes approximately 10 minutes to build if dependencies are installed at runtime. How should the engineer author this component to ensure fast startup times and strict reproducibility?

Use a lightweight Python function component (@dsl.component) and list the C++ dependencies inside packages_to_install.

Write a Python script that executes apt-get install and compiles the C++ code inside the function body at runtime.

Build a custom Docker container containing the C++ binary and CUDA drivers, push it to Artifact Registry, and define the task using @dsl.container_component with ContainerSpec.

Deploy the C++ binary to a persistent Compute Engine VM and call it via SSH from a lightweight KFP component.

Test Your Knowledge

A fraud detection model deployed in production suddenly begins generating anomalous classification predictions. Compliance auditors require the ML engineering team to trace the production model artifact back to the exact training dataset version, the preprocessing code parameters, and the evaluation metrics generated during training. Which Google Cloud capability provides this audit trail automatically?

Query Vertex ML Metadata (MLMD) to inspect the Context, Execution, Artifact, and Event lineage graph associated with the model artifact.

Inspect the pipeline DAG definition YAML file stored in Cloud Source Repositories.

Review Cloud Audit Logs and grep for GCS file download events in the bucket access logs.

Examine the container stdout logs stored in Cloud Logging for the historical Vertex AI Training job.

Test Your Knowledge

An ML team runs a daily Vertex AI Pipeline that extracts features from 2 TB of tabular data (taking 3.5 hours) before training an XGBoost model. On days when the upstream raw data and hyperparameter settings do not change, the team wants the pipeline to skip the 3.5-hour feature extraction step automatically and immediately proceed to downstream tasks. How can the team achieve this optimization?

Write custom Python logic inside the component to check BigQuery table row counts and raise an exception if unchanged.

Configure Cloud Composer to conditionally skip the task using a ShortCircuitOperator.

Store preprocessed features in BigQuery and query them using a manual IF statement in SQL.

Enable execution caching on the pipeline or component (enable_caching=True), allowing Vertex AI to match execution fingerprints and reuse output artifacts.

Test Your Knowledge

In the KFP v2 SDK, what is the fundamental architectural distinction between a pipeline Parameter and a pipeline Artifact?

Parameters are stored in Cloud Storage buckets, whereas Artifacts are passed directly via environment variables.

Parameters represent small primitive values (int, float, str, bool) passed by value via CLI flags and logged in metadata, whereas Artifacts represent large complex data objects (Dataset, Model, Metrics) stored in Cloud Storage and passed by URI reference.

Parameters are only supported in TFX pipelines, whereas Artifacts are only supported in KFP v2 pipelines.

Artifacts are mutable during pipeline execution, whereas Parameters are immutable constants hardcoded at compile time.

Up Next

5.4 Ray on Agent Platform for Distributed Python Workloads

Continue learning

Google Cloud Professional Machine Learning Engineer

GCP ML Engineer

5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage

Key Takeaways

5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage

1. KFP v2 Component Architectures: Lightweight vs. Containerized

1. Lightweight Python Function-Based Components

2. Custom Containerized Components

2. Artifacts vs. Parameters: Data Passing Mechanics

Standard KFP v2 Artifact Types

3. Vertex ML Metadata (MLMD): Provenance and Lineage

The Four Core MLMD Entities

Why Lineage Matters for Enterprise Governance

4. Vertex AI Pipelines Execution Caching

Caching Mechanics and Best Practices

Google Cloud Professional Machine Learning Engineer

1Introduction & Exam Overview

2Section 1: Architecting Low-Code AI Solutions (~13%)

3Section 2: Collaborating Within and Across Teams to Manage Data and Models (~16%)

4Section 3: Scaling Prototypes into ML Models (~21%)

5Section 4: Serving and Scaling Models (~20%)

6Section 5: Automating and Orchestrating ML Pipelines (~18%)

7Section 6: Monitoring AI Solutions (~13%)

GCP ML Engineer

5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage

Key Takeaways

5.3 Reusable Pipeline Components, Artifacts and ML Metadata Lineage

1. KFP v2 Component Architectures: Lightweight vs. Containerized

1. Lightweight Python Function-Based Components

2. Custom Containerized Components

2. Artifacts vs. Parameters: Data Passing Mechanics

Standard KFP v2 Artifact Types

3. Vertex ML Metadata (MLMD): Provenance and Lineage

The Four Core MLMD Entities

Why Lineage Matters for Enterprise Governance

4. Vertex AI Pipelines Execution Caching

Caching Mechanics and Best Practices