2.10 Experiment Tracking, Artifact Lineage and Model Registry Governance

2.10 Experiment Tracking, Artifact Lineage and Model Registry Governance

Blueprint reference: Section 2.3, "Tracking and comparing model artifacts, versions, and lineage (e.g., Experiments on Agent Platform and Gemini Enterprise Agent Platform ML Metadata)."

Moving machine learning from experimental exploration to regulated, repeatable production requires systematic experiment tracking, artifact lineage, and rigorous model lifecycle governance. On Google Cloud, Experiments on Agent Platform, Agent Platform ML Metadata, and the Model Registry carry those three responsibilities.

---

1. Where This Section Sits

Notebook surfaces themselves — Agent Platform Workbench versus Colab Enterprise, runtime templates, least-privilege service accounts, private IP with IAP, and idle shutdown — are covered in Sections 2.5 and 2.6. This section picks up after the prototype works and asks the governance question: how is the run recorded, how is the resulting artifact catalogued, and how is its provenance proved later.

Three services answer that, and the exam expects you to keep their roles distinct:

A common wrong answer conflates the first and third: Experiments compares attempts, the Model Registry catalogues shippable versions. An experiment run is not a model version, and promoting one means explicitly registering it.

---

2. Systematic Tracking with Vertex AI Experiments & TensorBoard

In iterative ML development, data scientists experiment with multiple architectures, optimizers, learning rates, and feature sets. Ad-hoc spreadsheet logging fails to provide reproducibility or lineage.

+---------------------------------------------------------------------------------------+
|                         VERTEX AI EXPERIMENTS DATA MODEL                              |
|                                                                                       |
|   Experiment: `fraud-detection-transformer`                                           |
|   ├── Experiment Run: `run-lr-001-batch-64`                                           |
|   │   ├── Parameters: {learning_rate: 0.001, batch_size: 64, optimizer: 'Adam'}      |
|   │   ├── Metrics: {val_loss: 0.214, pr_auc: 0.942, f1_score: 0.891}                |
|   │   ├── Artifacts: `gs://my-bucket/artifacts/model_v1.tar.gz`                      |
|   │   └── TensorBoard Time Series: Managed TensorBoard Instance Log                   |
|   └── Experiment Run: `run-lr-0001-batch-128`                                         |
|       ├── Parameters: {learning_rate: 0.0001, batch_size: 128, optimizer: 'AdamW'}   |
|       └── Metrics: {val_loss: 0.187, pr_auc: 0.965, f1_score: 0.923}                |
+---------------------------------------------------------------------------------------+

Key Capabilities of Vertex AI Experiments

• Parameters & Metrics Tracking: Using the google-cloud-aiplatform Python SDK, data scientists log hyperparameters (aiplatform.log_params()) and scalar evaluation metrics (aiplatform.log_metrics()) directly from code.
• Artifact & Lineage Recording: Connects input datasets (BigQuery tables, GCS CSVs) to intermediate model checkpoints and output evaluation confusion matrices in Vertex ML Metadata.
• Managed Vertex AI TensorBoard: Google Cloud provides a fully managed, enterprise-scale TensorBoard service without requiring local daemon hosting. Features include:
  • Real-time loss and accuracy curve visualization across multiple runs.
  • Histogram of weights and gradients to diagnose vanishing/exploding gradients.
  • Profiling GPU memory utilization and compute bottlenecks during training execution.

---

3. Enterprise Model Governance: Vertex AI Model Registry

Vertex AI Model Registry serves as the central catalog for trained machine learning models across an organization. It bridges the gap between model development and production serving.

                                VERTEX AI MODEL REGISTRY ECOSYSTEM
                                                |
         +--------------------------------------+--------------------------------------+
         |                                                                             |
  [ Model Lifecycle & Versioning ]                                            [ Governance & Deployment ]
         |                                                                             |
  - Immutable Model Versions (v1, v2, v3)                                     - Vertex Model Cards (Ethics & Bias)
  - Mutable Version Aliases (@default, @prod)                                 - Deployment Lineage (Vertex Endpoints)
  - Custom Container Image Specifications                                     - Cross-Project Sharing (Shared VPC / IAM)
  - Pre-built Serving Containers (TF, PyTorch, SKLearn)                       - Batch Prediction & Endpoint Deployments

Model Versioning and Mutable Aliases

• Immutable Versioning: Every time a model is registered or updated with a new artifact, Vertex AI Model Registry assigns an incremental, immutable version number (1, 2, 3). The underlying artifact URI, container image, and schema cannot be altered.
• Mutable Version Aliases: Aliases act as symbolic tags pointing to specific model versions. Common alias patterns include @default, @staging, @canary, and @production.
  • When a model passes automated evaluation gates, engineers reassign the @production alias from Version 1 to Version 2.
  • Upstream production inference clients querying @production automatically receive predictions from the new model version without modifying client code or endpoint URL configurations.

Model Cards for Governance and Responsible AI

Vertex AI Model Registry integrates Vertex Model Cards, structured documentation artifacts that record critical operational and ethical characteristics of the model:

• Intended Use Cases: Approved production applications and explicitly out-of-scope or prohibited use cases.
• Training & Evaluation Data Lineage: Datasets utilized, demographic splits, and historical data timeframes.
• Performance & Fairness Benchmarks: Accuracy, precision-recall metrics broken down across protected demographic slices, and fairness metrics (e.g., disparate impact ratio).
• Maintenance Schedule: Model expiration dates and mandatory retraining schedules.

Cross-Project Model Sharing & IAM Governance

In multi-project enterprise architectures (e.g., ml-dev-project, ml-staging-project, ml-prod-project):

• Models registered in a centralized governance project can be shared with downstream production deployment projects using granular IAM permissions (roles/aiplatform.viewer and roles/aiplatform.user).
• Model artifacts stored in Cloud Storage buckets can be locked down using Customer-Managed Encryption Keys (CMEK) and accessed across projects via service accounts.