17.2 CI/CD/CT for ML with Cloud Build

The exam guide lists deploying models in continuous integration, continuous delivery, and continuous training (CI/CD/CT) pipelines (for example, Cloud Build). ML systems ship code (preprocessing, training, pipeline definitions, serving containers), data-dependent artifacts (trained models), and configuration. Each needs automation.

The Three Loops

Cloud Build Essentials

• A build config (cloudbuild.yaml) lists steps. Each step runs in a Docker container, and steps share a workspace and the cloudbuild Docker network.
• Steps can use Google-provided builders, community builders, or custom builder images.
• Build triggers start builds on pushes, pull requests, or tags from GitHub, Bitbucket, Cloud Source Repositories, and others. Builds can also be started manually with gcloud or the API.
• Builds produce artifacts such as container images, which you push to Artifact Registry.
• Cloud Build meets SLSA level 3 supply-chain requirements. Pair it with vulnerability scanning and build provenance for secure ML containers.
• Approval gates on triggers and private pools (builds with private network access) support regulated environments.

What ML CI Tests

Google's MLOps guidance lists CI tests beyond normal unit tests:

1. Unit tests for feature engineering logic (for example, one-hot encoding handles unseen categories).
2. Unit tests for model methods.
3. Training converges: loss goes down, and the model can overfit a few sample records.
4. No NaN values from division by zero or extreme values.
5. Each pipeline component produces its expected artifacts.
6. Integration between pipeline components.

What ML CD Verifies

Before models or pipelines reach production:

• Infrastructure compatibility: required packages are in the serving environment, and memory, compute, and accelerators are available.
• Prediction service API tests with expected inputs and responses, which catch input-schema changes between versions.
• Load tests for QPS and latency.
• Data validation for retraining or batch prediction.
• Model performance targets met before deployment.

Environment promotion (Google's recommended pattern)

A Reference CI/CD/CT Flow on Google Cloud

1. A data scientist opens a pull request that changes feature code and the training component.
2. Cloud Build (CI) runs unit tests, builds the training and serving container images, scans them, pushes them to Artifact Registry, compiles the KFP pipeline, and runs a small integration pipeline on sample data.
3. On merge, Cloud Build (CD) uploads the compiled pipeline as a versioned template in Artifact Registry and deploys it to pre-production (for example, updating the pipeline schedule).
4. CT: schedules or events run the pipeline. It validates data, trains, evaluates, and uploads a model version to Model Registry with a challenger alias.
5. Model CD: if validation gates pass, the pipeline (or a Cloud Build job triggered by the pipeline's result) deploys the challenger as a canary on the endpoint. After approval and guardrail checks, the champion alias moves and traffic shifts to 100%.
6. Monitoring (Chapter 19) feeds alerts back as retraining triggers or new experiment cycles.

# cloudbuild.yaml (simplified)
steps:
- name: python:3.11
  entrypoint: bash
  args: ["-c", "pip install -r requirements.txt && pytest tests/"]
- name: gcr.io/cloud-builders/docker
  args: ["build", "-t", "us-central1-docker.pkg.dev/$PROJECT_ID/ml/trainer:$SHORT_SHA", "trainer/"]
- name: gcr.io/cloud-builders/docker
  args: ["push", "us-central1-docker.pkg.dev/$PROJECT_ID/ml/trainer:$SHORT_SHA"]
- name: python:3.11
  entrypoint: bash
  args: ["-c", "pip install kfp google-cloud-aiplatform && python compile_and_upload.py --image-tag $SHORT_SHA"]

Versioning Everything

CI/CD for Gen AI Applications

• Prompt templates, system instructions, and tool definitions live in source control and pass through CI.
• CI runs Gen AI evals on a fixed evaluation set and fails the build if rubric pass rates or safety scores fall below thresholds (Chapter 7).
• Model version upgrades, such as moving before a retirement date, run through the same evaluation and canary process.

Exam Traps

• Treating CI/CD as only "build the container." ML CI adds data and model tests, and CD adds validation against infrastructure and performance targets.
• Deploying models directly from a notebook to production and skipping registry, approvals, and canaries.
• Retraining automatically in production with no human approval for regulated models.
• Confusing CT (new model from new data, same code) with CI/CD (new code or pipeline).