9.2 CI/CD Pipelines: Cloud Build, Artifact Registry & Cloud Deploy

CI/CD Pipelines: Cloud Build, Artifact Registry & Cloud Deploy

Architectural Objective: Modern software delivery requires automated, secure, and auditable pipelines from source commit to production runtime. A Google Professional Cloud Architect must design serverless continuous integration workflows using Cloud Build with private VPC connectivity, implement enterprise container governance with Artifact Registry and vulnerability scanning, enforce cryptographic supply chain security with Binary Authorization, and orchestrate progressive multi-environment delivery pipelines with Cloud Deploy.

---

Cloud Build Architecture & Serverless CI Orchestration

Google Cloud Build is a fully managed, serverless continuous integration and build execution platform that scales elastically without requiring the management of dedicated build servers (such as Jenkins masters/workers).

+-----------------------------------------------------------------------------------+
|                         CLOUD BUILD PIPELINE ARCHITECTURE                         |
+-----------------------------------------------------------------------------------+
| Trigger: Git Push / Webhook / Pub/Sub ──> Cloud Build Orchestrator                |
|                                                  │                                |
|                        ┌─────────────────────────┴──────────────────────────┐     |
|                        │ Ephemeral VM Worker Environment                    │     |
|                        │  ├── Shared Volume Mount: /workspace               │     |
|                        │  ├── Step 1: docker (Build Image)                  │     |
|                        │  ├── Step 2: gcloud / pytest (Run Tests)           │     |
|                        │  ├── Step 3: cosign / kms (Sign Attestation)       │     |
|                        │  └── Step 4: crane / docker push (Push to AR)      │     |
|                        └────────────────────────────────────────────────────┘     |
+-----------------------------------------------------------------------------------+

Build Configuration Anatomy (cloudbuild.yaml)

A build is defined as a series of containerized build steps. Each step runs as a distinct container with the local project source code mounted automatically at /workspace.

• Sequential vs. Parallel Execution: By default, build steps run sequentially. Architects can configure steps to run in parallel using the waitFor attribute (waitFor: ['-'] starts a step immediately at build launch), significantly optimizing total build latency.
• Custom Builders: Any container image hosted in Artifact Registry, Docker Hub, or Google Cloud Container Registry can serve as a build step, allowing teams to execute custom tooling (e.g., Maven, Go, Helm, Terraform, or SonarQube).
• Substitution Variables: Cloud Build provides built-in substitution variables ($PROJECT_ID, $BUILD_ID, $COMMIT_SHA, $BRANCH_NAME, $TAG_NAME) and supports custom user-defined substitutions ($_DEPLOY_ENV, $_IMAGE_TAG) configured at the trigger level.

Cloud Build Triggers

Builds are initiated automatically by Cloud Build Triggers reacting to diverse enterprise events:

1. Version Control Integration: Webhook-based or native GitHub/GitLab Enterprise apps reacting to branch pushes, tag creations, or pull request comments.
2. Pub/Sub Triggers: Asynchronous triggers activated by publishing JSON messages to a Cloud Pub/Sub topic (ideal for multi-pipeline orchestration).
3. Webhook Triggers: Direct HTTP POST invocations authenticated with secret tokens stored in Secret Manager.

---

Private Worker Pools: Secure VPC Connectivity for Builds

By default, Cloud Build executes jobs in a shared, Google-managed public worker pool with dynamic public IP addresses. However, enterprise compliance often requires build workers to access private corporate resources without exposing them to the public internet.

+-----------------------------------------------------------------------------------+
|                   CLOUD BUILD PRIVATE POOL VPC CONNECTIVITY                       |
+-----------------------------------------------------------------------------------+
| Google-Managed Service Project                 Customer VPC Network               |
| ┌──────────────────────────────┐              ┌─────────────────────────────────┐ |
| │ Cloud Build Private Pool     │ Private      │ Custom VPC (e.g., 10.50.0.0/16) │ |
| │ (Dedicated, Isolated Worker  │ Service      │  ├── Private GKE Control Plane  │ |
| │  Compute Instances)          │ Access (PSA) │  ├── Internal Artifact Server   │ |
| │  [Worker 1]    [Worker 2]    │<============>│  ├── Cloud SQL (Private IP)     │ |
| └──────────────────────────────┘ (VPC Peering)│  └── Cloud Interconnect / VPN   │ |
|                                               │      └── On-Prem Datacenter     │ |
|                                               └─────────────────────────────────┘ |
+-----------------------------------------------------------------------------------+

Architectural Features of Private Pools

• Private Service Access (PSA): Private Pools communicate with customer VPC networks through a private VPC peering connection established via Private Service Access.
• Access to Private Endpoints: Enables build steps to directly communicate with Private GKE Cluster Control Planes, private Cloud SQL databases, internal package servers (Artifactory, Nexus), and on-premises systems via Cloud VPN/Interconnect.
• Static Egress & No Public IPs: Private pool workers can be configured with noExternalIP: true and routed through customer Cloud NAT gateways for deterministic egress IP whitelisting.
• Custom Sizing & Concurrency: Configurable machine types (from e2-standard-2 to e2-highcpu-32 and GPU-attached workers) with dedicated disk capacities up to 1000 GB.

---

Artifact Registry & Container Security

Artifact Registry is Google Cloud's enterprise package repository management service, superseding legacy Container Registry (GCR).

+-----------------------------------------------------------------------------------+
|                     ARTIFACT REGISTRY ENTERPRISE CAPABILITIES                     |
+-----------------------------------------------------------------------------------+
| 1. UNIVERSAL FORMATS    | Docker/OCI Containers, Maven, npm, Python (pip),        |
|                         | Apt (Debian), Yum (RPM), Helm Charts, Go Modules.       |
+-------------------------+---------------------------------------------------------+
| 2. GRANULAR IAM         | Project-level, repository-level, and package-level IAM. |
+-------------------------+---------------------------------------------------------+
| 3. CMEK & ENCRYPTION    | Customer-Managed Encryption Keys via Cloud KMS.         |
+-------------------------+---------------------------------------------------------+
| 4. REMOTE & VIRTUAL     | Upstream caching proxy for Docker Hub / PyPI;           |
|    REPOSITORIES         | Virtual single endpoints merging local + remote repos.  |
+-----------------------------------------------------------------------------------+

Vulnerability Scanning & Container Analysis API

Artifact Registry integrates with the Container Analysis API to deliver continuous vulnerability management:

1. On-Push Scanning: When a container image is pushed to Artifact Registry, Container Analysis automatically unpacks the image layers, identifies OS packages (Debian, Alpine, RHEL) and application language packages (Java, Node.js, Python, Go), and correlates them against the National Vulnerability Database (NVD).
2. Continuous Vulnerability Re-Analysis: When a new Common Vulnerabilities and Exposures (CVE) identifier is published, Container Analysis re-evaluates all existing images in the repository without requiring re-builds or re-pushes, immediately alerting security teams to newly vulnerable running containers.
3. Software Bill of Materials (SBOM): Automatically extracts and exports standardized SBOMs (in SPDX or CycloneDX formats) detailing every dependency, binary, and license embedded in the container.

---

Binary Authorization: Cryptographic Software Supply Chain Security

Binary Authorization is an enterprise admission control security service for Google Kubernetes Engine (GKE) and Cloud Run. It guarantees that only verified, cryptographically signed container images can run in production environments.

+-----------------------------------------------------------------------------------+
|                     BINARY AUTHORIZATION ADMISSION WORKFLOW                       |
+-----------------------------------------------------------------------------------+
| [CI/CD: Cloud Build]                                                              |
|   1. Build container image & push to Artifact Registry.                           |
|   2. Run SAST / Security / Container Analysis Vulnerability Scan.                 |
|   3. IF Vulnerability Scan passes:                                                |
|      Cloud KMS signs an Attestation -> Stored in Container Analysis Note.         |
|                                                                                   |
| [Runtime: GKE Cluster / Cloud Run]                                                |
|   4. Deploy request submitted (`kubectl apply` / Cloud Deploy rollout).           |
|   5. Binary Authorization Admission Controller intercepts request:                |
|      ├── Verifies cryptographic signature against trusted Attestor public key.    |
|      ├── Valid Signature Found ──> [ ALLOW DEPLOYMENT ]                           |
|      └── Missing / Invalid Signature ──> [ DENY DEPLOYMENT ]                      |
+-----------------------------------------------------------------------------------+

Binary Authorization Architecture Components

1. Policy: A project-level configuration defining the rules for container deployment (e.g., allow all, deny all, require attestations from specific attestors, or whitelist specific repository paths).
2. Attestor: A named authority (e.g., qa-security-attestor or vuln-scan-attestor) that validates whether a requirement has been satisfied. Attestors are associated with a Container Analysis Note and verify cryptographic signatures using asymmetric Cloud KMS public keys or PGP keys.
3. Attestation (Occurrence): A cryptographic digital signature generated by a CI/CD pipeline step (using Cloud KMS roles/cloudkms.cryptoKeyVersions.useToSign) binding a container image's SHA256 digest to an Attestor Note.
4. Breakglass Mechanism: In an emergency production incident (e.g., critical zero-day hotfix), engineers can bypass Binary Authorization enforcement by adding a break-glass annotation to the Pod specification. This allows the pod to deploy immediately while generating a high-severity security audit log in Cloud Logging for compliance post-mortems.

---

Cloud Deploy: Managed Continuous Delivery for GKE, Cloud Run & Anthos

Google Cloud Deploy is an opinionated, serverless continuous delivery service that automates the progression of container releases across structured multi-target environments.

+-----------------------------------------------------------------------------------+
|                     CLOUD DEPLOY MULTI-TARGET PROMOTION PIPELINE                  |
+-----------------------------------------------------------------------------------+
| [ Release Created: v1.4.0 (Skaffold Rendered Manifests & Images) ]                |
|                                                                                   |
|    ──> Target 1: dev-cluster (GKE / Cloud Run)                                    |
|        └── Rollout: Automatic deployment -> Validation Tests pass                 |
|                                                                                   |
|    ──> Target 2: staging-cluster (GKE / Cloud Run)                                |
|        └── Rollout: Automatic deployment -> Performance & Integration Tests       |
|                                                                                   |
|    ──> Target 3: prod-cluster (GKE Multi-Region)                                  |
|        └── [ MANUAL APPROVAL GATE: requireApproval: true ]                       |
|        └── Progressive Canary Rollout (25% -> 50% -> 100%)                        |
|        └── Automated Rollback on SLI breach                                       |
+-----------------------------------------------------------------------------------+

Core Cloud Deploy Entities & Concepts

• Delivery Pipeline: Defines the hierarchical progression of deployment targets (e.g., dev $\rightarrow$ staging $\rightarrow$ prod) and references a skaffold.yaml configuration.
• Target: Defines the destination runtime infrastructure (a GKE cluster endpoint, a Cloud Run service region, or an Anthos multi-cloud cluster) and associated execution service accounts.
• Release: An immutable snapshot of the delivery pipeline specification, rendered Kubernetes/Cloud Run manifests (using Skaffold with Kustomize or Helm), and container image digests. Releases are created once and promoted immutably through targets.
• Rollout: The actual deployment execution of a specific Release to an individual Target.

Cloud Deploy Architectural Advantages

1. Separation of Concerns: Developers create releases; Cloud Deploy orchestrates deployment mechanics with strict audit trails, eliminating the need to give broad cluster admin permissions to CI runners.
2. Approval Gates: Production targets can enforce manual approval policies (requireApproval: true), requiring designated SRE leads to approve promotions via the Cloud Console, gcloud CLI, or automated approval webhooks.
3. One-Click & Automated Rollback: If a deployment causes production errors, Cloud Deploy can instantly rollback (gcloud deploy rollouts rollback) to the previous stable release, re-applying known-good manifests within seconds.

---

Concrete Architectural Scenario: Secure Financial Microservice Pipeline

Scenario Profile

• Workload: Core payment authorization engine deployed across private GKE Autopilot clusters in us-east4 and us-central1.
• Security Mandate: Strict compliance with SLSA Level 3 supply chain security, automated vulnerability gatekeeping, no internet access during builds, and mandatory cryptographic release signing.

Solution Architecture Blueprint

1. Build Environment: Developers push code to GitHub Enterprise. A Cloud Build Trigger executes on a Cloud Build Private Pool connected via Private Service Access to the internal VPC. Build steps execute in parallel to compile binaries, run unit tests, and build Docker containers.
2. Container Registry & Scanning: The image is pushed to Artifact Registry. The Container Analysis API executes on-push vulnerability scanning. If no critical CVEs are detected, Cloud Build calls Cloud KMS to sign a cryptographic attestation for the image digest.
3. Continuous Delivery: Cloud Build creates a Cloud Deploy Release. Cloud Deploy rolls out the release to the dev target, auto-promotes to staging, and pauses at prod requiring SRE approval.
4. Admission Control: When the production rollout executes, GKE's Binary Authorization Admission Controller validates the Cloud KMS signature against the authorized Attestor policy, preventing any unauthorized or un-scanned container from launching.

---

[!IMPORTANT] Exam Watch: On the Google Cloud Architect exam, whenever a scenario mandates that build workers must access private on-premises databases or private GKE master control planes without public internet exposure, choose Cloud Build Private Pools with Private Service Access. When the requirement is to guarantee that only vulnerability-scanned, authorized images run in GKE, choose Binary Authorization with Container Analysis and Cloud KMS Attestors.