14.3 Progressive Delivery: Traffic Splitting, Canary, and Blue/Green Rollouts

Progressive Delivery: Traffic Splitting, Canary, and Blue/Green Rollouts

Releasing software into production environments introduces inherent operational risk. Even with rigorous automated testing and CI pipelines, subtle bugs, performance regressions, or database contention issues can escape into production. Progressive Delivery combines continuous delivery with advanced deployment strategies, traffic shifting, and real-time observability to expose new software versions to a small audience before full-scale release, minimizing the "blast radius" of potential defects.

For the Google Cloud Professional Cloud Developer exam, you must master the mechanics of Rolling updates, Blue/Green deployments, and Canary rollouts across both Google Kubernetes Engine and Cloud Run, including Cloud Service Mesh Layer 7 routing and automated rollback patterns.

---

Deployment Release Strategies: Rolling, Blue/Green & Canary

Choosing the correct deployment strategy involves balancing deployment speed, infrastructure cost, rollback complexity, and database schema compatibility:

Kubernetes Rolling Updates: maxSurge & maxUnavailable

The default Kubernetes Deployment strategy is the RollingUpdate. It replaces old pods with new pods incrementally according to two parameters declared in spec.strategy.rollingUpdate:

• maxSurge: The maximum number of pods that can be created above the desired replica count. Can be an absolute number (e.g., 2) or a percentage (e.g., 25%).
• maxUnavailable: The maximum number of pods that can be unavailable during the update process relative to the desired replica count.

spec:
  replicas: 20
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # Allows up to 25 pods during rollout (20 + 5)
      maxSurge: 25%
      # Ensures at least 20 pods remain available at all times (zero dropped capacity)
      maxUnavailable: 0

Exam Trap: During a Rolling Update, pods running version N and pods running version N+1 exist simultaneously in the cluster, processing requests from the same database. Any database schema modifications MUST be backward and forward compatible (using the Expand-Contract / Parallel Run design pattern). If a database migration drops a column required by version N, rolling updates will fail catastrophically.

Blue/Green Deployments

In a Blue/Green deployment, two identical environments exist:

• Blue: Current production environment serving 100% of live traffic.
• Green: Target environment running the newly deployed application version.

1. Blue is active (100% live user traffic).
2. Deploy version 2 to Green environment.
3. Run integration smoke tests against Green's private test endpoint.
4. Update Service label selector or Load Balancer backend from Blue to Green.
5. Traffic changes propagate asynchronously; existing connections and DNS caches can keep using old paths.
6. If errors occur, by changing routing revert the Service selector back to Blue.

In Kubernetes, the cutover is executed by modifying the Service's spec.selector:

## Request cutover to Green
kubectl patch service order-service -p '{"spec":{"selector":{"version":"green"}}}'

Canary Deployments

A Canary deployment exposes a new version to a small, controlled percentage of real user traffic (e.g., 5%) while 95% of traffic remains on the stable baseline. The canary version's telemetry—error rates, HTTP 5xx responses, transaction latencies, and Cloud Logging exceptions—is monitored continuously. If telemetry remains healthy, traffic is incrementally shifted (5% -> 25% -> 50% -> 100%). If anomalies occur, traffic can be routed back back to 100% baseline, impacting only a tiny fraction of requests.

---

Cloud Run Traffic Management & Tagged Revisions

Cloud Run provides built-in, serverless progressive delivery capabilities without requiring a service mesh or external load balancer.

Revision Tagging (--tag)

Every deployment to Cloud Run creates an immutable Revision. By applying traffic tags, developers can assign dedicated, deterministic URLs to specific revisions without assigning service-URL traffic to them; tag URLs still obey the service's IAM and ingress settings:

## Step 1: Deploy new revision with a 'canary' tag and 0% public traffic
gcloud run deploy payment-service \
    --image=us-central1-docker.pkg.dev/my-project/app-repo/payment-service:v2.0.0 \
    --region=us-central1 \
    --no-traffic \
    --tag=canary

Cloud Run provisions a dedicated URL for the revision: https://canary---payment-service-67abc-uc.a.run.app

Automated integration test suites can target this tagged URL directly to validate configuration using isolated test records and explicitly approved access; a tagged revision still shares backing resources unless separated before routing real end-user traffic.

Percentage-Based Traffic Splitting

Once pre-release verification succeeds, developers split traffic between the stable production revision and the canary revision using --to-revisions:

## Step 2: Route 90% to stable revision and 10% to canary revision
gcloud run services update-traffic payment-service \
    --region=us-central1 \
    --to-revisions=payment-service-00001-v1=90,payment-service-00002-v2=10

Rapid Zero-Downtime Rollback

If the canary revision introduces unexpected latency or exceptions, rolling back does not require rebuilding or redeploying container images. Because previous revisions remain available for routing (but may need new instances), routing 100% of traffic back to the previous revision takes effect in the control plane's propagation interval:

## Instant rollback to known-good revision
gcloud run services update-traffic payment-service \
    --region=us-central1 \
    --to-revisions=payment-service-00001-v1=100

Alternatively, to always route 100% of traffic to the latest successfully deployed revision:

gcloud run services update-traffic payment-service \
    --region=us-central1 \
    --to-latest

---

Advanced Layer 7 Routing with Cloud Service Mesh / Anthos Service Mesh

For microservices running on GKE, Cloud Service Mesh (managed Anthos Service Mesh based on open-source Istio) provides Layer 7 intelligent traffic routing via Envoy sidecar proxies deployed alongside application containers.

Cloud Service Mesh decouples traffic routing policies from application code using two primary Custom Resource Definitions (CRDs):

1. VirtualService: Defines routing rules, URI path matching, header matching, and destination weight splits.
2. DestinationRule: Defines policies applied to traffic once routed, including load balancing algorithms (round robin, least connection), connection pools, and pod subset labels.

Weight-Based Canary Routing Manifest

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: order-service-dest
spec:
  host: order-service
  subsets:
    - name: v1-stable
      labels:
        version: v1
    - name: v2-canary
      labels:
        version: v2
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: order-service-vs
spec:
  hosts:
    - order-service
  http:
    - route:
        - destination:
            host: order-service
            subset: v1-stable
          weight: 95
        - destination:
            host: order-service
            subset: v2-canary
          weight: 5

Header-Based & Cookie-Based Routing

Cloud Service Mesh allows routing requests based on HTTP headers, enabling selected cohorts identified through a trusted gateway or authenticated application logic to test new versions in production without impacting general users:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: order-service-vs
spec:
  hosts:
    - order-service
  http:
    # Rule 1: Route internal QA / Beta testers to canary based on header
    - match:
        - headers:
            x-release-channel:
              exact: canary
      route:
        - destination:
            host: order-service
            subset: v2-canary
    # Rule 2: Default rule for all other users routes to stable baseline
    - route:
        - destination:
            host: order-service
            subset: v1-stable

Fault Injection: Chaos Engineering for Microservices

Before releasing a new revision, developers can use Cloud Service Mesh to inject artificial faults into traffic destined for the service, validating that upstream calling services properly handle timeouts and circuit breaking:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: payment-service-fault
spec:
  hosts:
    - payment-service
  http:
    - fault:
        # Inject artificial 5-second delay into 10% of requests
        delay:
          percentage:
            value: 10.0
          fixedDelay: 5s
        # Inject HTTP 503 Service Unavailable into 5% of requests
        abort:
          percentage:
            value: 5.0
          httpStatus: 503
      route:
        - destination:
            host: payment-service
            subset: v2-canary

---

Automated Rollback Strategies with Cloud Monitoring

In enterprise progressive delivery pipelines, human operators should not manually watch dashboards during a canary release. Instead, deployment pipelines integrate with Cloud Monitoring alerting policies to implement deliberate automated rollback logic; Cloud Monitoring alone does not execute gcloud commands:

1. Baseline Telemetry Capture: Capture 5-minute pre-release baseline error rate (loadbalancing.googleapis.com/https/request_count filtered by response code class 5XX) and latency P99.
2. Canary Deployment: Shift 5% traffic to canary revision via Cloud Run or Cloud Service Mesh.
3. Automated Evaluation Gate: Execute a Cloud Build or Cloud Deploy verification step querying Cloud Monitoring metrics for the canary revision over a 10-minute window.
4. Trigger Rollback on SLO Violation: If the canary error rate exceeds 1% or latency P99 increases by more than 20%, an automated webhook triggers gcloud run services update-traffic --to-revisions=[STABLE_REV]=100, removing the canary's service-URL traffic after routing propagation; tag access and in-flight requests are separate.