3.1 Schema Evolution and gRPC Routing

Schema Evolution and Compatibility Rules

In microservice architectures, different services are deployed independently. A newer service version may send data to an older service version (forward compatibility), or an older client may call a newer service version (backward compatibility).

To preserve backward and forward compatibility without breaking production services, you must follow strict Protobuf schema evolution rules:

Rule 1: Never Change the Numeric Tag of an Existing Field

Field names are completely stripped during compilation and serialization. The binary payload on the wire contains only the numeric tag and the encoded value. If you change a field tag from 2 to 5, an older client sending tag 2 will result in the receiving service failing to parse the field or populating the wrong data.

Rule 2: Never Remove a Field Without Reserving Its Tag and Name

If you deprecate or delete a field (e.g., discount_code with tag 5), you must explicitly mark both the tag number and the field name as reserved:

reserved 5;
reserved "discount_code";

Exam Trap: If a developer deletes field 5 without reserving it, another developer may subsequently assign tag 5 to a new field (e.g., loyalty_points). When an older client sends a request containing the legacy discount_code string on tag 5, the new server will attempt to parse that string as loyalty_points (or decode unexpected bytes), leading to silent data corruption or deserialization errors. Declaring tags and names as reserved causes the Protobuf compiler (protoc) to fail the build if anyone attempts to reuse them.

Rule 3: Unknown Fields Are Preserved

In proto3, unknown fields are preserved by binary parsing/serialization in supporting proto3 implementations. Converting through JSON or copying only known fields can lose them. This allows intermediary microservices to route messages without dropping newly added fields.

Rule 4: Renaming Fields Does Not Break the Wire Protocol

Because field names are not transmitted over the wire, renaming a field in the .proto file (e.g., changing order_id to purchase_id while preserving tag 1) preserves binary field identity. It can still break generated source bindings or Protobuf JSON consumers, because those interfaces expose names.

---

The Four gRPC Communication Patterns

1. Unary RPC:
   Client -------- Request (Single Message) --------> Server
   Client <------- Response (Single Message) <------- Server

2. Server Streaming RPC:
   Client -------- Request (Single Message) --------> Server
   Client <======= Stream of Response Messages ====== Server

3. Client Streaming RPC:
   Client ======= Stream of Request Messages ======> Server
   Client <------- Response (Single Message) <------- Server

4. Bidirectional Streaming RPC:
   Client ======= Stream of Request Messages ======> Server
   Client <====== Stream of Response Messages ====== Server

• Unary RPC: Standard request-response pattern. Ideal for CRUD operations, authentication checks, and database lookups.
• Server Streaming RPC: The client sends a single request and receives a sequence of responses over a persistent stream. Ideal for real-time order status tracking, stock ticker feeds, and log streaming.
• Client Streaming RPC: The client streams a sequence of messages to the server and awaits a single summary response. Ideal for telemetry ingestion, IoT sensor batch uploads, and multi-part file transfers.
• Bidirectional Streaming RPC: Both client and server send a stream of messages concurrently using two independent read/write channels over a single multiplexed HTTP/2 connection. Ideal for real-time collaboration tools, multiplayer gaming, and conversational AI synchronization.

---

The Layer 4 vs. Layer 7 Load Balancing Challenge

A primary architectural trap when deploying gRPC microservices in Kubernetes or behind cloud load balancers is connection-level hotspotting caused by Layer 4 (L4) load balancers.

The Root Cause: HTTP/2 Persistent Multiplexing

In HTTP/1.1, clients open multiple short-lived TCP connections, and L4 balancers distribute these connections evenly across backends. In HTTP/2 and gRPC, however, a client establishes a single long-lived TCP connection to the backend and multiplexes hundreds of concurrent RPC calls across that single connection.

• What Happens with Layer 4 Balancing (e.g., Standard Kubernetes ClusterIP Service or TCP Proxy Load Balancer): The L4 load balancer only balances TCP connections. The single TCP connection from the client is routed to one backend pod. All hundreds of subsequent RPC calls travel over that same TCP connection, pinning 100% of the traffic to that single backend pod while remaining backend pods sit at 0% CPU utilization!

• The Solution: Layer 7 (Application) Load Balancing: An L7 load balancer (such as Google Cloud Application Load Balancers, an Envoy sidecar proxy, or a service mesh like Cloud Service Mesh / Istio) understands HTTP/2 frames. It unpacks individual multiplexed RPC requests from the incoming connection and distributes them individually across all available backend pods.

• Alternative Solution: Client-Side Load Balancing: The gRPC client library maintains a pool of TCP connections to all healthy backend endpoints (resolved via DNS or xDS control plane) and performs round-robin balancing; pick-first alone does not distribute calls across all endpoints directly in application memory.

---

Deploying and Configuring gRPC on Google Cloud

Deploying gRPC to Cloud Run

Cloud Run fully supports gRPC over HTTP/2 with automatic L7 request-level load balancing across auto-scaled container instances:

1. Edge TLS Termination: Cloud Run automatically handles TLS certificates and terminates TLS at Google's global edge infrastructure.
2. Cleartext HTTP/2 (h2c) Contract: Because Cloud Run terminates TLS at the edge, traffic forwarded from the edge proxy to your container instance is unencrypted cleartext HTTP/2 (h2c). The gRPC server running inside your container must be configured to accept cleartext HTTP/2 connections on the designated PORT (e.g., using net.Listen("tcp", ":" + port) in Go without TLS credentials).
3. Deployment Command:

gcloud run deploy order-grpc-service \
    --image=us-central1-docker.pkg.dev/my-project/apps/order-grpc:v1.0 \
    --use-http2 \
    --port=50051 \
    --region=us-central1

Deploying gRPC on GKE

• Configure the chosen GKE Gateway or Ingress controller for its supported HTTP/2 backend protocol. For GKE Ingress, the Service annotation cloud.google.com/app-protocols can select HTTP2; Gateway configuration uses its documented Service appProtocol values. These controller-specific mechanisms are not interchangeable.
• For east-west traffic between microservices within GKE, deploy Cloud Service Mesh (managed Istio) to provide automated mTLS, L7 request routing, distributed tracing injection, and client-side load balancing across pods without code changes.

Official references checked October 18, 2026: Google Cloud 1.