9.3 Service Mesh, mTLS & Zero-Trust Service Identity

9.3 Service Mesh, mTLS & Zero-Trust Service Identity

Quick Answer: A service mesh is a dedicated infrastructure layer for service-to-service communication. Its data plane is a fleet of proxies — typically Envoy, a CNCF graduated project — that intercept every request in and out of a workload. Its control plane watches the Kubernetes API and programs those proxies. The result is automatic mutual TLS, L7 traffic control, and uniform telemetry, all without touching application code. "Service mesh" is named explicitly in the CNCF cloud native definition, so KCNA expects you to know what it is and what problem it solves.

---

1. The Problem a Mesh Solves

Once a monolith becomes forty microservices, every service needs the same non-business capabilities: retries with backoff, timeouts, circuit breaking, mTLS, request tracing, and per-route authorisation. Implementing those in each service means:

• writing them once per language (Go, Java, Python, Node),
• upgrading them in lockstep across forty repositories,
• and accepting that the behaviour will drift anyway.

A mesh moves that logic out of the application and into the platform. The application makes a plain HTTP call to payments; the proxy beside it handles TLS, retries, timeouts, load balancing, and telemetry.

---

2. Data Plane and Control Plane

┌─────────────────── CONTROL PLANE ───────────────────┐
│  watches the Kubernetes API (Services, Endpoints,    │
│  routing CRDs, policies) and issues workload         │
│  certificates; pushes config to every proxy          │
└──────────────┬──────────────────────┬────────────────┘
               │ config + certs       │
     ┌─────────▼─────────┐  ┌─────────▼─────────┐
     │ Pod: orders       │  │ Pod: payments     │
     │ ┌──────┐ ┌──────┐ │  │ ┌──────┐ ┌──────┐ │
     │ │ app  │─│proxy │─┼──┼─│proxy │─│ app  │ │
     │ └──────┘ └──────┘ │  │ └──────┘ └──────┘ │
     └───────────────────┘  └───────────────────┘
                 └────── mTLS on the wire ──────┘

The application container is unaware any of it exists. Injection is done by a mutating admission webhook — the mechanism from section 5.3 — which appends the proxy container to the Pod spec at creation time.

---

3. Mutual TLS and Workload Identity

The headline security feature. In a plain cluster, Pod-to-Pod traffic is unencrypted and a receiving service has no cryptographic proof of who is calling — it sees only an IP address that may belong to a different workload a minute later.

A mesh gives every workload a short-lived X.509 certificate whose identity is derived from its ServiceAccount, and both proxies present certificates on every connection:

• Encryption in transit for all service-to-service traffic, automatically.
• Authentication of both ends — the caller is cryptographically identified, not merely IP-inferred.
• Automatic rotation, typically every 24 hours or less, with no application involvement.
• Identity-based authorization — policies reference serviceAccount: orders, not a CIDR.

This is what "zero trust" means concretely: the network is assumed hostile, and every request is authenticated and authorised regardless of where it originated. SPIFFE/SPIRE is the CNCF project that standardises this workload-identity format, and meshes commonly emit SPIFFE-compatible identities.

Enforcement modes usually run PERMISSIVE first — accepting both plaintext and mTLS during migration — before switching to STRICT.

---

4. Layer 7 Traffic Control

Because the proxy parses HTTP and gRPC, it can act on request semantics that layer 3/4 controls cannot see:

That last row is the crisp distinction from section 7.3: NetworkPolicy operates at L3/L4 and can only allow or deny a port. Method- and path-level authorisation requires a mesh (or a CNI-specific L7 policy such as CiliumNetworkPolicy).

---

5. Observability for Free

Because every request crosses a proxy, the mesh emits golden signals — request rate, error rate, latency distribution, and saturation — for every service pair, without instrumenting anything. It also propagates trace headers automatically, so a distributed trace shows the full call graph as long as applications forward the incoming headers.

The honest caveat: the mesh can propagate context but cannot create spans inside your application. Meaningful traces still require application-level instrumentation, typically with OpenTelemetry.

---

6. Sidecar vs Ambient

Sidecar meshes also historically suffered a lifecycle problem — the proxy could start after the app or die before it finished draining. The restartable sidecar container from section 3.5 (an initContainers entry with restartPolicy: Always) is the Kubernetes-native fix.

---

7. The Main Implementations

Do you need one? A mesh is real operational weight: another control plane, another certificate authority, another upgrade cycle, and a new class of failure. For a handful of services, application libraries and NetworkPolicy are usually enough. The mesh earns its cost at scale, under a compliance requirement for encryption in transit, or when progressive delivery needs precise traffic control.