1.5 Highly Available (HA) Control Plane Topologies
Key Takeaways
- High Availability (HA) in Kubernetes eliminates Single Points of Failure (SPOF) by running multiple redundant control plane nodes fronted by a Layer 4 Load Balancer.
- Stacked etcd co-locates etcd static pods on each control plane node, requiring fewer servers but coupling API server compute with etcd disk I/O.
- External etcd isolates etcd members onto dedicated standalone instances, maximizing performance and blast-radius protection at the cost of infrastructure overhead.
- Bootstrapping HA clusters with kubeadm requires specifying --control-plane-endpoint (LB DNS/VIP) and --upload-certs to automatically distribute PKI secrets across masters.
- Additional control plane nodes join using kubeadm join ... --control-plane --certificate-key <key>, which automatically adds the new member to the existing etcd Raft cluster.
1.5 Highly Available (HA) Control Plane Topologies
In enterprise production environments, a single-master Kubernetes deployment constitutes an unacceptable Single Point of Failure (SPOF). If the control plane host suffers hardware failure, kernel panics, or unrecoverable disk corruption, all cluster administration stops: scheduling halts, failed pods cannot be replaced, autoscaling fails, and the Kubernetes API becomes unresponsive (even though existing data plane pods may temporarily continue running).
To achieve High Availability (HA), Kubernetes deploys multiple redundant control plane nodes synchronized across independent physical fault domains. This section dissects the two primary HA control plane topologies, load-balancing mechanisms, certificate distribution, and multi-master consensus management.
1. Architectural Topologies: Stacked vs. External etcd
When designing an HA control plane, Kubernetes architects must choose between two distinct structural models for state persistence:
+---------------------------------------------------------------------------------------+
| TOPOLOGY COMPARISON |
| |
| [TOPOLOGY 1: STACKED ETCD TOPOLOGY] |
| |
| +--------------------------+ +--------------------------+ +---------------------+ |
| | CONTROL PLANE NODE 1 | | CONTROL PLANE NODE 2 | | CONTROL PLANE NODE 3| |
| | - kube-apiserver | | - kube-apiserver | | - kube-apiserver | |
| | - kube-controller-mgr | | - kube-controller-mgr | | - kube-controller-m| |
| | - kube-scheduler | | - kube-scheduler | | - kube-scheduler | |
| | - etcd (local static pod| | - etcd (local static pod| | - etcd (local pod) | |
| +--------------------------+ +--------------------------+ +---------------------+ |
| \ | / |
| +---------------------------+---------------------------+ |
| | |
| [Raft Peer Sync: Port 2380] |
| |
| --------------------------------------------------------------------------------- |
| |
| [TOPOLOGY 2: EXTERNAL ETCD TOPOLOGY] |
| |
| +--------------------+ +--------------------+ +--------------------+ |
| | CONTROL PLANE CP1 | | CONTROL PLANE CP2 | | CONTROL PLANE CP3 | |
| | (apiserver/cm/sch) | | (apiserver/cm/sch) | | (apiserver/cm/sch) | |
| +--------------------+ +--------------------+ +--------------------+ |
| \ | / |
| +-------------------------+-------------------------+ |
| | (HTTPS Port 2379) |
| +-------------------------+-------------------------+ |
| / | \ |
| +--------------------+ +--------------------+ +--------------------+ |
| | DEDICATED ETCD ET1 |<===>| DEDICATED ETCD ET2 |<===>| DEDICATED ETCD ET3 | |
| +--------------------+ +--------------------+ +--------------------+ |
| (Port 2380 Peer Sync) |
+---------------------------------------------------------------------------------------+
Detailed Trade-Off Matrix:
| Architectural Attribute | Stacked etcd Topology | External etcd Topology |
|---|---|---|
| Infrastructure Footprint | Minimal: Requires 3 nodes minimum (Control Plane + etcd co-located). | High: Requires 6 nodes minimum (3 Control Plane nodes + 3 dedicated etcd nodes). |
| Resource Isolation | Coupled: etcd competes with API server and OS for memory and disk I/O. | Dedicated: etcd has isolated CPU, RAM, and dedicated fast NVMe SSD storage. |
| Blast Radius | High: Losing a control plane node simultaneously destroys an etcd cluster member. | Isolated: Control plane node crashes do not impact etcd quorum or storage stability. |
| Operational Complexity | Low: Fully automated deployment, member join, and cert lifecycle via kubeadm. | High: Requires independent clustering scripts, manual PKI distribution, and maintenance. |
| Scaling Model | Tied 1:1. Adding an API server node requires adding an etcd member. | Decoupled. Scale API servers horizontally (e.g., 5 nodes) while keeping etcd at 3 nodes. |
| Recommended Use Case | Standard on-premise and cloud production clusters up to 1,000 nodes. | Ultra-high-scale enterprise clusters (1,000+ nodes) with extreme write throughput. |
2. Load Balancing the Control Plane: HAProxy & Keepalived
Because multiple kube-apiserver instances run actively across control plane nodes, worker nodes and human administrators must not point to individual node IP addresses. Instead, all traffic is routed through a single Control Plane Endpoint (Virtual IP / DNS Name) backed by a Layer 4 Load Balancer.
+-----------------------------------------------------------------------------+
| CONTROL PLANE LOAD BALANCING |
| |
| [Worker Nodes / kubectl CLI] |
| | |
| v |
| [Virtual IP (VIP): 192.168.1.100:6443 (Managed by Keepalived VRRP)] |
| | |
| v |
| [HAProxy TCP Load Balancer (Round Robin / Leastconn)] |
| | |
| +-------+-------+----------------------------------+ |
| | (TCP 6443) | (TCP 6443) | (TCP 6443) |
| v v v |
| +-----------+ +-----------+ +-----------+ |
| | Control | | Control | | Control | |
| | Plane CP1 | | Plane CP2 | | Plane CP3 | |
| +-----------+ +-----------+ +-----------+ |
+-----------------------------------------------------------------------------+
Sample /etc/haproxy/haproxy.cfg Configuration:
frontend kubernetes-frontend
bind 192.168.1.100:6443
mode tcp
option tcplog
default_backend kubernetes-backend
backend kubernetes-backend
mode tcp
option tcp-check
balance roundrobin
server cp1 192.168.1.11:6443 check fall 3 rise 2
server cp2 192.168.1.12:6443 check fall 3 rise 2
server cp3 192.168.1.13:6443 check fall 3 rise 2
[!NOTE] Layer 4 Passthrough: The load balancer must operate in Layer 4 TCP passthrough mode (not Layer 7 HTTP termination) so that mTLS client certificates passed by
kubectlandkubeletterminate directly onkube-apiserver.
3. Multi-Master Bootstrapping with Kubeadm (--upload-certs)
In older Kubernetes releases, joining secondary control plane nodes required manual SCP copying of PKI certificates (ca.crt, ca.key, sa.key, etcd/ca.crt, etcd/ca.key). Modern kubeadm automates this using the Automated Certificate Upload Mechanism.
+-----------------------------------------------------------------------------+
| KUBEADM HA CERTIFICATE DISTRIBUTION FLOW |
| |
| [PRIMARY MASTER: CP1] |
| $ kubeadm init --control-plane-endpoint "lb.k8s.lan:6443" --upload-certs |
| |
| 1. Generates PKI CA, etcd, and SA keys. |
| 2. Generates random 32-byte AES decryption key (Certificate Key). |
| 3. Encrypts certificates with Key -> Saves to Secret in kube-system |
| (Secret: kubeadm-certs, TTL: 2 Hours). |
| 4. Prints Join Command with --control-plane and --certificate-key. |
| |
| [SECONDARY MASTER: CP2 / CP3] |
| $ kubeadm join lb.k8s.lan:6443 --token <tok> --discovery-token-ca-cert-hash \
| sha256:<hash> --control-plane --certificate-key <key> |
| |
| 1. Fetches encrypted Secret from CP1 via bootstrap token. |
| 2. Decrypts certs locally using --certificate-key. |
| 3. Adds local etcd member to Raft cluster via peer port 2380. |
| 4. Spawns local apiserver, controller-manager, scheduler static pods. |
+-----------------------------------------------------------------------------+
Step-by-Step Multi-Master Initialization Workflow:
Step 1: Initialize Primary Master (CP1)
sudo kubeadm init \
--control-plane-endpoint "k8s-lb.production.lan:6443" \
--upload-certs \
--pod-network-cidr=10.244.0.0/16
The output displays two distinct join commands:
- One for joining additional control plane nodes (contains
--control-planeand--certificate-key). - One for joining worker nodes (standard token and hash only).
Step 2: Joining Secondary Control Plane Nodes (CP2 and CP3)
Execute on CP2 and CP3:
sudo kubeadm join k8s-lb.production.lan:6443 \
--token 9v7q8z.1234567890abcdef \
--discovery-token-ca-cert-hash sha256:7f8e9d0a... \
--control-plane \
--certificate-key e4a1b2c3d4e5f60718293a4b5c6d7e8f90123456789abcdef0123456789a
Handling Expired Certificate Keys (Past 2-Hour Window):
If more than 2 hours have elapsed, the uploaded Secret is automatically garbage-collected. Generate a fresh key and re-upload certificates with:
sudo kubeadm init phase upload-certs --upload-certs
4. Multi-Master Consensus & Leader Election
While all kube-apiserver instances run Active-Active behind the load balancer, components with single-writer reconciliation logic (kube-controller-manager and kube-scheduler) run in Active-Passive mode using Lease Objects (coordination.k8s.io).
# Inspecting active leader leases:
kubectl get leases -n kube-system
NAME HOLDER AGE
kube-controller-manager cp-node-1.lan_a8b2... 14m
kube-scheduler cp-node-2.lan_c9d4... 14m
If cp-node-1 fails, its lease renewal heartbeat fails. Within 15 seconds (--leader-elect-lease-duration), standby controller managers on cp-node-2 or cp-node-3 acquire the lease and seamlessly resume reconciliation loops.
An administrator is bootstrapping a 3-node HA control plane cluster using kubeadm. Three hours after running kubeadm init on the first master, the administrator attempts to join the second master node, but the command fails with an error indicating the certificate encryption key cannot find the target secret. What is the cause and resolution?
Which of the following describes the key operational difference between stacked etcd and external etcd HA topologies in Kubernetes?
In a High Availability Kubernetes cluster with three control plane nodes fronted by an HAProxy load balancer, how do the kube-controller-manager and kube-scheduler daemons coordinate to prevent conflicting state reconciliations?