6.3 Control Plane Failures, Static Pods & Component Logs

6.3 Control Plane Failures, Static Pods & Component Logs

The Kubernetes control plane is the brains of the cluster. When a control plane component fails—whether due to a corrupted configuration file, expired TLS certificates, an invalid command-line argument, or etcd unavailability—the entire cluster's declarative reconciliation engine breaks down. In severe cases, kubectl stops functioning entirely.

On the CKA examination and in enterprise disaster recovery scenarios, administrators must be capable of recovering control plane components directly from the host operating system without relying on high-level Kubernetes APIs.

---

1. Static Pod Mechanics & Manifest Lifecycle

In clusters provisioned with kubeadm, the four core control plane components operate as Static Pods:

1. kube-apiserver (/etc/kubernetes/manifests/kube-apiserver.yaml)
2. etcd (/etc/kubernetes/manifests/etcd.yaml)
3. kube-controller-manager (/etc/kubernetes/manifests/kube-controller-manager.yaml)
4. kube-scheduler (/etc/kubernetes/manifests/kube-scheduler.yaml)

+-----------------------------------------------------------------------------------------+
|                           STATIC POD RECONCILIATION ARCHITECTURE                        |
|                                                                                         |
|  Filesystem: /etc/kubernetes/manifests/                                                 |
|  +--------------------+  +---------------+  +---------------------+  +---------------+  |
|  | kube-apiserver.yaml|  |   etcd.yaml   |  |kube-controller-mgr  |  |kube-scheduler |  |
|  +---------+----------+  +-------+-------+  +----------+----------+  +-------+-------+  |
|            |                     |                     |                     |          |
|            v                     v                     v                     v          |
|  [Kubelet File Watcher] (monitors directory specified by staticPodPath in config.yaml)  |
|            |                                                                            |
|            v                                                                            |
|  [CRI gRPC Socket] -> containerd creates / restarts containers                          |
|            |                                                                            |
|            v                                                                            |
|  [Mirror Pod Creator] -> Creates read-only Mirror Pod in kube-apiserver (when API is up)|
+-----------------------------------------------------------------------------------------+

Critical Characteristics of Static Pods:

• Managed Solely by Kubelet: The local kubelet daemon directly watches /etc/kubernetes/manifests/. It creates, restarts, or deletes the corresponding containers when files are added, modified, or removed.
• No Scheduler Dependency: Static pods bypass kube-scheduler completely and are bound to the local node.
• Mirror Pods: Once kube-apiserver is running, the kubelet creates a read-only Mirror Pod in the kube-system namespace so status is visible via kubectl get pods -n kube-system. However, deleting a mirror pod with kubectl delete pod will not stop the static pod; the kubelet will immediately recreate the mirror pod.

[!CAUTION] Static Pod Directory Traps: If you create a backup file inside /etc/kubernetes/manifests/ (e.g., cp kube-apiserver.yaml kube-apiserver.yaml.bak), the kubelet will attempt to parse .bak as a valid static pod manifest, leading to container port collisions (e.g., multiple instances attempting to bind to port 6443). Always store backup manifests outside the static pod directory, such as in /tmp or /root/manifests_backup/.

---

2. Low-Level Control Plane Diagnostics with crictl and Host Logs

When kube-apiserver is down, running kubectl commands yields error messages such as: The connection to the server <host>:6443 was refused or Client.Timeout exceeded while awaiting headers.

In this state, you must SSH into the control plane host and utilize container runtime tools:

# 1. SSH into the Control Plane node and elevate to root
ssh master-node
sudo -i

# 2. Configure crictl endpoint if not already configured
cat <<EOF > /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

# 3. List all containers (including exited/crashed ones)
crictl ps -a

# 4. Filter for crashing control plane containers
crictl ps -a --name kube-apiserver
crictl ps -a --name etcd

# 5. Extract logs from a failed container instance
crictl logs <container-id>

Direct Log File Paths on Host Filesystem:

If crictl is unresponsive, inspect raw log files written to disk:

• Container Log Directory: /var/log/pods/kube-system_kube-apiserver-<node>_<uid>/
• Active symlinks: /var/log/containers/kube-apiserver-*.log
• Kubelet daemon systemd log: journalctl -u kubelet -e --no-pager -n 100

---

3. High-Frequency Control Plane Failure Modes & Fixes

Scenario 1: Manifest Syntax Corruption & Path Typos

An administrator modified /etc/kubernetes/manifests/kube-apiserver.yaml and introduced a typo in a command-line flag (e.g., --etcd-server instead of --etcd-servers) or invalid YAML indentation.

Diagnosis:

# Inspect kubelet logs for manifest processing errors
journalctl -u kubelet -e --no-pager | grep -i "error parsing manifest"
# Or inspect container termination reasons
crictl ps -a --name kube-apiserver
crictl logs <exited-container-id>

Output: unknown flag: --etcd-server

Remediation: Correct the flag in /etc/kubernetes/manifests/kube-apiserver.yaml. The kubelet detects file modification in inotify and immediately respawns the container.

---

Scenario 2: etcd Communication Failure (Port or Certificate Mismatch)

kube-apiserver cannot start because it cannot establish mutual TLS with etcd.

Diagnosis: crictl logs <apiserver-container-id> displays:

Error: context deadline exceeded
Failed to create client: connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"

Root Cause Checklist:

1. Check /etc/kubernetes/manifests/kube-apiserver.yaml parameters:
   • --etcd-servers=https://127.0.0.1:2379 (Verify IP and port)
   • --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
   • --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
   • --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
2. Verify the etcd container is alive: crictl ps --name etcd
3. Verify etcd port 2379 is listening: ss -tulpn | grep 2379

---

Scenario 3: Expired Control Plane TLS Certificates

Control plane certificates in /etc/kubernetes/pki/ have a 1-year lifespan. When they expire, all inter-component TLS handshakes fail.

+-----------------------------------------------------------------------------------------+
|                        CONTROL PLANE CERTIFICATE RENEWAL PIPELINE                       |
|                                                                                         |
|  1. CHECK EXPIRATION                                                                    |
|     $ kubeadm certs check-expiration                                                    |
|                                    |                                                    |
|                                    v                                                    |
|  2. RENEW ALL CERTIFICATES                                                              |
|     $ kubeadm certs renew all                                                           |
|     - Updates certificates in /etc/kubernetes/pki/                                      |
|     - Updates embedded client certs in /etc/kubernetes/*.conf                           |
|                                    |                                                    |
|                                    v                                                    |
|  3. UPDATE ADMIN KUBECONFIG                                                             |
|     $ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config                               |
|     $ chown $(id -u):$(id -g) $HOME/.kube/config                                        |
|                                    |                                                    |
|                                    v                                                    |
|  4. RESTART STATIC PODS                                                                 |
|     - Move manifests temporarily: mv /etc/kubernetes/manifests/*.yaml /tmp/             |
|     - Wait 10s -> Move back: mv /tmp/*.yaml /etc/kubernetes/manifests/                  |
|     - Or restart kubelet: systemctl restart kubelet                                     |
+-----------------------------------------------------------------------------------------+

---

4. kube-controller-manager & kube-scheduler Failures

When kube-apiserver is healthy but controllers fail:

• Symptoms: Pods remain in Pending indefinitely (kube-scheduler down); Deployments do not scale when updated (kube-controller-manager down).
• Diagnostics: Check their dedicated kubeconfigs located at /etc/kubernetes/controller-manager.conf and /etc/kubernetes/scheduler.conf.

# Verify scheduler static pod health
kubectl get pods -n kube-system -l component=kube-scheduler

# If crashing, check logs directly
kubectl logs -n kube-system kube-scheduler-<master-node>