10.2 Cluster & Node-Level Troubleshooting

10.2 Cluster & Node-Level Troubleshooting

Quick Answer: Troubleshoot top-down. Confirm the control plane answers, then node health, then networking and DNS, then the workload. A large share of incidents reported as "my Pod is broken" are in fact a NotReady node, an exhausted disk, or a CNI failure. The state a Pod is stuck in names the layer at fault: Pending is scheduling, ImagePullBackOff is registry or credentials, CrashLoopBackOff is the application.

Section 10.1 covered probes and container-level debugging. This section covers everything above the Pod.

---

1. A Triage Order That Works

1. CONTROL PLANE   kubectl get --raw='/readyz?verbose'
                   kubectl get componentstatuses          (legacy but quick)
                   ↓ answers?
2. NODES           kubectl get nodes -o wide
                   kubectl describe node <n>   ← conditions + events + allocated resources
                   ↓ all Ready?
3. CLUSTER ADD-ONS kubectl get pods -n kube-system        ← CoreDNS, CNI, kube-proxy
                   ↓ all Running?
4. NAMESPACE       kubectl get events -n <ns> --sort-by=.lastTimestamp
                   kubectl get pods -n <ns> -o wide
                   ↓ narrowed to one workload?
5. WORKLOAD        kubectl describe pod / logs --previous / exec   (section 10.1)

Running step 5 first is the classic mistake: you read an application stack trace for twenty minutes before noticing the node it sits on has a full disk.

---

2. Node Conditions and What They Mean

kubectl describe node reports a condition block that is the fastest read in the whole system:

NotReady — What Actually Happened

Ready: Unknown means the kubelet stopped renewing its Lease in kube-node-lease. Common causes, in rough order of frequency:

1. The kubelet process crashed or is wedged — systemctl status kubelet, journalctl -u kubelet.
2. The container runtime (containerd/CRI-O) is down — the kubelet cannot report container status.
3. The CNI plugin is failing — the node cannot satisfy the Kubernetes networking model.
4. Network partition between node and control plane.
5. Resource exhaustion on the node making the kubelet unresponsive.
6. Expired certificates — the kubelet's client certificate lapsed and it can no longer authenticate.

When the node controller declares a node lost, Pods managed by controllers are recreated elsewhere; naked Pods and StatefulSet Pods with attached volumes are the slow cases, because a volume must be detached before it can be attached elsewhere.

---

3. Eviction Under Pressure

When a node runs low on memory or disk, the kubelet reclaims resources before the kernel OOM killer does something less discriminating. The eviction order follows QoS class, which is where section 4.1's material becomes operational:

Evicted first  ─────────────────────────────────────►  Evicted last
  BestEffort        Burstable (over its request)        Guaranteed
 (no requests)     (requests < limits)              (requests == limits)

Distinguish two lookalike failures:

The fix differs completely: eviction means the node is under-provisioned or workloads lack requests; OOMKilled means one container's limit is too low or it leaks.

---

4. Events Are the Highest-Value Surface

kubectl get events -A --sort-by=.lastTimestamp | tail -40
kubectl get events -n production --field-selector type=Warning

Events explain why the control plane did what it did — FailedScheduling with the exact predicate that failed, FailedMount, Unhealthy probe failures, NodeNotReady, Preempted.

The trap: Events are stored in etcd with a default TTL of about one hour. An incident investigated the next morning has no events left. Shipping events to a durable store (an events exporter into Loki or Elasticsearch) is standard practice in any cluster you have to operate.

---

5. Reading a Pending Pod

Pending always means "not yet placed or not yet started", and kubectl describe pod names the reason precisely:

Allocatable, not capacity. A node's schedulable resources are allocatable — capacity minus kube-reserved, system-reserved, and eviction thresholds. A "16-core" node does not offer 16 cores to workloads, which is why a Pod requesting 16 CPUs never schedules on it.

---

6. Control Plane and Add-On Failures

On a kubeadm cluster the control plane runs as static Pods, so it can be inspected with kubectl -n kube-system logs kube-apiserver-<node> — or, when the API is unreachable, directly on the node with crictl ps and crictl logs, which talk to the container runtime without going through Kubernetes at all.

---

7. Node-Level Commands

When kubectl cannot tell you enough, the answers are on the node:

systemctl status kubelet
journalctl -u kubelet -n 200 --no-pager
crictl ps -a                     # containers, independent of the API server
crictl logs <container-id>
df -h /var/lib/containerd /var/log   # DiskPressure is nearly always here
free -h

crictl is the CRI-level equivalent of docker ps, and it is the tool that still works when the API server is the thing that is broken.