6.1 Systematic Kubernetes Troubleshooting Methodology

6.1 Systematic Kubernetes Troubleshooting Methodology

Troubleshooting in Kubernetes is often perceived as daunting due to the distributed, multi-layered architecture of the platform. A single symptom—such as a web application returning an HTTP 502 Bad Gateway—can stem from a failure in physical networking, a corrupted CoreDNS configuration, an unhealthy Pod readiness probe, a saturated cgroup memory limit, an iptables forwarding bug in kube-proxy, or a control plane certificate expiration.

Without a structured diagnostic methodology, administrators waste critical time guessing at solutions. For the Certified Kubernetes Administrator (CKA) examination and enterprise operations, mastering a systematic, deterministic troubleshooting hierarchy is essential.

---

1. The 4-Tier Kubernetes Diagnostic Model

To rapidly isolate the root cause of an outage, divide the cluster into four distinct operational layers and evaluate them hierarchically:

+-----------------------------------------------------------------------------------------+
|                        THE 4-TIER KUBERNETES DIAGNOSTIC MODEL                           |
|                                                                                         |
|  [TIER 4: APPLICATION & WORKLOAD LAYER]                                                 |
|  - Pod Phase (Pending, Running, CrashLoopBackOff, OOMKilled, Completed)                 |
|  - Container Exit Codes (0, 1, 137, 139, 143), Logs (stdout/stderr), Probe Failures    |
|  - Manifest Syntax, Environment Variables, ConfigMaps, Secrets, Volume Mounts          |
|                                    ^                                                    |
|                                    |                                                    |
|  [TIER 3: NETWORKING & STORAGE SERVICES LAYER]                                          |
|  - ClusterIP, NodePort, LoadBalancer Services & Endpoints / EndpointSlices             |
|  - Ingress Controllers & Routing Rules, CoreDNS Resolution (/etc/resolv.conf)          |
|  - PersistentVolumes, PersistentVolumeClaims, StorageClasses, CSI Attachments           |
|                                    ^                                                    |
|                                    |                                                    |
|  [TIER 2: NODE AGENT & DATA PLANE RUNTIME]                                              |
|  - Kubelet Systemd Daemon & Configuration (/var/lib/kubelet/config.yaml)               |
|  - Container Runtime (containerd, CRI-O, crictl ps/logs), CNI Plugins (/etc/cni/net.d) |
|  - Host Resources (DiskPressure, MemoryPressure, PIDPressure, Kernel OOM Reaper)       |
|                                    ^                                                    |
|                                    |                                                    |
|  [TIER 1: CONTROL PLANE & CLUSTER INFRASTRUCTURE]                                       |
|  - kube-apiserver, etcd Database & Quorum, kube-controller-manager, kube-scheduler     |
|  - Static Pod Manifests (/etc/kubernetes/manifests), TLS PKI Certs (/etc/kubernetes/pki)|
|  - Administrative Kubeconfig Files (/etc/kubernetes/admin.conf, ~/.kube/config)         |
+-----------------------------------------------------------------------------------------+

Triage Flow Rules:

1. Bottom-Up for Infrastructure Failures: If kubectl commands fail entirely (The connection to the server <host>:6443 was refused), start immediately at Tier 1 (Control Plane & Infrastructure).
2. Top-Down for Workload Failures: If kubectl works but a deployment is not serving traffic, start at Tier 4 (Workload), verify Tier 3 (Service/DNS), and drop down to Tier 2 (Node/Kubelet) only if container-level inspection indicates host-level resource exhaustion or network isolation.

---

2. The Diagnostic Toolchain Hierarchy

Effective administrators transition seamlessly between different abstraction levels depending on where the fault resides:

| Level | Primary Tools | Typical Commands | When to Use | | :--- | :--- | :--- | | Level 1: Cluster & API Scope | kubectl | kubectl get nodes -o wide<br>kubectl get events -A --sort-by=.metadata.creationTimestamp<br>kubectl cluster-info | Initial situational awareness; identifying failing nodes, pending pods, and recent cluster-wide warning events. | | Level 2: Resource Inspection | kubectl | kubectl describe <resource> <name><br>kubectl get <resource> <name> -o yaml | Deep inspection of resource state, status conditions, controller events, selector mismatches, and spec definitions. | | Level 3: Container Log & Shell | kubectl | kubectl logs <pod> -c <container> --previous<br>kubectl exec -it <pod> -- /bin/sh<br>kubectl debug -it <pod> | Inspecting application runtime output, checking environment variables, testing local loopback networking, and examining filesystems. | | Level 4: Host Runtime (Node SSH) | crictl, nerdctl | crictl pods<br>crictl ps -a<br>crictl logs <container-id><br>crictl inspect <container-id> | Direct container runtime debugging when kubelet cannot report to the API server or when static pods are failing. | | Level 5: Host Systemd & OS | systemctl, journalctl, ip, ss | systemctl status kubelet containerd<br>journalctl -u kubelet -e --no-pager<br>dmesg -T \| grep -i oom<br>ss -tulpn | Low-level OS, daemon failure, kernel cgroup kills, memory/swap pressure, port conflicts, and disk exhaustion diagnostics. |

---

3. High-Yield Cluster Event Analysis

The Kubernetes Event subsystem records state transitions, errors, and controller decisions across the cluster. Because events are namespaced and have a default retention period of only 1 hour, querying them chronologically is one of the fastest diagnostic shortcuts:

# Stream all cluster-wide events sorted chronologically
kubectl get events -A --sort-by='.metadata.creationTimestamp'

# Filter specifically for Warning and Error events across all namespaces
kubectl get events -A --field-selector type=Warning --sort-by='.metadata.creationTimestamp'

# Inspect events for a specific namespace in real time
kubectl get events -n production --watch

Common Event Reason Codes & Root Causes:

---

4. Failure Domain Isolation Heuristic

When presented with a broken cluster or workload, follow this 5-step elimination algorithm:

[STEP 1: Check API Server Reachability]
  $ kubectl get nodes
  ├── SUCCESS: Control plane is alive -> Proceed to Step 2.
  └── FAILURE: Error connection refused / unauthorized
        └── Action: SSH to Control Plane Node -> Check kubelet, static pod manifests, and PKI certs.

[STEP 2: Check Node Statuses]
  $ kubectl get nodes -o wide
  ├── All Ready -> Proceed to Step 3.
  └── Node NotReady
        └── Action: $ kubectl describe node <node> -> SSH to node -> systemctl status kubelet containerd.

[STEP 3: Check Core Add-on Health]
  $ kubectl get pods -n kube-system
  ├── CoreDNS / CNI (Calico/Flannel) / kube-proxy Running -> Proceed to Step 4.
  └── Core Add-ons Crashing / Pending
        └── Action: Inspect CNI logs, CoreDNS ConfigMap, and network routing.

[STEP 4: Check Target Workload Status]
  $ kubectl get pods -n <ns> -o wide
  ├── Pending -> Check scheduler events (Taints, Affinities, Requests, PVs).
  ├── ContainerCreating / CrashLoopBackOff / OOMKilled -> Check describe, logs --previous, probes.
  └── Running (but traffic failing) -> Proceed to Step 5.

[STEP 5: Check Networking & Endpoint Plumbing]
  $ kubectl get svc,ep,endpointslices -n <ns>
  └── Endpoints <none> -> Label selector mismatch or Readiness probe failing.
  └── Endpoints populated -> Check targetPort, Ingress routing, and NetworkPolicies.

[!IMPORTANT] CKA Exam Triage Rules:

1. Always execute kubectl config use-context <context-name> before running any commands.
2. Check the namespace! If a resource is not found, verify if it was created in default instead of the requested namespace.
3. If you edit a manifest and kubectl apply errors due to immutable field restrictions (e.g., changing pod selectors on an existing Deployment), delete the resource with kubectl delete -f file.yaml --force --grace-period=0 and recreate it.