6.1 Systematic Kubernetes Troubleshooting Methodology

Q: A newly deployed Pod named analytics-worker remains stuck in the Pending state indefinitely without any containers starting. Which command provides the definitive root-cause explanation for why the pod was not scheduled?

kubectl describe pod analytics-worker and inspect the Events section at the bottom of the output. When a Pod is Pending and has not been scheduled to a node, no containers exist yet, meaning kubectl logs and crictl logs will yield no output. The kube-scheduler records predicate failures (such as Insufficient memory, node taint mismatches, or unschedulable volume constraints) as Kubernetes Events attached to the Pod object, visible via kubectl describe pod .

Q: An administrator observes that an application Pod has restarted 15 times over the last hour. What specific kubectl command syntax should be used to inspect the stdout/stderr messages generated by the crashed container immediately prior to its most recent restart?

kubectl logs -c --previous. The --previous (or -p) flag on kubectl logs instructs the API server and kubelet to return the log stream of the previous terminated container instance rather than the currently running or restarting container. kubectl exec cannot run inside a crashed container, and standard kubectl logs without --previous only displays the current container instance.

Key Takeaways

Kubernetes troubleshooting demands a top-down and bottom-up 4-tier triage model: Infrastructure & Node Layer, Control Plane & Core Daemons, Workload & Pod Lifecycle, and Service/Ingress Networking & Storage.
Diagnostic velocity relies on an established CLI tool hierarchy: triage with 'kubectl get/describe', drill down with 'kubectl logs' and 'kubectl exec', and drop into host-level 'crictl' and 'journalctl' when the control plane is unreachable.
Cluster event streams ('kubectl get events -A --sort-by=.metadata.creationTimestamp') reveal transient failures, scheduling rejections, image pull timeouts, and probe failures before they escalate.
Effective root-cause isolation requires separating the Control Plane (API server, etcd, scheduler, controllers) from the Data Plane (kubelet, container runtime, CNI, kube-proxy).
During time-constrained performance exams like the CKA, strict adherence to timeboxing (5–7 minutes per task) and systematic verification prevents compounding configuration errors.

Last updated: August 2026

6.1 Systematic Kubernetes Troubleshooting Methodology

Troubleshooting in Kubernetes is often perceived as daunting due to the distributed, multi-layered architecture of the platform. A single symptom—such as a web application returning an HTTP 502 Bad Gateway—can stem from a failure in physical networking, a corrupted CoreDNS configuration, an unhealthy Pod readiness probe, a saturated cgroup memory limit, an iptables forwarding bug in kube-proxy, or a control plane certificate expiration.

Without a structured diagnostic methodology, administrators waste critical time guessing at solutions. For the Certified Kubernetes Administrator (CKA) examination and enterprise operations, mastering a systematic, deterministic troubleshooting hierarchy is essential.

1. The 4-Tier Kubernetes Diagnostic Model

To rapidly isolate the root cause of an outage, divide the cluster into four distinct operational layers and evaluate them hierarchically:

+-----------------------------------------------------------------------------------------+
|                        THE 4-TIER KUBERNETES DIAGNOSTIC MODEL                           |
|                                                                                         |
|  [TIER 4: APPLICATION & WORKLOAD LAYER]                                                 |
|  - Pod Phase (Pending, Running, CrashLoopBackOff, OOMKilled, Completed)                 |
|  - Container Exit Codes (0, 1, 137, 139, 143), Logs (stdout/stderr), Probe Failures    |
|  - Manifest Syntax, Environment Variables, ConfigMaps, Secrets, Volume Mounts          |
|                                    ^                                                    |
|                                    |                                                    |
|  [TIER 3: NETWORKING & STORAGE SERVICES LAYER]                                          |
|  - ClusterIP, NodePort, LoadBalancer Services & Endpoints / EndpointSlices             |
|  - Ingress Controllers & Routing Rules, CoreDNS Resolution (/etc/resolv.conf)          |
|  - PersistentVolumes, PersistentVolumeClaims, StorageClasses, CSI Attachments           |
|                                    ^                                                    |
|                                    |                                                    |
|  [TIER 2: NODE AGENT & DATA PLANE RUNTIME]                                              |
|  - Kubelet Systemd Daemon & Configuration (/var/lib/kubelet/config.yaml)               |
|  - Container Runtime (containerd, CRI-O, crictl ps/logs), CNI Plugins (/etc/cni/net.d) |
|  - Host Resources (DiskPressure, MemoryPressure, PIDPressure, Kernel OOM Reaper)       |
|                                    ^                                                    |
|                                    |                                                    |
|  [TIER 1: CONTROL PLANE & CLUSTER INFRASTRUCTURE]                                       |
|  - kube-apiserver, etcd Database & Quorum, kube-controller-manager, kube-scheduler     |
|  - Static Pod Manifests (/etc/kubernetes/manifests), TLS PKI Certs (/etc/kubernetes/pki)|
|  - Administrative Kubeconfig Files (/etc/kubernetes/admin.conf, ~/.kube/config)         |
+-----------------------------------------------------------------------------------------+

Triage Flow Rules:

Bottom-Up for Infrastructure Failures: If kubectl commands fail entirely (The connection to the server <host>:6443 was refused), start immediately at Tier 1 (Control Plane & Infrastructure).
Top-Down for Workload Failures: If kubectl works but a deployment is not serving traffic, start at Tier 4 (Workload), verify Tier 3 (Service/DNS), and drop down to Tier 2 (Node/Kubelet) only if container-level inspection indicates host-level resource exhaustion or network isolation.

2. The Diagnostic Toolchain Hierarchy

Effective administrators transition seamlessly between different abstraction levels depending on where the fault resides:

Level	Primary Tools	Typical Commands	When to Use
Level 1: Cluster & API Scope	`kubectl`	`kubectl get nodes -o wide`; `kubectl get events -A --sort-by=.metadata.creationTimestamp`; `kubectl cluster-info`	Initial situational awareness; identifying failing nodes, pending pods, and recent cluster-wide warning events.
Level 2: Resource Inspection	`kubectl`	`kubectl describe <resource> <name>`; `kubectl get <resource> <name> -o yaml`	Deep inspection of resource state, status conditions, controller events, selector mismatches, and spec definitions.
Level 3: Container Log & Shell	`kubectl`	`kubectl logs <pod> -c <container> --previous`; `kubectl exec -it <pod> -- /bin/sh`; `kubectl debug -it <pod>`	Inspecting application runtime output, checking environment variables, testing local loopback networking, and examining filesystems.
Level 4: Host Runtime (Node SSH)	`crictl`, `nerdctl`	`crictl pods`; `crictl ps -a`; `crictl logs <container-id>`; `crictl inspect <container-id>`	Direct container runtime debugging when `kubelet` cannot report to the API server or when static pods are failing.
Level 5: Host Systemd & OS	`systemctl`, `journalctl`, `ip`, `ss`	`systemctl status kubelet containerd`; `journalctl -u kubelet -e --no-pager`; `dmesg -T \| grep -i oom`; `ss -tulpn`	Low-level OS, daemon failure, kernel cgroup kills, memory/swap pressure, port conflicts, and disk exhaustion diagnostics.

3. High-Yield Cluster Event Analysis

The Kubernetes Event subsystem records state transitions, errors, and controller decisions across the cluster. Because events are namespaced and have a default retention period of only 1 hour, querying them chronologically is one of the fastest diagnostic shortcuts:

# Stream all cluster-wide events sorted chronologically
kubectl get events -A --sort-by='.metadata.creationTimestamp'

# Filter specifically for Warning and Error events across all namespaces
kubectl get events -A --field-selector type=Warning --sort-by='.metadata.creationTimestamp'

# Inspect events for a specific namespace in real time
kubectl get events -n production --watch

Common Event Reason Codes & Root Causes:

Event Reason	Message Pattern	Root Cause / Immediate Action
`FailedScheduling`	`0/3 nodes available: 3 Insufficient memory`	Total requested memory exceeds node allocatable capacity; adjust requests or add nodes.
`FailedScheduling`	`0/3 nodes available: 3 node(s) had untolerated taint`	Workload lacks required toleration for master or custom tainted worker nodes.
`FailedMount`	`MountVolume.SetUp failed: secret 'db-creds' not found`	Referenced Secret or ConfigMap does not exist in the workload's namespace.
`FailedAttachVolume`	`Multi-Attach error for volume "pvc-xxxx"`	ReadWriteOnce volume is still attached to a previous node that crashed or has not released the lock.
`BackOff`	`Back-off restarting failed container`	Application crashed on startup; inspect `kubectl logs <pod> --previous`.
`Unhealthy`	`Liveness probe failed: HTTP probe failed with statuscode: 500`	Application internal healthcheck failed; container will be restarted by kubelet.

4. Failure Domain Isolation Heuristic

When presented with a broken cluster or workload, follow this 5-step elimination algorithm:

[STEP 1: Check API Server Reachability]
  $ kubectl get nodes
  ├── SUCCESS: Control plane is alive -> Proceed to Step 2.
  └── FAILURE: Error connection refused / unauthorized
        └── Action: SSH to Control Plane Node -> Check kubelet, static pod manifests, and PKI certs.

[STEP 2: Check Node Statuses]
  $ kubectl get nodes -o wide
  ├── All Ready -> Proceed to Step 3.
  └── Node NotReady
        └── Action: $ kubectl describe node <node> -> SSH to node -> systemctl status kubelet containerd.

[STEP 3: Check Core Add-on Health]
  $ kubectl get pods -n kube-system
  ├── CoreDNS / CNI (Calico/Flannel) / kube-proxy Running -> Proceed to Step 4.
  └── Core Add-ons Crashing / Pending
        └── Action: Inspect CNI logs, CoreDNS ConfigMap, and network routing.

[STEP 4: Check Target Workload Status]
  $ kubectl get pods -n <ns> -o wide
  ├── Pending -> Check scheduler events (Taints, Affinities, Requests, PVs).
  ├── ContainerCreating / CrashLoopBackOff / OOMKilled -> Check describe, logs --previous, probes.
  └── Running (but traffic failing) -> Proceed to Step 5.

[STEP 5: Check Networking & Endpoint Plumbing]
  $ kubectl get svc,ep,endpointslices -n <ns>
  └── Endpoints <none> -> Label selector mismatch or Readiness probe failing.
  └── Endpoints populated -> Check targetPort, Ingress routing, and NetworkPolicies.

Important

CKA Exam Triage Rules:

Always execute kubectl config use-context <context-name> before running any commands.
Check the namespace! If a resource is not found, verify if it was created in default instead of the requested namespace.
If you edit a manifest and kubectl apply errors due to immutable field restrictions (e.g., changing pod selectors on an existing Deployment), delete the resource with kubectl delete -f file.yaml --force --grace-period=0 and recreate it.

Loading diagram...

Systematic 4-Tier Kubernetes Troubleshooting Workflow

Test Your Knowledge

An administrator executes kubectl get pods and receives the following error message: The connection to the server 192.168.1.100:6443 was refused - did you specify the right host or port?. According to systematic troubleshooting methodology, what is the most appropriate initial diagnostic step?

SSH directly into the control plane host at 192.168.1.100, inspect systemctl status kubelet, and check container runtime logs to determine if kube-apiserver is running.

Inspect the CoreDNS ConfigMap in namespace kube-system to verify if cluster DNS resolution is failing.

Delete and recreate the cluster's CNI DaemonSet to restore pod network connectivity.

Execute kubectl get events -A from a worker node to locate recent API server warnings.

Test Your Knowledge

A newly deployed Pod named analytics-worker remains stuck in the Pending state indefinitely without any containers starting. Which command provides the definitive root-cause explanation for why the pod was not scheduled?

kubectl logs analytics-worker --previous

crictl logs analytics-worker on the primary control plane node

kubectl describe pod analytics-worker and inspect the Events section at the bottom of the output

journalctl -u kube-apiserver -n 50 on the master node

Test Your Knowledge

An administrator observes that an application Pod has restarted 15 times over the last hour. What specific kubectl command syntax should be used to inspect the stdout/stderr messages generated by the crashed container immediately prior to its most recent restart?

kubectl logs <pod-name> --all-namespaces --tail=100

kubectl logs <pod-name> -c <container-name> --previous

kubectl describe pod <pod-name> --show-events

kubectl exec -it <pod-name> -- cat /var/log/app.log

Sections you finish are checked off in the contents.

Up Next

6.2 Node Failure, Kubelet Diagnostics & Systemd Services

Continue learning