PracticeVideosBlogFlashcardsEspañol

Free exam prep

CKA

Practice QuestionsStudy Guide

6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics

Key Takeaways

  • Pod startup transitions through distinct phases (Pending, ContainerCreating, Running, CrashLoopBackOff, OOMKilled, Error, ImagePullBackOff); each failure state points to a specific layer in the container lifecycle.
  • Container exit codes reveal the root cause of crashes: Exit Code 0 (clean exit, missing long-running foreground process), Exit Code 1 (application runtime exception), Exit Code 137 (SIGKILL / OOMKilled by cgroup or kernel), Exit Code 139 (Segmentation fault), Exit Code 143 (SIGTERM graceful termination).
  • Differentiating probe failures is critical: startupProbe failures prevent containers from ever reaching Ready, livenessProbe failures trigger recurring container restarts, and readinessProbe failures remove the pod from Service Endpoints without restarting the container.
  • InitContainers execute sequentially to completion before any app container starts; if an initContainer fails or loops, application containers remain in 'Init:CrashLoopBackOff' or 'Init:0/1'.
  • Advanced pod debugging leverages 'kubectl logs <pod> --previous' for crashed instances, ephemeral debug containers ('kubectl debug -it <pod> --image=busybox --target=<container>'), and host kernel ring buffers ('dmesg -T | grep -i oom').
Last updated: August 2026

6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics

Pods are the atomic scheduling units of Kubernetes. When a pod fails to reach or maintain the Running and Ready state, applications suffer downtime. Because pods encapsulate container runtime processes, Linux namespaces, cgroups, network plumbing, storage volume mounts, and health probes, diagnosing pod failures requires a structured, step-by-step approach.

1. The Pod Lifecycle State Machine & Failure Taxonomy

+-----------------------------------------------------------------------------------------+
|                         POD LIFECYCLE & FAILURE TAXONOMY                                |
|                                                                                         |
|  [SUBMITTED]                                                                            |
|      |                                                                                  |
|      v                                                                                  |
|  [PENDING] -------------> Failure: FailedScheduling (Insufficient resources, Taints)   |
|      |                                                                                  |
|      v                                                                                  |
|  [CONTAINERCREATING] ---> Failure: ImagePullBackOff / ErrImagePull (Registry/Tag typo) |
|      |               ---> Failure: FailedMount / MountVolume.SetUp (Secret/PVC missing) |
|      |               ---> Failure: Init:CrashLoopBackOff (InitContainer failed)         |
|      v                                                                                  |
|  [RUNNING]                                                                              |
|      |                                                                                  |
|      +---> [CRASHED / EXITED] ---> Exit Code 0 (Completed / No foreground process)     |
|      |                        ---> Exit Code 1 (App error / bad config / unhandled exc)|
|      |                        ---> Exit Code 137 (OOMKilled - cgroup limit exceeded)   |
|      |                        ---> Exit Code 139 (SIGSEGV - Segmentation Fault)        |
|      |                        ---> Exit Code 143 (SIGTERM - Graceful stop timeout)     |
|      v                                                                                  |
|  [CRASHLOOPBACKOFF] -----> Kubelet applies exponential backoff restart delay:          |
|                            10s -> 20s -> 40s -> 80s -> 160s -> max 300s (5 mins)        |
|                                                                                         |
|  [UNHEALTHY PROBES] -----> Liveness Failure: Container restarted by kubelet             |
|                     -----> Readiness Failure: Pod removed from Endpoints (No restart)   |
+-----------------------------------------------------------------------------------------+

Detailed Breakdown of Pod Failure States:

  1. ImagePullBackOff / ErrImagePull:

    • Symptoms: Container cannot be created because the container image cannot be retrieved.
    • Root Causes: Typo in image repository or tag (e.g., nginx:1.999), non-existent private repository, network failure communicating with registry, or missing/malformed imagePullSecrets.
    • Diagnostic: kubectl describe pod <name> -> Inspect Events.

  2. CrashLoopBackOff:

    • Symptoms: The container starts, executes for a brief period, crashes/terminates, and is repeatedly restarted by the kubelet with an exponential backoff penalty (10s, 20s, 40s, ..., up to 300s).
    • Root Causes: Missing environment variables, missing configuration files, application panics, failed database connection on startup, or running a base OS container (like ubuntu or busybox) without a long-running foreground command.
    • Diagnostic: kubectl logs <name> --previous and kubectl describe pod <name>.

  3. OOMKilled (Out Of Memory):

    • Symptoms: Pod state displays OOMKilled or CrashLoopBackOff with Exit Code: 137 and Reason: OOMKilled.
    • Root Causes: The container's memory usage exceeded the strict .spec.containers[].resources.limits.memory defined in its manifest, triggering the Linux cgroup memory controller to send SIGKILL (signal 9) to the process ($128 + 9 = 137$).
    • Diagnostic: kubectl describe pod <name> -> Last State: Terminated, Reason: OOMKilled, Exit Code: 137.

2. Container Exit Code Reference Table

When a container terminates, the Linux kernel and container runtime record an integer exit status. Interpreting this exit code provides immediate insight into why the process stopped:

Exit CodeSignal / TypeTechnical Meaning & Common Causes
0SuccessProcess completed cleanly. For long-running server pods, this indicates the container entrypoint finished (e.g., a batch script exited or daemon went to background).
1Application ErrorGeneral application runtime exception, syntax error in interpreted scripts (Python, Node), or unhandled fatal error.
126Cannot ExecuteContainer entrypoint or command found, but lacks executable permissions (chmod +x).
127Command Not FoundSpecified command or args binary does not exist inside the container filesystem (e.g., calling /bin/bash in an Alpine container that only has /bin/sh).
137SIGKILL (128 + 9)Process immediately terminated by external signal 9. Most commonly OOMKilled by cgroup limit, or terminated forcefully by kubectl delete --force.
139SIGSEGV (128 + 11)Segmentation fault; application attempted to access unallocated memory or corrupted pointer in compiled binaries (C/C++, Go, Rust).
143SIGTERM (128 + 15)Graceful termination signal sent by Kubernetes during pod deletion or probe failure; container did not shut down before grace period expired.

3. Diagnosing Health Probe Failures

Kubernetes provides three distinct probe mechanisms. Misunderstanding the differences leads to severe misdiagnoses:

+-----------------------------------------------------------------------------------------+
|                               HEALTH PROBE MATRIX                                       |
|                                                                                         |
|  +-------------------+  +------------------------------------------------------------+  |
|  | STARTUP PROBE     |  | Protects slow-starting containers (e.g., legacy Java/JVM).  |  |
|  |                   |  | All other probes are DISABLED until startupProbe succeeds. |  |
|  |                   |  | Failure Action: Container is KILLED and RESTARTED.          |  |
|  +-------------------+  +------------------------------------------------------------+  |
|  +-------------------+  +------------------------------------------------------------+  |
|  | LIVENESS PROBE    |  | Detects runtime deadlocks or frozen processes.             |  |
|  |                   |  | Failure Action: Container is KILLED and RESTARTED.          |  |
|  +-------------------+  +------------------------------------------------------------+  |
|  +-------------------+  +------------------------------------------------------------+  |
|  | READINESS PROBE   |  | Detects if container is ready to serve network traffic.    |  |
|  |                   |  | Failure Action: Pod IP REMOVED from Service Endpoints.      |  |
|  |                   |  | (CONTAINER IS NOT RESTARTED!).                             |  |
|  +-------------------+  +------------------------------------------------------------+  |
+-----------------------------------------------------------------------------------------+

Common Probe Configuration Pitfalls:

  • Liveness Probe Too Aggressive: Setting initialDelaySeconds: 0 on a heavy application causes the liveness probe to fail before the app boots, entering an endless restart loop.
  • Readiness vs. Liveness Confusion: If an external database is down, the pod's readiness probe should fail (taking it out of service routing), but its liveness probe should NOT fail (restarting the pod won't fix an external database).
# Example of properly tuned probes
spec:
  containers:
  - name: web
    image: app:v1
    startupProbe:
      httpGet:
        path: /healthz
        port: 8080
      failureThreshold: 30
      periodSeconds: 10   # Allows up to 300s (5 mins) for initial startup
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 10
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      periodSeconds: 5
      failureThreshold: 2

4. InitContainer Failures

initContainers execute sequentially before application containers start. If an initContainer fails:

  • Pod status displays Init:0/1, Init:Error, or Init:CrashLoopBackOff.
  • Application containers never start.
# Check status of initContainers
kubectl describe pod <pod-name>

# Stream logs from the specific initContainer
kubectl logs <pod-name> -c <init-container-name>

5. Ephemeral Debug Containers & Kernel OOM Verification

Ephemeral Containers (kubectl debug)

In modern Kubernetes, distroless and minimal container images lack debugging utilities like curl, sh, or netstat. Ephemeral debug containers attach to an existing pod's namespaces:

# Attach an interactive debug container sharing the target pod's process namespace
kubectl debug -it target-pod --image=busybox:1.36 --target=app-container

# Create a copy of a broken pod with modified command or environment variables
kubectl debug target-pod -it --copy-to=debug-pod --container=app-container -- /bin/sh

Verifying Host-Level Kernel OOM Events

If Kubernetes events do not explicitly say OOMKilled, check the host Linux kernel ring buffer:

# SSH to worker node where pod was running
ssh node01

# Inspect dmesg for Linux OOM Killer invocations
dmesg -T | grep -i -E "oom-killer|killed process"
Loading diagram...
Pod Startup Failure & Exit Code Diagnostic Decision Tree
Test Your Knowledge

An administrator observes that a newly deployed pod payment-processor immediately transitions to CrashLoopBackOff. Running kubectl describe pod payment-processor shows Last State: Terminated, Reason: OOMKilled, and Exit Code: 137. What is the precise root cause and remediation?

A
B
C
D
Test Your Knowledge

A web application pod auth-service has been running for weeks. Suddenly, traffic to the application ceases, but running kubectl get pods shows auth-service has a status of Running with 0/1 READY and RESTARTS: 0. What is the most likely explanation?

A
B
C
D
Test Your Knowledge

An administrator writes a YAML manifest to run a temporary debugging utility: kubectl run debug-box --image=busybox. The pod is created, but immediately displays Completed and then cycles into CrashLoopBackOff with Exit Code: 0. What is the reason for this behavior?

A
B
C
D
Up Next

6.6 Workload Scheduling Failures & Resource Constraint Resolution

Continue learning