10.1 Container Troubleshooting & Health Probes

10.1 Container Troubleshooting & Health Probes

Quick Answer: Kubernetes automates workload self-healing using three health probes: Startup (protects initialization), Liveness (restarts unresponsive containers), and Readiness (controls traffic routing to Service Endpoints). When failures occur, diagnosing container states—such as CrashLoopBackOff, ImagePullBackOff, and OOMKilled (Exit Code 137)—requires a systematic troubleshooting workflow leveraging kubectl describe, kubectl logs -p, and kubectl exec.

In containerized environments, applications can fail in complex ways: processes might deadlock without exiting, memory leaks can exhaust node resources, or network dependencies may temporarily drop. Kubernetes provides native health checking mechanisms to detect these failures automatically and self-heal workloads, as well as a rich set of diagnostic tools to help engineers troubleshoot cluster issues.

---

Kubernetes Health Probes: Startup, Liveness, and Readiness

The kubelet uses three types of health probes to monitor container health continuously.

                       ┌───────────────────────────┐
                       │     Pod Starts Boot       │
                       └─────────────┬─────────────┘
                                     │
                                     ▼
                       ┌───────────────────────────┐
                       │   Startup Probe Active    │◄────────┐
                       └─────────────┬─────────────┘         │ Fails (retries up to
                                     │                       │  failureThreshold)
                             Success │                       │
                                     ▼                       │
                      ┌──────────────────────────────┐───────┘
                      │ Liveness & Readiness Active  │
                      └──────┬────────────────┬──────┘
                             │                │
              Liveness Fails │                │ Readiness Fails
                             ▼                ▼
               ┌───────────────────┐    ┌─────────────────────────┐
               │ Kubelet Kills &   │    │ Pod IP Removed from     │
               │ Restarts Container│    │ Service Endpoints       │
               └───────────────────┘    └─────────────────────────┘

The Three Health Probe Types

1. Startup Probe: Determines whether the application inside the container has successfully booted. All other probes (Liveness and Readiness) are disabled until the Startup probe succeeds. This prevents slow-starting legacy applications (e.g., processes running database schema migrations on startup) from being killed prematurely by the Liveness probe.
2. Liveness Probe: Determines whether the container process is running cleanly or has entered an unrecoverable state (such as a deadlock or infinite loop). If the Liveness probe fails, kubelet kills the container and triggers its restart policy (restartPolicy: Always).
3. Readiness Probe: Determines whether the container is ready to accept incoming network traffic. If the Readiness probe fails, the container is not killed; instead, the Pod's IP address is immediately removed from the Endpoints / EndpointSlices of all matching Kubernetes Services, preventing traffic from routing to an unready application.

Health Probe Comparison Matrix

Probe Handler Mechanisms & Configuration

Probes evaluate health through one of three handler mechanisms:

• httpGet: Performs an HTTP GET request on a specified path and port. Returns success if the HTTP response status code is greater than or equal to 200 and less than 400.
• tcpSocket: Attempts to open a TCP socket connection to a specified port. Returns success if the TCP handshake completes.
• exec: Executes a specified shell command inside the container. Returns success if the command exits with code 0.

apiVersion: v1
kind: Pod
metadata:
  name: web-app-probes
spec:
  containers:
  - name: nginx
    image: nginx:1.25
    ports:
    - containerPort: 80
    startupProbe:
      httpGet:
        path: /healthz
        port: 80
      initialDelaySeconds: 5
      periodSeconds: 10
      failureThreshold: 12 # Gives app up to 120s to boot
    livenessProbe:
      httpGet:
        path: /healthz
        port: 80
      periodSeconds: 15
      timeoutSeconds: 2
    readinessProbe:
      httpGet:
        path: /ready
        port: 80
      periodSeconds: 5

---

Pod Lifecycle Phases & Common Failure States

A Pod's high-level status is represented by its Phase, visible via kubectl get pods.

Pod Lifecycle Phases

• Pending: The Pod spec has been accepted by the API server, but one or more containers have not been created or scheduled (e.g., waiting on node assignment or downloading container images).
• Running: The Pod has been bound to a worker node, all containers have been created, and at least one container is running or currently starting/restarting.
• Succeeded: All containers in the Pod have terminated successfully with an exit code of 0 and will not be restarted (common for completed Job objects).
• Failed: All containers in the Pod have terminated, and at least one container terminated with a non-zero exit code.
• Unknown: The state of the Pod cannot be obtained, typically due to communication failure between the control plane and the node kubelet.

Common Container Failure States & Exit Codes

When a Pod fails to reach Running or Ready states, it enters specific error conditions:

┌───────────────────────┬────────────────────────────────────────────────────────┐
│ Failure State         │ Root Cause & Mechanism                                 │
├───────────────────────┼────────────────────────────────────────────────────────┤
│ CrashLoopBackOff      │ Container repeatedly starts, crashes (exit code != 0), │
│                       │ and is restarted by kubelet with exponential backoff    │
│                       │ delays (10s, 20s, 40s... up to 5 minutes).             │
├───────────────────────┼────────────────────────────────────────────────────────┤
│ ImagePullBackOff      │ Kubelet cannot retrieve the specified container image. │
│ (or ErrImagePull)     │ Causes: typo in image name/tag, missing image, or      │
│                       │ missing imagePullSecrets for private registries.       │
├───────────────────────┼────────────────────────────────────────────────────────┤
│ OOMKilled             │ Linux Kernel Out-Of-Memory killer terminates container │
│ (Exit Code 137)       │ exceeding `resources.limits.memory`.                   │
│                       │ Exit code math: 128 + Signal 9 (SIGKILL) = 137.        │
├───────────────────────┼────────────────────────────────────────────────────────┤
│ CreateContainerConfig │ Kubelet fails to assemble container configuration,     │
│ Error                 │ usually due to a missing ConfigMap or Secret reference.│
└───────────────────────┴────────────────────────────────────────────────────────┘

---

Systematic Kubernetes Troubleshooting Workflows

Diagnosing issues efficiently requires following a structured four-step diagnostic workflow using standard kubectl commands.

Step 1: Inspect Pod Status Overview

kubectl get pods -n <namespace> -o wide

Identifies Pod phase, restart counts, age, and assigned worker node IP.

Step 2: Inspect Kubernetes Events and Detailed Metadata

kubectl describe pod <pod-name> -n <namespace>

Displays the event log at the bottom of the output. Crucial for detecting image pull failures, probe execution errors, volume mount issues, and OOMKilled events.

Step 3: Analyze Container Log Streams

# Stream logs from current running container instance
kubectl logs <pod-name> -c <container-name> -n <namespace>

# Stream logs from previous crashed container instance
kubectl logs <pod-name> -c <container-name> --previous -n <namespace>

Analyzing --previous logs is vital when diagnosing containers stuck in CrashLoopBackOff to identify the unhandled application exception or stack trace that caused the crash.

Step 4: Interactive In-Container Shell Debugging

kubectl exec -it <pod-name> -c <container-name> -n <namespace> -- /bin/sh

Allows direct execution inside the running container to test network connectivity (curl, nslookup), inspect filesystem contents, and verify environment variable values.