3.5 Pod & Container Lifecycle, Restart Policies & Graceful Termination

3.5 Pod & Container Lifecycle, Restart Policies & Graceful Termination

Quick Answer: A Pod moves through Pending → Running → Succeeded/Failed, while each container inside it independently reports Waiting, Running, or Terminated. The Pod-level restartPolicy (Always, OnFailure, Never) decides whether the kubelet restarts a container that exits. Deleting a Pod triggers a graceful termination: endpoints are removed, the preStop hook fires, SIGTERM is delivered, and after terminationGracePeriodSeconds (default 30) anything still running receives SIGKILL.

Understanding this sequence is what separates "a rolling update caused 502s" from "a rolling update was invisible to users". KCNA asks about it directly, and the same knowledge underpins the deployment-strategy material later in this guide.

---

1. The Pod Lifecycle End to End

kubectl apply
     │
     ▼
[ API server: authn → authz → admission → persisted to etcd ]
     │
     ▼
PHASE: Pending ── scheduler binds Pod to a node
     │            kubelet pulls images, attaches volumes,
     │            runs init containers in order
     ▼
PHASE: Running ── app containers started; probes active
     │
     ├─► container exits 0 ──► restartPolicy decides
     ├─► container exits ≠0 ─► restartPolicy decides
     └─► deletion requested ─► graceful termination sequence
     ▼
PHASE: Succeeded (all containers exited 0, no restarts)
   or  Failed    (at least one container exited non-zero)

Pod Phase vs Container State

These two are frequently confused. The phase is a coarse, Pod-wide summary; the state is per container and far more useful for debugging.

A Pod can sit in phase Running while one of its containers is stuck in Waiting with CrashLoopBackOff. That is exactly why READY 1/2 shows up in kubectl get pods — the phase alone will not tell you.

---

2. Restart Policy

spec.restartPolicy is set on the Pod, applies to all containers in it, and cannot be overridden per container.

Exponential Back-Off

When a container keeps failing, the kubelet does not restart it in a tight loop. It applies exponential back-off starting at 10 seconds and doubling (10s, 20s, 40s, 80s, 160s) up to a cap of 5 minutes. The back-off counter resets after the container has run successfully for 10 minutes. A container caught in this loop is displayed with the reason CrashLoopBackOff — which is a symptom, never a root cause. The root cause is in kubectl logs --previous.

---

3. Init Containers and Sidecars

Init containers run sequentially, each to completion, before any application container starts. If one fails, the kubelet retries it according to the Pod's restartPolicy; with restartPolicy: Never the whole Pod goes to Failed. Typical uses: waiting for a dependency to answer, running a schema migration, pre-populating a shared emptyDir, or fetching a certificate.

Because init containers are separate containers, they can carry a different image and different privileges from the app — a classic security pattern is to give the init container the tooling (git, curl, psql) and keep the runtime image distroless.

Sidecar containers are the modern refinement. A sidecar is declared as an entry in initContainers with restartPolicy: Always, which makes it start before the app containers and keep running alongside them, shutting down after them. That ordering solves two long-standing problems: a log shipper that starts too late to catch startup logs, and a service-mesh proxy that dies before the app finishes draining.

spec:
  initContainers:
  - name: wait-for-db          # classic init container: runs to completion
    image: busybox:1.36
    command: ['sh','-c','until nc -z db 5432; do sleep 2; done']
  - name: log-shipper          # sidecar: starts first, stops last
    image: fluent/fluent-bit:3.0
    restartPolicy: Always
  containers:
  - name: app
    image: myapp:v2.1.0

---

4. Lifecycle Hooks

Two optional hooks let you run code at container boundaries. Each supports an exec command or an httpGet request.

• postStart fires immediately after the container is created. It runs asynchronously with the entrypoint, so there is no guarantee it completes before the main process starts. If it fails, the container is killed.
• preStop fires before SIGTERM is delivered and blocks the termination sequence until it finishes or the grace period expires. This is the hook that makes graceful shutdown work for applications that cannot be modified to trap SIGTERM.

---

5. Graceful Termination, Step by Step

When a Pod is deleted — by kubectl delete, by a rolling update, by an eviction — this sequence runs:

1. Pod marked Terminating; deletionTimestamp set
   ├─ 2a. Endpoints/EndpointSlice controller removes the Pod IP
   │      → kube-proxy and Ingress stop sending NEW traffic
   └─ 2b. kubelet runs the preStop hook (if defined)
3. SIGTERM sent to PID 1 of each container
4. ...grace period counts down (terminationGracePeriodSeconds, default 30)...
5. SIGKILL sent to anything still alive
6. Pod object removed from the API

Steps 2a and 2b run concurrently, and that concurrency is the source of the most common production bug in Kubernetes. Endpoint removal has to propagate to every node's kube-proxy and to every Ingress controller, which takes a moment. If the application exits the instant it receives SIGTERM, in-flight requests that were routed just before propagation completed are dropped — visible to users as intermittent 502s during every deploy.

The standard remedy is a short preStop sleep, which holds the container open long enough for endpoint removal to propagate before the process begins shutting down:

spec:
  terminationGracePeriodSeconds: 60
  containers:
  - name: web
    image: myapp:v2.1.0
    lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "sleep 10"]

Sizing rule: terminationGracePeriodSeconds must exceed the preStop duration plus the application's own drain time. Set it too low and SIGKILL truncates the drain you just paid for.