14.2 Container Health Probes, Lifecycle Hooks & Graceful Shutdown

Container Health Probes, Lifecycle Hooks & Graceful Shutdown

In modern cloud-native architectures, container instances are ephemeral. Pods are routinely terminated, rescheduled, and scaled up or down during rolling updates, node auto-repairs, horizontal scaling events, and spot VM preemptions. To prevent dropped connections, corrupted state, and user-facing HTTP 502/503 errors, developers must deeply understand container health checking mechanisms, the container termination lifecycle, and operating system signal handling.

For the Google Cloud Professional Cloud Developer exam, you must know how to configure Kubernetes Startup, Liveness, and Readiness probes, implement preStop hooks, handle OS signals (SIGTERM vs. SIGKILL) in application code, and adhere to Cloud Run's specific shutdown windows.

---

Kubernetes Health Checking Probes: Startup, Liveness & Readiness

Kubernetes provides three distinct probe mechanisms executed periodically by the local kubelet agent on each node:

The Startup Probe: Preventing Premature Death Loops

Applications such as enterprise Java frameworks, large Python ML models, or legacy monoliths may take 60 to 180 seconds to initialize before they can respond to health checks. Historically, configuring a high initialDelaySeconds on a liveness probe risked delaying the detection of real deadlocks.

The Startup Probe completely solves this problem: while the startup probe is executing, all liveness and readiness checks are disabled. The application is given a generous time budget (failureThreshold * periodSeconds) to initialize:

spec:
  containers:
    - name: api-service
      image: us-central1-docker.pkg.dev/my-project/app-repo/api-service:v1.0
      # 1. Startup Probe: Allows up to 30 * 10s = 300 seconds to initialize
      startupProbe:
        httpGet:
          path: /healthz/startup
          port: 8080
        periodSeconds: 10
        failureThreshold: 30
      # 2. Liveness Probe: Activated only AFTER startupProbe succeeds
      livenessProbe:
        httpGet:
          path: /healthz/liveness
          port: 8080
        periodSeconds: 10
        failureThreshold: 3
      # 3. Readiness Probe: Controls traffic routing into the container
      readinessProbe:
        httpGet:
          path: /healthz/readiness
          port: 8080
        periodSeconds: 5
        failureThreshold: 2

Liveness vs. Readiness: The Critical Exam Trap

Exam Trap: Never check external dependencies (such as databases, third-party APIs, or Redis caches) inside a Liveness Probe. If dependency failures cross the configured liveness threshold, many containers can restart and worsen the outage. Use application circuit breakers and carefully designed readiness checks; making every Pod unready for a shared dependency can also remove all serving capacity. Choose the behavior from the service's ability to serve degraded responses, rather than treating any dependency ping as a universal health test.

Probe Handler Mechanisms

Kubernetes supports HTTP, TCP, exec, and supported gRPC probe mechanisms:

1. httpGet: Kubelet performs an HTTP GET request to the specified IP, port, and path. HTTP status codes from 200 through 399 indicate success; codes 400+ indicate failure.
2. tcpSocket: Kubelet attempts to open a TCP socket connection to the specified port. If the three-way handshake succeeds, the probe succeeds.
3. grpc: Checks the standard gRPC Health Checking Protocol where supported.
4. exec: Kubelet executes a diagnostic shell command inside the container namespace. An exit code of 0 indicates success; any non-zero exit code indicates failure.

---

Container Termination Lifecycle: preStop, SIGTERM & SIGKILL

When a pod is deleted (via manual kubectl delete, HPA scale-down, or rolling update), Kubernetes executes a strict, multi-step decommissioning sequence to ensure zero-downtime draining:

1. API Server marks Pod status as 'Terminating'
                   |
                   v
2. TWO ACTIONS EXECUTE CONCURRENTLY:
   +---------------------------------------+   +---------------------------------------+
   | Service Endpoint Removal              |   | Kubelet executes preStop Hook         |
   | * Pod IP removed from Endpointslices  |   | * Synchronous execution inside        |
   | * iptables / IPVS / Envoy proxies     |   |   container namespace (e.g. sleep 10) |
   |   stop routing new traffic            |   | * Blocks SIGTERM dispatch until done  |
   +---------------------------------------+   +---------------------------------------+
                   |
                   v
3. Kubelet sends SIGTERM (Signal 15) to PID 1 inside container
                   |
                   v
4. The grace countdown ALREADY started before preStop (Default total: 30s)
   * Application drains in-flight requests
   * Closes database connection pools
   * Flushes logs, traces, and metrics
                   |
                   v
5. If process has not exited before grace period expires:
   Kubelet sends SIGKILL (Signal 9, Exit Code 137) to forcefully terminate process

Why the preStop Hook is Crucial

In distributed Kubernetes clusters, removing a pod's IP from Service endpoints, Cloud Load Balancer backends, and kube-proxy iptables rules requires asynchronous propagation across all nodes. This propagation is asynchronous and depends on the controller and load-balancer configuration.

If the kubelet immediately sends SIGTERM to the container, the application may immediately close its HTTP listening socket while external load balancers and mesh proxies are still actively routing in-flight packets to that pod. Those in-flight requests encounter immediate TCP connection refused (RST) or HTTP 502 Bad Gateway errors.

Configuring a preStop hook introduces an intentional pause, giving the network routing mesh sufficient time to drain and detach the pod before the application begins its shutdown:

 lifecycle:
   preStop:
     exec:
       command: ["/bin/sh", "-c", "sleep 10"]

Exam Rule: If your container has a 10-second preStop sleep and your application requires 15 seconds to drain active requests, your total shutdown time is 25 seconds. Ensure that terminationGracePeriodSeconds (default 30 seconds) is configured higher than the sum of preStop plus application drain time (e.g., set terminationGracePeriodSeconds: 60).

---

Graceful Shutdown Implementation in Application Code

To achieve true zero-downtime deployments, application code must explicitly register signal listeners for SIGTERM and coordinate shutdown activities.

The Linux PID 1 Signal Forwarding Problem

In Linux operating systems, Process ID 1 (PID 1) receives special treatment from the kernel: it does not inherit default signal handlers. If a process running as PID 1 does not explicitly register a handler for SIGTERM, the Linux kernel simply drops the signal. The application never knows it is being terminated and runs until terminationGracePeriodSeconds expires, at which point the kernel sends SIGKILL.

Dockerfile Shell Form vs. Exec Form:

## BAD: Shell form - launches /bin/sh -c 'node app.js'
## /bin/sh becomes PID 1 and DOES NOT forward SIGTERM to node!
CMD node app.js

## GOOD: Exec form - launches node directly as PID 1
CMD ["node", "app.js"]

When running complex containers or multi-process binaries, developers should use lightweight init daemons such as tini or dumb-init to handle signal forwarding and reap zombie child processes.

Concrete Graceful Shutdown Pattern (Go)

package main

import (
	"context"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	server := &http.Server{Addr: ":8080"}

	// Channel to listen for OS termination signals
	stop := make(chan os.Signal, 1)
	signal.Notify(stop, syscall.SIGTERM, syscall.SIGINT)

	go func() {
		if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
			log.Fatalf("HTTP server error: %v", err)
		}
	}()
	log.Println("Server listening on port 8080")

	// Block until SIGTERM or SIGINT is received
	<-stop
	log.Println("Received SIGTERM signal. Initiating graceful shutdown...")

	// 1. Create a timeout context for the draining window
	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
	defer cancel()

	// 2. Stop accepting new requests and drain active in-flight requests
	if err := server.Shutdown(ctx); err != nil {
		log.Printf("HTTP shutdown error: %v", err)
	}

	// 3. Close database pools, message broker connections, and flush metrics
	closeDatabasePools()
	flushTelemetry(ctx)

	log.Println("Graceful shutdown complete. Exiting cleanly.")
}

---

Cloud Run Container Lifecycle & The 10-Second Shutdown Window

Cloud Run enforces a serverless execution model with lifecycle semantics tailored for request-driven compute:

Instance States: Starting, Serving, Throttled, Terminating

1. Starting: Cloud Run provisions the container sandbox. The application must listen for HTTP requests on the port specified by the PORT environment variable (default 8080) within the startup timeout limit (containers must start within four minutes; startup-probe timing is constrained by the documented startup budget, not the 60-minute request timeout).
2. Serving: The container actively processes concurrent requests up to its --concurrency limit.
3. Throttled (Idle with CPU Throttling): When all active requests complete, if CPU throttling is enabled, Cloud Run throttles CPU to near-zero. Memory is preserved, and the instance remains warm for subsequent requests.
4. Terminating: When Cloud Run decides to scale down instances (due to traffic reduction or service redeployment), it initiates instance termination.

The Strict 10-Second SIGTERM Window

Unlike Kubernetes where terminationGracePeriodSeconds can be extended to several minutes, Cloud Run allows a maximum of 10 seconds between dispatching SIGTERM and forcefully killing the container with SIGKILL.

Cloud Run Termination Timeline:
+-------------------------------------------------------------+
| T = 0s: Cloud Run sends SIGTERM to container process        |
| T = 0s - 10s: Graceful Shutdown Window                      |
|   - Stop receiving incoming requests (returns 503 to new)   |
|   - Complete in-flight requests                              |
|   - Close open Cloud SQL / Memorystore connections          |
|   - Flush Cloud Logging / Trace buffers                     |
| T = 10s: Cloud Run sends SIGKILL (Hard container teardown)  |
+-------------------------------------------------------------+

Exam Trap: If a Cloud Run container takes longer than 10 seconds to finish writing a file or processing a transaction during shutdown, Cloud Run sends SIGKILL and terminates the instance immediately. Required asynchronous work should be durably dispatched, for example to Cloud Run Jobs (whose non-GPU task attempts support up to 168 hours, with a 1-hour maximum for GPU tasks) or processed asynchronously via Cloud Tasks or Pub/Sub.

Official references checked October 18, 2026: Google Cloud 1, Google Cloud 2.