14.1 GKE Resources, OOM Diagnosis, and Autoscaling

Resource Specifications: Requests, Limits & QoS Classes

Kubernetes schedules and protects workloads based on CPU and memory declarations inside the resources block of container specifications:

• CPU (Compressible): Measured in cores or millicores (1000m = 1 vCPU; 250m = 0.25 vCPU). If a container attempts to consume more CPU than its limit, the Linux kernel Completely Fair Scheduler (CFS) throttles execution by restricting CPU cycles, but the process is not killed.
• Memory (Uncompressible): Measured in bytes or mebibytes (Mi, Gi). Memory cannot be throttled. If a container allocates memory beyond its configured limit, the Linux Out-Of-Memory (OOM) killer terminates the process immediately.

Quality of Service (QoS) Classes

Kubernetes classifies pods into three QoS classes based on how requests and limits are defined. This classification dictates eviction priority when a cluster node experiences resource pressure:

Eviction Priority Order (First Evicted to Last Evicted):
BestEffort (First)  ==>  Burstable (Middle)  ==>  Guaranteed (Last)

Diagnosing OOMKilled (Exit Code 137)

When a container exceeds its memory limit or when the host node runs out of physical memory, the Linux kernel sends SIGKILL (signal 9) to the offending process. Kubernetes reports this status as OOMKilled with exit code 137 (128 + 9 = 137):

## Inspecting an evicted pod
kubectl describe pod order-service-6789b-f4x9q

Output shows:

Containers:
  order-service:
    State:          Terminated
      Reason:       OOMKilled
      Exit Code:    137
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

Troubleshooting and Remediation Steps for OOMKilled:

1. Differentiate Container OOM vs. Node OOM: Check kubectl describe node. If the node condition reports MemoryPressure: True, the host itself is depleted. If the node has plenty of free memory, the container exceeded its own limits.memory.
2. Java / JVM Cgroup Awareness: Use a supported JVM that recognizes the container's cgroup version and memory limit. Set a tested heap budget below the container limit, leaving room for native memory, threads and buffers; an arbitrary 75% setting is not universal. Inspect runtime flags and observed memory rather than assuming every older JVM has the same behavior.
3. Memory Leak Profiling: Use supported heap profiles and separate heap-dump tools to identify memory leaks before artificially inflating memory limits.

---

Autoscaling Architectures: HPA v2, VPA & Cluster Autoscaler

GKE provides three interrelated autoscaling engines that operate across different abstraction layers:

                    +---------------------------------+
                    |       Workload Demand           |
                    +---------------------------------+
                                     |
             +-----------------------+-----------------------+
             |                                               |
             v                                               v
+-------------------------+                     +-------------------------+
| Horizontal Pod Auto-    |                     | Vertical Pod Auto-      |
| scaler (HPA v2)         |                     | scaler (VPA)            |
| * Adjusts REPLICA COUNT |                     | * Adjusts CPU & MEMORY  |
| * Scales horizontally   |                     |   REQUESTS / LIMITS     |
+-------------------------+                     +-------------------------+
             |                                               |
             +-----------------------+-----------------------+
                                     |
                                     v
                    +---------------------------------+
                    | Pods in PENDING state (No Room) |
                    +---------------------------------+
                                     |
                                     v
                    +---------------------------------+
                    | Cluster Autoscaler (CA)         |
                    | * Adjusts NODE COUNT in pool    |
                    +---------------------------------+

Horizontal Pod Autoscaler (HPA v2)

HPA monitors resource consumption and adjusts the number of pod replicas. In autoscaling/v2, HPA supports scaling based on CPU utilization, memory utilization, and custom Cloud Monitoring metrics:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-processor-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-processor
  minReplicas: 3
  maxReplicas: 50
  metrics:
    # Target 70% average CPU utilization
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    # Target custom Cloud Monitoring metric: Pub/Sub backlog
    - type: External
      external:
        metric:
          name: pubsub.googleapis.com|subscription|num_undelivered_messages
          selector:
            matchLabels:
              resource.labels.subscription_id: "orders-sub"
        target:
          type: AverageValue
          averageValue: "50"

HPA Requests, Targets, and Capacity

For CPU utilization targets, HPA compares usage to CPU requests, not limits. A Pod using 300m with a 500m request is at 60% utilization. The basic recommendation is ceil(current replicas * current metric / desired metric): four replicas at 90% with a 60% target recommend ceil(4 * 90 / 60) = 6. Stabilization, tolerance, missing metrics, readiness and min/max replicas can modify the applied result. Missing CPU requests can prevent a meaningful utilization calculation.

For external Pub/Sub backlog metrics, install and authorize a supported metrics adapter, select the exact subscription, and choose a target per replica that matches measured processing capacity. More replicas do not create database capacity. Bound concurrency/pools, monitor oldest-message age as well as count, and ensure node capacity can accommodate the requested Pods. The adapter and scaling identity need Monitoring permissions independently of the worker's Pub/Sub permissions.

Vertical Pod Autoscaler (VPA)

VPA monitors historical CPU and memory consumption over time and automatically optimizes container resource requests and limits. Common VPA modes include (additional modes depend on Kubernetes/GKE support):

1. Off: Generates resource recommendations only; does not alter pods.
2. Initial: Assigns optimal resource requests only when new pods are created.
3. Auto: applies resource recommendations using supported update mechanisms; recreation and in-place behavior depend on the configured mode and GKE version.

Exam Trap: Never combine HPA and VPA on the same resource metrics (CPU or memory). If HPA is configured to scale replicas when CPU exceeds 70%, and VPA is configured in Auto mode to increase CPU requests when CPU is high, the two controllers enter a destructive race condition: VPA will increase CPU requests while HPA simultaneously adds more pods, resulting in runaway resource over-allocation.

Cluster Autoscaler (CA) & GKE Autopilot

• Cluster Autoscaler: Watches the Kubernetes scheduler for pods that cannot be scheduled due to insufficient CPU or memory capacity (Pending state). When pending pods are detected, CA adds Compute Engine VM nodes to the node pool. When nodes remain underutilized for a sustained period, CA gracefully drains pods and terminates idle nodes.
• GKE Autopilot: Completely eliminates manual node pool configuration and Cluster Autoscaler tuning. In Autopilot clusters, Google provisions and scales the underlying node infrastructure automatically, with Pod-based general-purpose billing and node-based billing for some specialized hardware/compute classes.