3.9 Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics

Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics

In modern cloud architectures, static workload sizing leads either to resource starvation during traffic spikes or wasted infrastructure spend during low-utilization periods. The Kubernetes Horizontal Pod Autoscaler (HPA) automates the horizontal scaling of Deployments, ReplicaSets, and StatefulSets by continuously monitoring compute utilization and application-level metrics. Mastering the autoscaling/v2 API specification, the mathematical scaling formula, metrics aggregation pipelines, and stabilization behavior tuning is a critical requirement for production cluster administration and the CKA exam.

---

1. HPA Architecture and Control Loop Mechanics

The HPA is implemented as a continuous control loop within kube-controller-manager. Periodically (governed by the --horizontal-pod-autoscaler-sync-period flag, default 15 seconds), the controller manager queries the Kubernetes Metrics APIs to evaluate workload utilization against target thresholds.

+---------------------------------------------------------------------------------------------------+
|                                 HPA CONTROL LOOP & METRIC PIPELINE                                |
|                                                                                                   |
|                                    +------------------------------+                               |
|                                    |   kube-controller-manager    |                               |
|                                    | (HPA Sync Period: every 15s) |                               |
|                                    +------------------------------+                               |
|                                                   |                                               |
|                     +-----------------------------+-----------------------------+                 |
|                     |                             |                             |                 |
|                     v                             v                             v                 |
|        +-------------------------+   +-------------------------+   +-------------------------+    |
|        | Resource Metrics API    |   | Custom Metrics API      |   | External Metrics API    |
|        | (metrics.k8s.io)        |   | (custom.metrics.k8s.io) |   | (external.metrics.k8s.io)|
|        +-------------------------+   +-------------------------+   +-------------------------+    |
|                     |                             |                             |                 |
|                     v                             v                             v                 |
|        +-------------------------+   +-------------------------+   +-------------------------+    |
|        | metrics-server          |   | Prometheus Adapter /    |   | CloudWatch / Datadog /  |    |
|        | (kubelet cAdvisor CPU/M)|   | KEDA (HTTP RPS, Latency)|   | SQS Queue Depth Adapter |    |
|        +-------------------------+   +-------------------------+   +-------------------------+    |
|                                                   |                                               |
|                                                   v                                               |
|                                    +------------------------------+                               |
|                                    | Scale Subresource (/scale)   |                               |
|                                    | Deployment.spec.replicas     |                               |
|                                    +------------------------------+                               |
+---------------------------------------------------------------------------------------------------+

The Three Metric Pipeline APIs

1. Resource Metrics (metrics.k8s.io):
   • Provides core container CPU and Memory utilization collected directly from the kubelet embedded cAdvisor engine.
   • Backed by the in-cluster metrics-server add-on.
2. Custom Metrics (custom.metrics.k8s.io):
   • Exposes application-level metrics originating from within the cluster (e.g., HTTP requests per second, active WebSocket connections, queue latency).
   • Backed by custom metric adapters such as the Prometheus Adapter or KEDA (Kubernetes Event-driven Autoscaling).
3. External Metrics (external.metrics.k8s.io):
   • Exposes metrics originating outside the Kubernetes cluster boundary (e.g., AWS SQS queue length, GCP Pub/Sub backlog depth, Azure Service Bus message count).
   • Handled by cloud-provider metric adapters.

---

2. The Mathematical Scaling Algorithm

The HPA calculates the desired number of replicas using a deterministic mathematical formula based on the ratio between the currently observed metric value and the target metric value:

$\text{desiredReplicas} = \left\lceil \text{currentReplicas} \times \left( \frac{\text{currentMetricValue}}{\text{desiredMetricValue}} \right) \right\rceil$

Where $\lceil x \rceil$ is the ceiling function (rounding up to the nearest integer).

The Tolerance Band (+/- 10%)

To prevent micro-oscillations and scale thrashing caused by minor metric fluctuations, the HPA controller applies a tolerance band (default: $0.1$ or $10%$, configurable via --horizontal-pod-autoscaler-tolerance): $\left| 1 - \frac{\text{currentMetricValue}}{\text{desiredMetricValue}} \right| \le 0.10 \implies \text{No Scaling Action}$

Step-by-Step Numerical Example

• Current Replicas: $4$
• Target CPU Utilization: $50%$
• Current Average CPU Utilization: $82%$

1. Compute Metric Ratio: $\text{Ratio} = \frac{82}{50} = 1.64$
2. Evaluate Tolerance Band: $| 1 - 1.64 | = 0.64 > 0.10 \quad (\text{Exceeds tolerance, scaling required})$
3. Apply Formula: $\text{desiredReplicas} = \lceil 4 \times 1.64 \rceil = \lceil 6.56 \rceil = 7$
4. Result: HPA scales the Deployment from $4$ to $7$ replicas.

Metric Target Types

• Utilization: Percentage of the Pod's requested resource (e.g., averageUtilization: 75). Requires resources.requests on all containers.
• AverageValue: Direct average value across all running Pods (e.g., averageValue: 500m CPU or averageValue: 100 requests/sec).
• Value: Absolute total value regardless of replica count (used exclusively in External and Object metrics).

[!IMPORTANT] When target utilization (type: Utilization) is specified, the metric is calculated as: $\text{Utilization} = \frac{\text{Total Actual Usage Across All Pods}}{\text{Total Requested Value Across All Pods}} \times 100\%$ If the relevant resource request is missing, utilization is undefined for that Pod. The controller accounts conservatively for missing metrics; if it has no usable value for a configured metric, it cannot make a scaling decision from that metric and may show <unknown>.

---

3. The autoscaling/v2 API Specification

The autoscaling/v2 API supports multiple metrics and fine-grained behavior controls. autoscaling/v1 remains stable for basic CPU-based HPAs; the old autoscaling/v2beta2 API is deprecated/removed and should not be used.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payment-processor-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-processor
  minReplicas: 2
  maxReplicas: 20
  metrics:
    # 1. Resource Metric: Average CPU Utilization
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    # 2. Resource Metric: Memory Average Value
    - type: Resource
      resource:
        name: memory
        target:
          type: AverageValue
          averageValue: 600Mi
    # 3. Custom In-Cluster Metric: HTTP Requests Per Second
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: 1500
    # 4. External Cloud Metric: AWS SQS Queue Backlog
    - type: External
      external:
        metric:
          name: aws_sqs_queue_depth
          selector:
            matchLabels:
              queue_name: payment_orders
        target:
          type: Value
          value: 5000

Multi-Metric Resolution Rule

When multiple metrics are configured in the metrics array, the HPA calculates the proposed desired replica count for each metric independently, and then selects the highest proposed replica count to guarantee system stability and headroom.

---

4. Advanced Scaling Behavior and Stabilization Windows

The behavior block provides granular control over the velocity of scaling up and scaling down, eliminating the classic "flapping" (rapid cycling between scale-up and scale-down) problem.

spec:
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      selectPolicy: Max
      policies:
        - type: Percent
          value: 100        # Double the replicas
          periodSeconds: 15
        - type: Pods
          value: 4          # Or add 4 pods
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 300  # 5-minute cooldown window
      selectPolicy: Min
      policies:
        - type: Percent
          value: 10         # Remove at most 10% of replicas
          periodSeconds: 60
        - type: Pods
          value: 1          # Or remove 1 pod per minute
          periodSeconds: 60

Key Behavior Parameters

• stabilizationWindowSeconds:
  • For scaleDown, defaults to 300 seconds (5 minutes). The controller looks back over the last 5 minutes of calculated desired states and picks the maximum replica count observed during that window. This ensures traffic dips do not prematurely destroy capacity needed for subsequent spikes.
  • For scaleUp, defaults to 0 seconds, enabling instantaneous burst scale-up when traffic spikes occur.
• selectPolicy:
  • Max: Chooses the policy that produces the highest change in replicas (fastest scale).
  • Min: Chooses the policy that produces the lowest change in replicas (most conservative scale).
  • Disabled: Completely disables scaling in that direction.

---

5. Imperative Management Commands

# Imperatively create an HPA targeting a Deployment with 70% CPU target
kubectl autoscale deployment frontend --cpu-percent=70 --min=2 --max=10

# Inspect HPA state and metrics
kubectl get hpa

# View detailed events, status conditions, and evaluation metrics
kubectl describe hpa frontend

---

6. HPA Status Conditions & Troubleshooting Guide

When running kubectl describe hpa <name>, inspect Conditions:

+-----------------------------------------------------------------------------------------+
|                         HPA TROUBLESHOOTING DECISION TREE                               |
|                                                                                         |
|   [Symptom: HPA shows TARGETS: <unknown>/70%]                                           |
|                                |                                                        |
|                                v                                                        |
|   1. Is metrics-server running in kube-system?                                          |
|      -> kubectl get pods -n kube-system -l k8s-app=metrics-server                       |
|      -> If missing: Deploy metrics-server manifest.                                     |
|                                |                                                        |
|                                v (Running)                                              |
|   2. Can kubectl top nodes / pods fetch metrics?                                        |
|      -> kubectl top pods                                                                |
|      -> If failing: Check metrics-server logs for TLS cert errors or                   |
|         add --kubelet-insecure-tls flag.                                                |
|                                |                                                        |
|                                v (Working)                                              |
|   3. Do ALL containers in the target Deployment specify resources.requests.cpu?         |
|      -> If requests are omitted, HPA CANNOT calculate percentage!                        |
|      -> Fix: Add resources.requests.cpu to container spec.                              |
+-----------------------------------------------------------------------------------------+

Common Root Causes of HPA Failures on CKA Exam

1. Missing resources.requests: If CPU requests are omitted, kubectl get hpa displays TARGETS: <unknown>/80%. HPA requires requests as the mathematical denominator.
2. Metrics-Server Flags: In test environments or bare-metal clusters with self-signed kubelet certificates, metrics-server crashes unless passed --kubelet-insecure-tls and --kubelet-preferred-address-types=InternalIP.
3. Network Policies: Overly restrictive NetworkPolicies blocking ingress to worker node port 10250 prevent metrics-server from scraping cAdvisor.