6.8 Metrics Server, Resource Utilization & Kubectl Top

6.8 Metrics Server, Resource Utilization & Kubectl Top

Resource observability is essential for capacity planning, detecting runaway memory leaks, preventing CPU starvation, and autoscaling workloads. Kubernetes decouples short-term operational resource metrics from full long-term historical time-series monitoring.

The Metrics Server provides the foundational pipeline for core Kubernetes autoscaling and CLI resource inspection (kubectl top), while tools like Prometheus and Grafana provide deep observability, alerting, and trend analysis.

---

1. Metrics Server Architecture & API Aggregation

Metrics Server is an add-on component that operates as an aggregated API server. It periodically (default resolution: 15 seconds) collects container and node CPU/memory telemetry from each kubelet's resource-metrics endpoint; kubelet obtains container measurements from its integrated cAdvisor instrumentation across every node in the cluster.

+-----------------------------------------------------------------------------------------+
|                        METRICS SERVER PIPELINE ARCHITECTURE                             |
|                                                                                         |
|  [KUBECTL TOP / HPA]                                                                    |
|         |                                                                               |
|         v (HTTPS: Query /apis/metrics.k8s.io/v1beta1/)                                  |
|  [KUBE-APISERVER (API Aggregator Proxy)]                                                |
|         |                                                                               |
|         v (Proxies request to extension API server)                                     |
|  [METRICS SERVER POD (kube-system)]                                                     |
|  - In-memory ephemeral metric cache (No persistent database)                           |
|  - Collects metrics over HTTPS (port 10250) every 15s by default                                    |
|         |                                                                               |
|         +-----------------------+-----------------------+                               |
|         |                       |                       |                               |
|         v                       v                       v                               |
|  [NODE 01: KUBELET]      [NODE 02: KUBELET]      [NODE 03: KUBELET]                     |
|  cAdvisor /stats/summary cAdvisor /stats/summary cAdvisor /stats/summary                |
+-----------------------------------------------------------------------------------------+

Core Functional Characteristics:

1. In-Memory Storage: Metrics Server does not persist data to disk or etcd. It retains only the most recent scrape in RAM.
2. API Endpoint Registration: It registers the API group metrics.k8s.io/v1beta1 with kube-apiserver via an APIService object.
3. Consumers: Powers kubectl top nodes, kubectl top pods, and the Horizontal Pod Autoscaler (HPA) / Vertical Pod Autoscaler (VPA).

---

2. Inspecting Resource Utilization with kubectl top

Once Metrics Server is functional, administrators can query real-time compute consumption:

# 1. Inspect resource utilization across all cluster nodes
kubectl top nodes

# Example Output:
# NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
# control-plane  250m         12%    1850Mi          48%
# worker-1       850m         42%    3200Mi          82%
# worker-2       120m          6%     980Mi          25%

# 2. Inspect resource utilization for pods in default namespace
kubectl top pods

# 3. Query all pods across all namespaces sorted by memory consumption
kubectl top pods -A --sort-by=memory

# 4. Query all pods across all namespaces sorted by CPU consumption
kubectl top pods -A --sort-by=cpu

# 5. Display individual container metrics within multi-container pods
kubectl top pods my-app-pod --containers

Metric Units Explained:

• CPU: Measured in millicores ($m$) or fractions of a vCPU/core ($1000m = 1 \text{ vCPU / Core}$). $250m = 0.25 \text{ CPU cores}$.
• Memory: Measured in bytes or Megabytes/Gigabytes ($Mi$ = Mebibytes $= 2^{20}$ bytes; $Gi$ = Gibibytes $= 2^{30}$ bytes).

---

3. High-Yield Metrics Server Troubleshooting

On the CKA exam and fresh cluster installations, kubectl top frequently fails with: error: Metrics API not available or error: metrics not available yet.

+-----------------------------------------------------------------------------------------+
|                     METRICS SERVER TROUBLESHOOTING DECISION TREE                        |
|                                                                                         |
|  $ kubectl top nodes -> FAILS                                                           |
|  1. Check Pod Status: $ kubectl get pods -n kube-system -l k8s-app=metrics-server       |
|     - If CrashLoopBackOff: $ kubectl logs -n kube-system deploy/metrics-server          |
|                                                                                         |
|  2. Common Error A: 'x509: certificate signed by unknown authority'                     |
|     - Cause: Kubelet serving cert is self-signed; Metrics Server TLS verification fails.|
|     - Production fix: issue kubelet serving certificates trusted by Metrics Server. For a disposable test cluster only, `--kubelet-insecure-tls` can confirm that certificate validation is the blocker.      |
|                                                                                         |
|  3. Common Error B: 'dial tcp: lookup node01 on 10.96.0.10:53: no such host'            |
|     - Cause: Metrics Server trying to reach nodes by hostname without cluster DNS records|
|     - Fix: Add `--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname`       |
+-----------------------------------------------------------------------------------------+

Inspecting Metrics Server Arguments:

Edit the deployment only after identifying the failing address or certificate path:

kubectl edit deployment metrics-server -n kube-system

Update the container command-line arguments:

spec:
  template:
    spec:
      containers:
      - name: metrics-server
        args:
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --metric-resolution=15s
        # Test-only diagnostic workaround; prefer trusted kubelet serving certs:
        # - --kubelet-insecure-tls

---

4. Full Monitoring Stack & The Four Golden Signals

For enterprise production clusters, Metrics Server is augmented with Prometheus (pull-based metrics collector and time-series database) and Grafana (visualization dashboards).

+-----------------------------------------------------------------------------------------+
|                             THE FOUR GOLDEN SIGNALS (SRE)                               |
|                                                                                         |
|  1. LATENCY       Time taken to service a request (e.g., HTTP 99th percentile < 200ms). |
|  2. TRAFFIC       Demand placed on system (e.g., HTTP requests/sec, IOPS, network bps). |
|  3. ERRORS        Rate of requests that fail (e.g., HTTP 5xx responses, error logs).    |
|  4. SATURATION    Fraction of resource capacity utilized (e.g., memory, CPU, disk IO).  |
+-----------------------------------------------------------------------------------------+

Complementary Frameworks:

• USE Method (Node/Infrastructure Focus): Utilization (percentage busy), Saturation (queued work), Errors (hardware/kernel errors).
• RED Method (Microservices Focus): Rate (requests per second), Errors (failed requests), Duration (request latency).