12.2 Troubleshooting Compute, Container & Storage Failures

Troubleshooting Compute, Container & Storage Failures

Compute and storage infrastructure forms the foundational execution layer of every cloud workload. When compute instances thrash, containers enter continuous crash loops, or block storage volumes throttle I/O operations, cloud engineers must rapidly interpret operating system metrics, hypervisor telemetry, and container runtime states to restore operational health.

---

1. Cloud Compute Diagnostics & Performance Anomalies

Diagnosing compute performance requires understanding the vital distinction between CPU utilization, system load average, memory pressure, and hypervisor resource contention.

+---------------------------------------------------------------------------------------------------+
|                         COMPUTE PERFORMANCE DIAGNOSTIC MATRIX                                     |
|                                                                                                   |
|  Metric / Symptom                 Root Cause Analysis               Primary Diagnostic Command    |
|  +------------------------------+---------------------------------+-----------------------------+ |
|  | High CPU %us (User Space)    | Unoptimized code, infinite loop | top / htop / pidstat -u 1   | |
|  | High CPU %sy (Kernel Space)  | Excessive context switching/syscalls | vmstat 1 / strace -c   | |
|  | High CPU %st (Steal Time)    | Noisy neighbor / Overcommitted host | top (%st > 5%) / mpstat 1| |
|  | High Load Average + Low CPU  | Disk I/O wait bottleneck (%wa)  | iostat -xz 1 / dmesg        | |
|  | Memory Exhaustion (Exit 137) | Linux OOM Killer invoked        | dmesg -T | grep -i oom      | |
|  +------------------------------+---------------------------------+-----------------------------+ |
+---------------------------------------------------------------------------------------------------+

High CPU Utilization vs. High Load Average

• High CPU Utilization (%us / %sy): Indicates that virtual CPU cores are actively processing instructions. %us (user space) points to application code, while %sy (system/kernel space) points to operating system overhead such as excessive context switching or interrupt handling.
• High System Load Average with Low CPU Utilization: Load average represents the average number of processes that are either in a runnable state (using or waiting for CPU) or in an uninterruptible sleep state (waiting for disk I/O, network sockets, or lock acquisition, represented as state D in ps). If a 4-vCPU instance has a load average of 28.0 but CPU utilization is only 15%, the system is starved for disk or network I/O, not compute power. Look at the %wa (I/O wait) metric in top.

CPU Steal Time (%st)

In multi-tenant public cloud environments, virtual machines run on physical hypervisor hosts shared with other tenants. CPU Steal (%st) measures the percentage of time a virtual CPU was ready to execute instructions but was forced to wait because the physical hypervisor processor was busy servicing other virtual machines (the noisy neighbor problem).

• Diagnosis: If %st consistently exceeds 3% to 5% in top or mpstat, the cloud instance is experiencing hypervisor-level CPU starvation.
• Remediation: Stop and start the cloud VM to force the hypervisor scheduler to instantiate the instance on a different physical host, upgrade to a larger compute-optimized instance family (e.g., AWS c6i / Azure Fsv2), or migrate to Dedicated Hosts / Dedicated Instances.

# Inspect real-time CPU breakdown including %us, %sy, %wa, and %st
top -b -n 1 | head -n 5

# Check process states; look for processes in 'D' state (uninterruptible I/O sleep)
ps aux | awk '{if ($8 ~ /D/) print $0}'

Memory Leaks & The Linux OOM Killer

When an application continuously allocates memory without freeing unreferenced objects (a memory leak), available physical RAM and swap space dwindle to zero. When the kernel cannot fulfill an allocation request, the Out-Of-Memory (OOM) Killer is invoked.

• Mechanism: The kernel calculates an oom_score for each running process based on memory consumption and oom_score_adj. It terminates the highest-scoring process using a SIGKILL (signal 9) to protect kernel stability.
• Linux Exit Code 137: When a process or container is terminated by the OOM killer, it exits with code 137 (128 + signal 9 (SIGKILL)).
• Triage: Inspect kernel ring buffer logs using dmesg -T | grep -i oom or journalctl -k to identify which process was killed and its memory footprint at termination.

---

2. Container & Kubernetes Workload Troubleshooting

Container orchestration platforms abstract compute nodes into a unified cluster, but introduce specialized workload lifecycle states that indicate specific configuration or resource failures.

+---------------------------------------------------------------------------------------------------+
|                         KUBERNETES POD FAILURE TAXONOMY                                           |
|                                                                                                   |
|   Pod Status               Common Underlying Root Causes             Resolution Workflow          |
|   +----------------------+-----------------------------------------+----------------------------+ |
|   | CrashLoopBackOff     | Application runtime bug, missing env    | kubectl logs --previous    | |
|   |                      | variables, failed health/liveness probe | kubectl describe pod <name>| |
|   |                      |                                         |                            | |
|   | ImagePullBackOff /   | Typo in image repository/tag, missing   | Verify secret in manifest; | |
|   | ErrImagePull         | imagePullSecrets, registry rate limits  | check docker auth token    | |
|   |                      |                                         |                            | |
|   | OOMKilled (Exit 137) | Container exceeded resources.limits.mem | Increase memory limit in   | |
|   |                      | or host node reached memory pressure    | deployment spec; fix leak  | |
|   |                      |                                         |                            | |
|   | Pending              | Insufficient CPU/RAM requests on nodes, | Add cluster nodes; remove  | |
|   |                      | unmatched nodeSelectors, taints/affinity| un-tolerated node taints   | |
|   +----------------------+-----------------------------------------+----------------------------+ |
+---------------------------------------------------------------------------------------------------+

CrashLoopBackOff

A pod in CrashLoopBackOff has been scheduled to a node, but the application container repeatedly starts, fails, and crashes. Kubernetes enforces an exponential backoff delay (10s, 20s, 40s... up to 5m) before restarting the container.

• Diagnostic Steps:
  1. Run kubectl describe pod <pod-name> and examine the Last State and Exit Code fields.
  2. Run kubectl logs <pod-name> --previous to read stdout/stderr output from the crashed container instance before its restart.
  3. Verify whether Liveness Probes or Startup Probes are failing due to aggressive initial delay timeouts before the application has finished warming up.

ImagePullBackOff & ErrImagePull

Indicates the kubelet on the worker node cannot fetch the container image from the container registry (e.g., Amazon ECR, Azure Container Registry, Docker Hub, Google Artifact Registry).

• Root Causes:
  • Typographical error in the image URL or tag (e.g., app:v1.2 instead of app:v1.2.0).
  • Missing or misconfigured imagePullSecrets required to authenticate against a private registry.
  • Worker node subnet lacks a NAT Gateway or VPC Endpoint to route traffic to the registry.
  • Anonymous pull rate limit exceeded (e.g., Docker Hub HTTP 429 Too Many Requests).

OOMKilled vs. Node Memory Pressure

• Container OOMKilled: If a container exceeds the memory threshold defined in resources.limits.memory in its pod specification, the container runtime (containerd/CRI-O) sends a SIGKILL specifically to that container. The pod restarts, and kubectl describe pod displays Reason: OOMKilled (Exit Code: 137).
• Node Eviction (NodeMemoryPressure): If the worker node itself runs out of physical memory, the kubelet begins evicting pods according to their Quality of Service (QoS) Class (BestEffort pods are evicted first, followed by Burstable, and finally Guaranteed pods).

Pending Pods

A pod remains in Pending state when the Kubernetes scheduler cannot find any worker node that satisfies the pod's scheduling requirements.

• Root Causes:
  • Resource Starvation: Total cluster resources.requests.cpu or resources.requests.memory are fully allocated.
  • Taints and Tolerations: Worker nodes have taints (e.g., node.kubernetes.io/unreachable or dedicated GPU taints sku=gpu:NoSchedule) that the pod does not have matching tolerations for.
  • Node Affinity / NodeSelector: The pod requires a node label (e.g., topology.kubernetes.io/zone: us-east-1a) that matches zero available nodes.
  • Unbound PersistentVolumeClaim (PVC): The pod requires a storage volume that has not yet been provisioned or is in a different Availability Zone.

# Inspect Kubernetes scheduling failures and event log
kubectl describe pod billing-service-78f9d-xkz42 | grep -A 10 Events:

# Check resource capacity and allocation across all cluster nodes
kubectl describe nodes | grep -A 5 "Allocated resources:"

---

3. Cloud Storage Troubleshooting & Bottlenecks

Cloud storage issues manifest across three primary failure domains: throughput/IOPS throttling, volume attachment deadlocks, and file system capacity exhaustion.

+---------------------------------------------------------------------------------------------------+
|                         CLOUD STORAGE FAILURE DOMAINS                                             |
|                                                                                                   |
|  [ 1. IOPS & Throughput Throttling ]                                                              |
|    ├── Burstable storage burst credit depletion (AWS gp2/gp3 burst balance = 0%)                  |
|    ├── Symptoms: High average queue length (aqu-sz), massive write latency, database timeouts      |
|    └── Fix: Resize volume for higher baseline IOPS, convert to Provisioned IOPS (io2/Ultra Disk)  |
|                                                                                                   |
|  [ 2. Storage Capacity & Inode Exhaustion ]                                                       |
|    ├── Block Exhaustion: df -h shows 100% disk usage (large logs, core dumps)                     |
|    ├── Inode Exhaustion: df -i shows 100% inode usage (millions of zero-byte session/temp files)  |
|    └── Unlinked Open Files: lsof +L1 shows deleted files held open by running processes           |
|                                                                                                   |
|  [ 3. Volume Attachment & Multi-Attach Deadlocks ]                                                |
|    ├── EBS volume stuck in 'attaching' or 'detaching' state (hypervisor lock / host crash)        |
|    ├── Multi-attach file system corruption: mounting standard ext4/XFS read-write on multiple VMs |
|    └── NFS Stale File Handle (ESTALE): File deleted on NFS server while client holds open handle |
+---------------------------------------------------------------------------------------------------+

IOPS and Throughput Throttling (Burst Credit Depletion)

Many cloud block storage tiers (e.g., AWS EBS gp2, Azure Standard SSD) use a token-bucket algorithm for I/O bursting. When I/O demand exceeds the baseline allowance, the volume consumes burst credits. When burst credits are exhausted:

• The volume is throttled down to its baseline performance (e.g., 3 IOPS per GB on older gp2 volumes).
• Operating system disk queue depth (aqu-sz in iostat) spikes dramatically.
• Write latency increases from sub-millisecond levels to hundreds of milliseconds, triggering application database connection pool exhaustion.
• Resolution: Migrate volumes to Provisioned IOPS (e.g., AWS EBS io2 Block Express, Azure Ultra Disk) or decouple write-intensive operations to ephemeral NVMe instance store volumes.

# Check extended I/O statistics; look for high await (>20ms) and aqu-sz (queue depth > 2)
iostat -xz 1 5

Disk Space Exhaustion: Block Storage vs. Inodes

A frequent CompTIA exam trap involves understanding that a file system can run out of space in two distinct ways:

1. Data Block Exhaustion (df -h): The physical or virtual gigabytes are 100% full. Common culprits include unrotated application log files, database transaction logs, and core dumps.
2. Inode Exhaustion (df -i): Inodes store file metadata (permissions, owner, block pointers). Every file and directory consumes exactly one inode regardless of file size. If an application generates millions of tiny 1-byte session files or temporary lock files, 100% of available inodes can be consumed even while df -h shows 80% free disk space. When inodes are exhausted, the operating system returns No space left on device upon any new file creation attempt.
3. Deleted Open File Descriptors (lsof +L1): When a log file is deleted with rm while a running daemon (e.g., Nginx or MySQL) still holds an open file descriptor to it, the disk blocks are not released back to the OS until the daemon is restarted or reloaded.

# Check block capacity
df -h

# Check inode capacity
df -i

# Find processes holding onto deleted unlinked files that are consuming disk space
lsof +L1

Volume Attachment & Multi-Attach Constraints

• Stuck Volume Attachments: If a hypervisor crashes while detaching an elastic block volume, the volume may remain locked in an attaching or detaching state. Cloud engineers must use cloud CLI force-detach APIs or stop the parent virtual machine.
• Availability Zone Boundaries: Cloud block storage volumes (AWS EBS, Azure Managed Disks) are physical constructs bound to a single Availability Zone. A volume created in us-east-1a cannot be attached to a virtual machine in us-east-1b.
• Multi-Attach Limitations: Standard cloud block volumes do not support simultaneous read-write attachment to multiple virtual machines using standard file systems like ext4, XFS, or NTFS. Doing so causes instant file system corruption because standard file systems lack distributed cache-coherency lock managers. Multi-attach block volumes require specialized clustered file systems (e.g., GFS2, OCFS2). For standard multi-node shared storage, use managed Network Attached Storage (AWS EFS, Azure Files, Google Cloud Filestore) over NFS/SMB.
• NFS Stale File Handle (ESTALE): Occurs on shared file systems when a client attempts to read/write a file using an open file handle, but the file was deleted or replaced on the underlying NFS server by another client or automated batch job. The client must unmount and remount the export or restart the client process.