12.3 System Performance, Resource Saturation, I/O Wait & OOM Killer Remediation
Key Takeaways
Compare utilization, saturation and errors for the affected resource, using the same time window and workload baseline.
A Linux load average includes runnable and uninterruptibly sleeping tasks. Compare it with CPU time, run queues and storage or other wait evidence.
Use available memory, paging rates, pressure and cgroup limits together. Correlate OOM logs with the correct workload and scope.
An oom_score_adj of -1000 excludes a task from ordinary kernel victim selection. Userspace policies and other termination mechanisms still apply.
Interpret I/O busy time, latency and queue depth together. Correlate network counters, health reports and startup timing with the actual slowdown.
12.3 System Performance, Resource Saturation, I/O Wait & OOM Killer Remediation
Define which operation is slow, who is affected, and when the symptom began. Compare the affected scope with a recent baseline under comparable traffic. Examine the host, virtual machine, container or service that runs the operation rather than attributing every delay to the machine's busiest resource.
The USE Method: Utilization, Saturation & Errors
Brendan Gregg's USE method asks three questions about each resource. How busy is it? What work has accumulated or stalled? What errors are increasing? Combine these answers before making a change.
| Resource | Utilization or capacity | Saturation and errors |
|---|---|---|
| CPU | Per-CPU and per-process activity with mpstat and pidstat | Runnable work, scheduler delays, pressure, thermal or hardware events |
| Memory | Available memory and the relevant service or cgroup limit | Paging rates, memory pressure, failed allocations and OOM records |
| Storage | Throughput and device busy time | Latency, queue depth, blocked tasks and kernel I/O errors |
| Network | Interface throughput compared with baseline and link capacity | Drops, retransmissions, measured latency or jitter, and link or path errors |
Understanding System Load Averages
Linux load averages describe runnable and uninterruptibly sleeping work over the 1-, 5-, and 15-minute scales. For example:
18:45:22 up 14 days, 3:12, 2 users, load average: 8.50, 4.20, 1.15
The short-term value is higher than the longer averages. On a system whose workload can use 8 logical CPUs, compare the recent rise with per-CPU utilization, runnable counts, and tasks blocked in the kernel. A load of 8 does not itself imply 100% CPU use, and a load of 16 does not prove exactly eight queued tasks. Runnable work may compete for CPU, while an uninterruptible wait can contribute to load with no CPU execution. CPU affinity and container limits can constrain a single workload even if the host has idle CPUs.
uptime
mpstat -P ALL 1 5
pidstat -u 1 5
vmstat 1 6
Read later interval reports when an initial report is since boot. In vmstat, r includes running or runnable processes, b reports processes blocked waiting for I/O, and cs is the context-switch rate. An increase in switches is a clue when it correlates with a delay and the workload's baseline. Identify the processes driving the change; a switch count alone cannot establish contention.
| CPU time | Interpretation |
|---|---|
| User, nice and system | Time spent in user tasks or kernel execution. Find the busy CPUs and processes rather than applying one universal percentage threshold. |
| Idle and I/O wait | I/O wait is an accounting category associated with outstanding I/O. It is not a per-device queue measurement, and the kernel documentation notes limits on its reliability. |
| Hardware and software interrupts | CPU time handling interrupts; correlate changes with relevant host and device activity. |
| Steal | Time a virtual CPU did not get to run while the hypervisor serviced other work. Compare it with past behavior and the resource policy before deciding the cause. |
A compute-bound thread can leave one core busy and other cores idle. If a process is blocked, correlate its wait with the device or service it uses. CPU upgrades and changing priority will have limited effect when the observed delay is waiting elsewhere.
Memory Performance, Thrashing & The OOM Killer
Use free -h to compare free memory and the available estimate, which includes some reclaimable capacity. A small free value by itself may reflect normal caching. The estimate does not guarantee that every allocation will succeed; cgroup limits and allocation constraints can restrict an individual workload. In repeated vmstat output, si and so describe swap activity. A system may have swap allocated without currently swapping heavily. Sustained paging, memory pressure stalls and slower requests together support a memory pressure investigation. Where supported, read /proc/pressure/memory or the service's cgroup pressure data.
The kernel can invoke OOM handling when it cannot satisfy memory demands in the relevant allocation scope. That can happen inside a memory-limited cgroup while the rest of the host has available memory. Compare kernel messages with the affected workload and limits. On cgroup v2, memory.events includes counters for memory limit and OOM-related events. Use those and the logs to determine which resource was exhausted.
journalctl -k -b | grep -Ei 'out of memory|oom-kill|killed process'
For example, a log naming a database process as an OOM victim establishes that this process was selected. It does not establish which allocation or leak triggered the incident. Compare memory over time, recent workload changes and the effective service limit. Address the underlying growth or limits with the service owner, then measure the result.
Kernel OOM eligibility and other termination mechanisms
Linux exposes a computed score at /proc/PID/oom_score and an adjustment at /proc/PID/oom_score_adj. An adjustment from -1000 through +1000 changes kernel victim scoring. The minimum, -1000, excludes a task from ordinary kernel OOM victim selection. A high positive adjustment increases its score without guaranteeing that it will be the first eligible victim. For a systemd service, OOMScoreAdjust= sets the kernel score adjustment for executed processes when applicable.
An exclusion shifts pressure to other tasks. It is a deliberate narrowly scoped policy rather than a remedy for low memory. A userspace policy such as systemd-oomd, a service stop or a different signal can still terminate the process. Confirm which mechanism acted before changing the score setting.
Disk I/O Latency & Remote Storage
iostat -xz 1 5
pidstat -d 1 5
Read the interval reports from the actual slowdown. With this iostat invocation the first report covers time since boot. Use -y when you want to omit that initial report. await includes time in the queue and time servicing the request, expressed in milliseconds. aqu-sz describes average queue depth. %util is elapsed time with I/O activity. A value near 100% can mean a serial device is busy nearly continuously, while a parallel SSD or array may still handle additional requests. Decide by comparing rates, latency and workload requirements with a known baseline. A high counter does not establish a failed controller or prove when an application request reached storage.
In a database case, assume low user/kernel CPU time, sustained waiting on the database volume, rising volume latency compared with the service baseline, and a slow query interval. Prioritize the storage path, then check which process or request pattern drives the load and whether the volume is healthy. Use kernel errors and device telemetry to distinguish an overloaded volume from a transport or hardware fault. For a remote filesystem, check its client-side request latency, path and server before treating the host's local disk as the cause. Apply the storage chapter's repair procedure only when the evidence indicates a filesystem fault.
Network, Startup & Other Symptoms
-
Packet drops, jitter or low throughput: Compare changes in
ip -s linkand supported driver counters fromethtool -Swith switch or peer statistics. Loss and variable latency may come from queues, errors or path changes. Controlled throughput testing between approved endpoints can isolate link capacity. Use socket and route observations to separate network problems from slow application responses. Confirm the counter and cause before changing buffers. -
Random disconnects, timeouts or sluggish terminals: Compare a local operation with the same workflow through the affected path. Correlate timestamps with interface changes, retransmissions, service logs and latency. A terminal slowdown can come from the client path or from a saturated server.
-
Slow startup: Compare
systemd-analyze timeandsystemd-analyze critical-chainwith a comparable boot, then use the boot journal for the units involved. A service high inblamedoes not alone establish the cause because startup can overlap. -
Many failed logins: Correlate authentication records with service throughput or CPU, and check whether a stale client or an unwanted request pattern explains the failures. Count alone does not establish the bottleneck, and changing authentication limits is a separate security decision.
-
Hardware errors or blocked tasks: Check the affected resource, kernel messages and device health. If the system is unresponsive, use a permitted console or monitoring data from before the outage. Preserve useful evidence before restarting.
-
Linux cgroup v2 memory controls, pressure stalls, kernel OOM adjustment, and systemd-oomd
A database host has high load and little user or system CPU activity. Its CPU reports high I/O wait, and interval measurements show the database volume’s request latency and queue rising above baseline during the same slowdown. Which resource path should be investigated first?
The storage path for the database volume
The CPU alone, because every high load value proves CPU saturation
Physical RAM alone, because wait time proves swapping
Hypervisor scheduling alone, because a high load value establishes steal
During an approved service design review, which setting excludes a service process from ordinary Linux kernel OOM victim selection, without promising protection from a separate userspace OOM policy?
Set the process nice value to -20
Write 1000 to the read-only oom_score reporting field
Disable the service’s startup timer
Set OOMScoreAdjust=-1000 in the applicable systemd service configuration
An iostat interval reports an await of 145.2 ms and a device %util of 99.8%. Which statement follows directly from these measurements?
The controller failed and the filesystem has remounted read-only
Reported requests averaged 145.2 ms including queue and service time, and the device showed I/O activity for nearly all of the interval
The device was idle for nearly all of the interval
The system was necessarily paging to swap throughout the interval
Sections you finish are checked off in the contents.
You've completed this section
Continue exploring other exams