11.2 Hardware, Boot Failure, Kernel Panic & Driver Diagnostics
Key Takeaways
Distinguish power and device detection, GRUB, kernel and initramfs, and host systemd before choosing a repair.
For PCI devices, a candidate module is different from an active binding. Check lspci and kernel messages together.
On RHEL 9, regenerate the main GRUB configuration under /boot/grub2. Preserve the forwarding stub on the EFI System Partition.
When repairing an initramfs, verify the installed target kernel and image path. A running fallback kernel may have a different release.
A failed required mount can interrupt systemd startup. Correct the verified cause and test the intended mount and dependent services.
11.2 Hardware, Boot Failure, Kernel Panic & Driver Diagnostics
Find the last successful stage before changing a device, image, or boot entry. A server that shows no signs of power needs a different investigation from a server that boots and accepts local logins but cannot receive an SSH connection.
Hardware Diagnostic Workflows
| Area | First evidence | How to interpret it |
|---|---|---|
| Power and console | Check the power feeds, power supply indicators, baseboard management controller (BMC) events and the console. | A missing remote login is not evidence of a power problem. Use out-of-band health data where the server supports it. |
| CPU | Compare sensors, thermal-zone readings and turbostat under a comparable workload. | Correlate temperature with frequency and workload. Frequency scaling or a low clock during idle is normal. |
| Memory | Read kernel Error Detection and Correction (EDAC) reports and available hardware health logs. | A correctable event was recovered by the reporting mechanism. A trend or an uncorrectable error needs investigation and platform-specific handling. |
| PCI device | Use lspci -nnk and the kernel log. | Enumeration, a matching module and an active driver binding are separate findings. |
| USB device | Use lsusb -t for topology and speed, and lsusb -v when detailed descriptors are needed. | Compare results with the capabilities of the device, cable and port before concluding that a slower link is a fault. |
CPU and memory diagnostics
On systems that expose thermal zones, find the relevant zone with its type file. A temperature of 78000 in a zone reporting millidegrees Celsius is 78 °C. Compare it with the platform's limits and a prior measurement under similar load. Linux CPUFreq's scaling_cur_freq can report a requested frequency depending on the driver, so use it in context. On a supported processor, turbostat can show average frequency while the CPU was busy (Bzy_MHz) and utilization over the same interval. Missing virtualization flags alone do not prove hardware failure; firmware or the virtualization layer may hide them.
Memory controllers may report correctable and uncorrectable errors, with detail depending on the hardware and driver. Use tools such as edac-util or rasdaemon where available. If rasdaemon is configured to record events, ras-mc-ctl --errors can display them on supported systems. Compare location, time and recurrence with the BMC or vendor diagnostics. Uncorrectable events can cause service interruption or a panic, but handling depends on the platform and error type. A standalone memory diagnostic such as Memtest86+ lets you test outside the running OS during a maintenance window. A passing run does not eliminate every intermittent fault.
Peripheral & Bus Diagnostics: PCI, USB & Kernel Drivers
For example, a PCI network adapter can appear in lspci -nnk while no network interface is available. Compare the driver fields:
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V
Kernel driver in use: igc
Kernel modules: igc
Here the driver line confirms a binding. If only the module list appears, check journalctl -k -b for probe, firmware or module-signature errors. Check the candidate with modinfo igc and compare its metadata with the running kernel. Use lsmod to check loaded modules and remember that built-in drivers are absent from that list. A blacklist in modprobe.d suppresses internal alias matching and does not guarantee that all explicit loads, built-in drivers or early boot uses are blocked. On a RHEL 9 Secure Boot setup, an untrusted module signature can block loading. Use the distribution's signing and key-trust procedure for a required module.
Bootloader Diagnostics: UEFI & GRUB2 Troubleshooting
Use the four stages you learned in the boot-process section:
- Firmware: If the machine never reaches a boot candidate, verify device detection and the selected boot entry. On a UEFI system with access to EFI variables,
efibootmgr -vcan show current entries and executable paths. The firmware reads the EFI System Partition (ESP) directly. If you are inspecting its files from Linux, check where the installed system mounts that partition. - Bootloader: A GRUB menu or prompt proves that firmware has already reached a loader. Check the entries or the GRUB environment before rebuilding boot artifacts.
- Kernel and early userspace: If the selected kernel runs but fails to mount the real root, check the selected image, root argument and required storage support.
- Host systemd: If the real root is available and a host mount or service fails, inspect the generated unit, its dependencies and the boot journal.
GRUB Shell and Rescue Prompt
A normal grub> prompt offers more facilities than grub rescue>. Rescue mode often means GRUB could not load its normal module from the configured prefix. Paths and built-in commands depend on the layout and build. In this example, inspection reveals that (hd0,gpt2)/boot/grub2 contains the matching GRUB modules. If the partition were a separate /boot filesystem, GRUB could see a path such as /grub2 instead.
grub rescue> ls
(hd0) (hd0,gpt1) (hd0,gpt2) (hd0,gpt3)
grub rescue> ls (hd0,gpt2)/boot/grub2/
grub rescue> set
grub rescue> set root=(hd0,gpt2)
grub rescue> set prefix=(hd0,gpt2)/boot/grub2
grub rescue> insmod normal
grub rescue> normal
Loading normal can return you to normal mode and read a usable menu configuration. It does not repair files or guarantee that a kernel will boot. Once in an authorized Linux recovery environment, use the distribution's persistent repair procedure. On RHEL 9 for both BIOS and UEFI, Red Hat documents building the generated file with grub2-mkconfig -o /boot/grub2/grub.cfg. The file under /boot/efi/EFI/redhat/ is a forwarding stub in this setup. If bootloader binaries need repair, use the firmware-specific procedure. For RHEL 9, a BIOS installation can need grub2-install on the verified boot disk, while the UEFI procedure uses bootloader and shim packages.
Kernel Panic Diagnostics & Backtrace Analysis
Read the first relevant error, note the selected kernel version and save available console or crash information. A panic can happen during startup or after the system has been running. A previous boot's journal is available only if it was retained. A configured crash dump can provide more evidence.
[ 2.418902] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
This message identifies failure to mount the root filesystem. It does not name the cause. Compare the boot entry's root= value, visible devices and filesystems, storage activation, and initramfs contents for the selected kernel. Investigate a kernel or image problem if a known-good installed entry boots on the same storage. Taint indicators record conditions such as an out-of-tree module or a prior warning. They are diagnostic context and do not establish what caused the panic.
Regenerating a specific initramfs
A dracut early userspace error can produce /run/initramfs/rdsosreport.txt. Save that report if it is available. A diagnostic boot with rd.debug can provide more information. From a known-good installed kernel or an appropriate rescue environment, confirm that the target kernel and its installed modules are present. Before writing, confirm that /boot refers to the installed system's boot filesystem and that the previous working entry remains available.
# Example: rebuild for this installed target on a RHEL-style system.
target=5.14.0-362.8.1.el9_3.x86_64
lsinitrd "/boot/initramfs-${target}.img"
dracut -f "/boot/initramfs-${target}.img" "$target"
lsinitrd "/boot/initramfs-${target}.img"
If the original image is missing, the first inspection will report that. Verify the new image and boot entry, including the module or configuration that was missing. A fallback or rescue kernel can differ from the target, so the running release from uname -r is evidence about the current environment. On a host using Debian's initramfs-tools, use its own update-initramfs procedure.
systemd Emergency Mode & /etc/fstab Mount Failures
A required local mount can fail and block normal startup. An optional mount configured with nofail has different dependency and ordering behavior. First locate the error:
journalctl -b -u data.mount
lsblk -f
blkid
findmnt --verify --verbose
Compare the intended device, filesystem type and identifier for /data. If a disk is unhealthy or the filesystem requires repair, follow the storage troubleshooting procedure and keep manual repairs offline. If the issue is a verified fstab configuration error, confirm whether the root filesystem is writable and safe to edit. systemd's emergency target does not require it to be either read-only or read-write. Correct the entry, run systemctl daemon-reload, and verify the regenerated configuration and the intended mount. If the volume is deliberately optional, nofail can allow startup without it. Check whether any dependent service would then write into the empty mount directory or fail without its data.
OS issues after a successful boot
-
Unreachable server: On the console, determine whether the system reached login. Check address, route and listener state, then inspect the remote service and any relevant policy. Successful local login narrows the failure away from early boot.
-
Unit or dependency failure: Use
systemctl status UNITandjournalctl -b -u UNITto find the first reported error. Check prerequisites and recent changes. For package dependency errors, inspect the configured repositories and the package manager's error before forcing a transaction. -
Command missing: Use
command -v NAMEand inspectPATHfor the affected shell. A systemd unit can have a different execution environment from an interactive shell. Verify the executable and unit configuration. -
Killed, crashed, or unresponsive process: A signal or exit status is a symptom. Check the service logs and, where configured, a crash record with
coredumpctl. Compare kernel OOM evidence and resource use before attributing a kill to memory pressure. Track process memory over a representative workload when a leak is suspected; one large sample alone does not show continued growth. -
dracut reference, dracut diagnostics and Linux kernel taint reference
-
Linux CPU frequency scaling and hardware reliability reporting
On a RHEL 9 host in the systemd emergency target, the healthy secondary volume for /data was replaced. The root filesystem is writable, and the journal and blkid confirm only its fstab UUID is stale. What should the administrator do?
Force an immediate reboot before changing fstab
Run e2fsck on the mounted root filesystem
Reinstall GRUB on the boot disk
Correct the /data UUID in fstab, reload systemd, and verify the intended mount before normal startup
A RHEL 9 host booted an older kernel after the newest installed kernel failed to find a storage device. The newer kernel and its modules are installed. Which action rebuilds the intended initramfs?
Force dracut to rebuild the image for the currently running fallback release
Run dracut with the newer installed release and its matching image path, then inspect that image with lsinitrd
Regenerate GRUB without touching either initramfs image
Use efibootmgr -v to copy drivers into the image
From the GRUB rescue prompt, you confirm that (hd0,gpt2)/boot/grub2 contains the correct modules and its boot menu configuration. What can return you to normal GRUB mode?
Run systemctl default from the GRUB prompt
Run efibootmgr to change the firmware boot order from the GRUB prompt
Set root to (hd0,gpt2) and prefix to (hd0,gpt2)/boot/grub2, then run insmod normal and normal
Build a new initramfs directly from the GRUB prompt
Sections you finish are checked off in the contents.