11.3 Storage, Filesystem Corruption, Disk Saturation & Inode Exhaustion

Key Takeaways

  • Identify the failed path’s filesystem with findmnt. Compare df -h and df -i, and check the relevant quota or mount options when capacity is available.

  • After the last name is removed, an open file can retain its allocation. Find it with lsof +L1, arrange for the process to reopen or close it, then verify free space.

  • Use same-filesystem scans such as du -x to locate high block or inode use. Review the results and application retention requirements before removing files.

  • Use e2fsck for unmounted ext filesystems. XFS uses mount-time log replay and xfs_repair for offline repair. The -L flag discards pending log data and can lose files.

  • Compare kernel I/O messages with device-specific health data. An overall SMART pass does not rule out intermittent I/O or transport faults.

Last updated: November 2026

11.3 Storage, Filesystem Corruption, Disk Saturation & Inode Exhaustion

Start with the path and the exact error. A full filesystem, exhausted inodes, an enforced quota, an intentional read-only mount and a filesystem reacting to I/O errors require different fixes. Confirm which device and filesystem back the affected directory before changing anything.

Storage Block Saturation vs. Inode Exhaustion

If creating a file reports No space left on device, check the filesystem backing the target directory. Run these against a directory that exists:

findmnt -T /var/app
df -h /var/app
df -i /var/app
EvidenceLikely conditionNext check
Available blocks near zeroData or filesystem metadata filled the allocation available to the callerCount reachable allocations with du -x; check for unlinked open files if the totals differ.
Free blocks but no available inodesThe filesystem cannot create another inode for a new file or directoryUse a same-filesystem inode scan to find high file counts.
Filesystem has available blocks and inodes; the application reports a quota errorA user, group or project limit may be reachedInspect the applicable block or inode quota and policy.

A directory entry maps a name to an inode. The inode holds metadata such as ownership, permissions, timestamps and data references. Hard-linked names can refer to the same inode. ext4 allocates inode tables when creating and extending the filesystem. Its inode size and allocation ratio depend on the filesystem creation options and profile. XFS allocates inode storage dynamically within filesystem limits. Check the actual filesystem and usage rather than inferring a fixed number from its type.

Worked check: millions of tiny files

An ext4 volume backing /var has room for data blocks, but file creation in an application directory fails. df -i /var shows all 6,553,600 inodes in use and zero available. The volume cannot allocate a new inode. Mail spools, session stores or applications that emit tiny logs can produce this symptom.

# GNU du counts inode use within /var without crossing into other filesystems.
du -x --inodes --max-depth=1 /var | sort -nr | head -n 15

Re-run that check inside the most affected subtree. Confirm which application owns the files and use its retention or cleanup procedure. A scan can be I/O intensive when it must walk millions of names. Schedule it accordingly, and re-run df -i afterward to verify recovery. To keep the condition from recurring, check whether an application is leaking files or retaining them longer than intended. If cleanup cannot meet demand, plan for a larger filesystem or an ext4 creation profile that accommodates more files.

Quota and read-only cases

On a quota-managed system, quota -s can show user quotas, while an administrator can use repquota on supported local filesystems or xfs_quota on XFS to inspect applicable limits. Check whether the failure concerns blocks or inodes and whether it affects a user, group or project. Prefer correcting excess usage before adjusting a limit. A read-only error is a different result. Inspect the actual mount options and recent kernel messages before deciding whether a policy change, recovery procedure or underlying storage repair is needed.

Deleted Files Holding Space (Open File Descriptors)

If df reports high filesystem usage but du accounts for far less, consider files that lost their last directory name while a process still held them open. unlink() removes a name and decreases the link count. When the last name is gone, the file persists while an open reference remains. du walks reachable names and misses it; df reports usage for the filesystem as a whole.

Check that both measurements concern the same mount and that the traversal had enough permissions. Other reasons for a discrepancy can include files hidden by an overmount, reserved blocks or filesystem metadata. Evidence of a large open file with a link count of zero supports the unlinked-file diagnosis.

Worked check: a released log name with an active writer

findmnt -T /var/log
lsof +aL1 /var
df -h /var

Use /var in the lsof command when it is the affected mounted filesystem. Without a mount restriction, lsof +L1 selects open entries with a link count below one. For example, one result can show a Java process at PID 4512 writing via descriptor 4, with a zero link count and a logical size of 85,899,345,920 bytes (80 GiB). Logical size does not necessarily equal blocks in use, especially for sparse files.

For a log, use the service’s documented log reopen operation or a controlled restart if required. Verify that all holders of the old file released it and compare df before and after. If the old content matters, preserve it under the incident procedure before closing the last reference. Writing through a descriptor’s procfs path can discard live data and does not coordinate the application’s writes, so it is not a routine space-recovery method.

Locating Storage Space Hogs

When a volume reaches saturation due to standard file accumulation, administrators employ targeted analysis workflows:

# 1. High-level filesystem overview: display filesystems, types, and free space
df -hT

# 2. Inspect root directory consumption without crossing into other mounted filesystems
du -x -h --max-depth=1 / | sort -hr | head -n 10

# 3. Zoom into saturated subtrees (e.g., /var) displaying immediate child directories only
du -x -h --max-depth=1 /var | sort -hr | head -n 10

# 4. Search for regular files larger than 1 GiB by apparent size
find /var -xdev -type f -size +1G -exec ls -lh {} +

Interactive Console Analysis via ncdu

ncdu scans a target tree and presents an interactive terminal view. Like du, it must traverse the data it measures. Use read-only mode for investigation:

# Scan /var confining search to single filesystem mount
ncdu -x -r /var
  • Navigation: Arrow keys navigate directory trees. Items are sorted automatically by size.
  • Actions: Press i to inspect the selected entry and q to exit. Read-only mode disables built-in deletion. Investigate errors and large files before choosing any remediation.

Filesystem Corruption & Repair Procedures

Read the kernel messages and identify the storage device and filesystem before starting a repair. If the device or storage path is unstable, preserve recoverable data or verify an independent backup first. Check the affected device’s identity, filesystem type and mount status. Repair can change metadata, so run it on an unmounted filesystem under a maintenance plan; for a root filesystem, use suitable rescue media and make sure it did not mount the target automatically.

ext4 with e2fsck

For an ext4 volume at /dev/sdb1 mounted on /mnt/data:

findmnt /mnt/data
# Stop services using it and close open files under the mount.
umount /mnt/data
# Confirm it is no longer mounted before operating on the device.
e2fsck -f /dev/sdb1

Review any requested repairs and the exit status before remounting. -f requests a check even when the filesystem is marked clean; -n opens it read-only and answers no to repairs. Results from checks on an active mount can be unreliable. If unmounting reports a busy target, use fuser -vm /mnt/data to locate users and stop or move them through the ordinary service controls.

XFS with xfs_repair

XFS normally replays its metadata log during mount. If a check finds that log recovery is pending and the storage is healthy, mount and unmount under controlled conditions to allow replay before attempting offline repairs. fsck.xfs is a compatibility stub in the usual configuration; e2fsck is for ext2, ext3 and ext4.

# Identify and unmount the XFS data volume first.
xfs_repair -n /dev/sdc1
# Review results and resolve any required log replay before writing repairs.
xfs_repair /dev/sdc1
Checkext filesystemsXFS
Read-only check of an unmounted filesysteme2fsck -fn DEVICExfs_repair -n DEVICE
Ordinary repair toole2fsck DEVICExfs_repair DEVICE
What to do if a dirty log prevents analysisFollow the ext tool and filesystem’s recovery guidanceOn healthy storage, attempt mount-time replay and clean unmount where possible

If XFS cannot replay a dirty log, xfs_repair -L can zero that log and discard pending metadata. The loss can include files, directory entries or changes to allocation. It does not guarantee that the volume can be mounted or that data will survive. Work from a backup or copy when practical and treat log zeroing as a last resort after reviewing recovery alternatives.

Drive Health & S.M.A.R.T. Telemetry Diagnostics

The smartmontools package supplies smartctl for device reports and smartd for monitoring. Options and reported fields vary across SATA/ATA, SAS/SCSI and NVMe. Some USB bridges or RAID controllers require an explicit device type to pass health commands to the drive. Verify that you are querying the intended device.

# Example for a supported ATA disk.
smartctl -i /dev/sda
smartctl -H /dev/sda
smartctl -a /dev/sda
# Schedule a self-test only if supported and appropriate.
smartctl -t short /dev/sda
smartctl -l selftest /dev/sda

Read the reported test duration and inspect the completed result. Avoid a long diagnostic on marginal media if it would interfere with prioritizing recovery.

Interpret health fields in context

On ATA drives, vendor-specific attributes may include counts for reallocated, pending or uncorrectable sectors and interface CRC errors. Use the vendor’s interpretation and normalized thresholds where applicable. Compare trends and correlate them with kernel messages and self-test results. Rising CRC errors can point to a link or cabling issue. A pending-sector count can support a media problem, but a single raw value is not a universal failure threshold. NVMe exposes different health fields such as critical warnings, available spare, and media or data integrity errors. An overall health pass does not resolve all storage faults.

Diagnose an unexpected read-only filesystem

Check the actual options with findmnt -T /var/app and look for recent kernel messages:

journalctl -k -b
# If appropriate, filter for your device and I/O or filesystem messages.
dmesg | grep -Ei 'I/O error|EXT4-fs|XFS|nvme|ata'

For example, block I/O errors can appear shortly before an ext4 log abort and a read-only remount when configured with errors=remount-ro. That sequence calls for investigating both the underlying media or path and the filesystem. A configured read-only mount can produce a write failure without any hardware error. Compare the actual device with the intended mount configuration and preserve data before trying a repair. Use the backup and recovery section’s mapfile-based workflow if a failing device needs imaging.

Test Your Knowledge

On the mounted /var filesystem, df shows near-full block usage. A same-filesystem du scan run with appropriate access accounts for far less. lsof reports a large regular file at link count zero still held open by the logger. Which action addresses the confirmed cause?

A

Format /var with more inodes

B

Use the logger’s documented reopen procedure or a controlled restart, then confirm it released the old file and filesystem space recovered

C

Run write-mode badblocks on the mounted device

D

Remove the current log’s directory entry again

Test Your Knowledge

A healthy storage copy of an XFS volume is unmounted and ordinary mount-time replay cannot recover its dirty log. What risk does the administrator take if using xfs_repair -L after reviewing recovery options?

A

The command converts it to ext4

B

The command performs a read-only check and leaves the log intact

C

The command schedules a safe online replay during the next boot

D

The command zeroes the XFS log and discards pending metadata, with possible loss of files or data

Test Your Knowledge

On an ext4 data mount, creating a file at /opt/app/config.json reports "No space left on device" while df shows 45 GiB of available data blocks. What command checks for inode exhaustion, and what is an appropriate next step if the filesystem has no free inodes?

A

Run df -i /opt/app, then identify concentrations of small files and apply the owner’s retention or cleanup procedure

B

Run smartctl -H, then raise the link speed

C

Run lsblk, then immediately reformat the filesystem

D

Run df -h again, then raise inode limits by editing /etc/fstab

Sections you finish are checked off in the contents.