10.2 Preserve System Journals

10.2 Preserve System Journals

Quick Answer: Make journals survive reboot by enabling persistent storage: create /var/log/journal and/or set Storage=persistent in /etc/systemd/journald.conf, then systemctl restart systemd-journald. Confirm journals appear under /var/log/journal and that journalctl -b -1 works after a reboot.

Why “preserve journals” is its own objective

Section 10.1 taught you to read logs. This objective teaches you to ensure journal data is kept on disk across reboots. On many RHEL installs, journald defaults to a mode where if /var/log/journal does not exist, journals stay in /run (tmpfs) and vanish on reboot. That breaks post-mortem analysis of the previous boot—exactly what you need after a failed change.

Typical EX200 wording:

• “Configure the system so that systemd journals are persistent.”
• “Ensure journal logs are preserved across reboots.”
• “Store journals under /var permanently.”

Success criteria: journals written under /var/log/journal, journald running with persistent storage, and previous boots still queryable after restart.

Volatile vs persistent storage

ls -la /run/log/journal
ls -la /var/log/journal

If only /run/log/journal/<machine-id>/ exists and /var/log/journal is missing, you are almost certainly not preserving journals yet.

Also check:

systemd-detect-virt   # optional context
df -h /var /run
hostnamectl           # machine identity; journal dirs often named by machine-id
cat /etc/machine-id

Journal files live in a subdirectory named after the machine-id.

Configuration file: journald.conf

Primary config:

sudo grep -E '^[#[:space:]]*Storage' /etc/systemd/journald.conf
sudo less /etc/systemd/journald.conf

Key setting:

[Journal]
Storage=persistent

Exam-safe approach used widely in Red Hat training:

1. Set Storage=persistent or ensure the directory exists for auto.
2. Create /var/log/journal explicitly.
3. Restart systemd-journald.
4. Reboot and verify previous boot still readable.

Drop-in snippets are also valid:

sudo mkdir -p /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/persistent.conf <<'EOF'
[Journal]
Storage=persistent
EOF

Prefer either editing the main file carefully (uncomment/change Storage=) or a drop-in—do not leave conflicting unknown customizations. On the exam, the simplest clear change that matches man page guidance is best.

# Example: uncomment/set in main file with your editor
sudo vim /etc/systemd/journald.conf
# Storage=persistent

Creating /var/log/journal (directory method)

Even with Storage=auto, creating the directory is a classic persistent-journal enablement:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
# or ensure ownership aligns with journald expectations:
sudo ls -ld /var/log/journal

After restarting journald, you should see a machine-id directory and *.journal files:

sudo systemctl restart systemd-journald
sudo ls -la /var/log/journal
sudo ls -la /var/log/journal/$(cat /etc/machine-id)

SELinux note: On enforcing systems, label issues are rare if you use mkdir on /var/log and let journald/tmpfiles set up, but if journals fail oddly, check context:

ls -ldZ /var/log/journal
# restorecon if you copied trees oddly
sudo restorecon -Rv /var/log/journal

Restart and flush

sudo systemctl restart systemd-journald
systemctl is-active systemd-journald
sudo journalctl --flush          # ask journald to flush to persistent storage when applicable
sudo journalctl --sync           # sync

systemctl restart systemd-journald is the usual post-config step. Avoid killing journald with kill -9 as a “fix.”

Verification that actually proves the objective

Live checks (before reboot)

# Persistent directory populated?
sudo find /var/log/journal -type f | head
# Config effective?
systemd-analyze cat-config systemd/journald.conf 2>/dev/null || true
grep -R Storage /etc/systemd/journald.conf /etc/systemd/journald.conf.d 2>/dev/null

After reboot (gold standard)

sudo systemctl reboot
# log back in
journalctl --list-boots
journalctl -b -1 -n 20 --no-pager

If -b -1 shows historical messages, persistence works. If --list-boots only shows the current boot and /var/log/journal is empty, recheck Storage= and directory creation.

Runtime storage indication

sudo journalctl -n 1 -o verbose | egrep '_BOOT_ID|_MACHINE_ID|SYSTEMD_IN_INITRD'
ls /run/log/journal /var/log/journal

Seeing active journal files under /var/log/journal is the practical proof used in labs.

Size limits and vacuum (supporting skills)

Persistent journals consume disk. journald enforces limits via settings such as:

SystemMaxUse=
SystemKeepFree=
SystemMaxFileSize=
RuntimeMaxUse=

Manage growth:

journalctl --disk-usage
sudo journalctl --vacuum-size=200M
sudo journalctl --vacuum-time=2weeks
sudo journalctl --vacuum-files=5

EX200 may not demand vacuum, but if /var fills, services fail—know that preserving journals does not mean unlimited growth. Do not vacuum away evidence mid-debug unless space is the problem.

Interaction with rsyslog and /var/log/messages

Persistent journal is independent of whether rsyslog writes /var/log/messages. You can have:

• Persistent journal + rsyslog files
• Volatile journal + rsyslog files (text history exists, journal previous boots do not)
• Persistent journal only

Tasks that say preserve system journals mean journald persistent storage, not “install a new syslog server.” Do not confuse copying /var/log/messages with fulfilling Storage=persistent.

Full exam procedure (recommended sequence)

Task: Ensure systemd journals persist across reboots.

# 1) Inspect current state
ls /var/log/journal 2>/dev/null || echo "no persistent dir"
ls /run/log/journal 2>/dev/null || true
grep -E '^#?Storage=' /etc/systemd/journald.conf

# 2) Configure
sudo mkdir -p /var/log/journal
sudo sed -i 's/^#\?Storage=.*/Storage=persistent/' /etc/systemd/journald.conf
# If sed is risky on exotic formatting, use vim and set Storage=persistent under [Journal]

# 3) Apply
sudo systemctl restart systemd-journald
sudo journalctl --flush

# 4) Confirm files on disk
sudo ls -la /var/log/journal/$(cat /etc/machine-id)

# 5) Reboot and re-check previous boot
sudo systemctl reboot
# after login:
journalctl --list-boots
journalctl -b -1 -n 5 --no-pager

Use an editor if you are not 100% confident in sed against commented lines—wrong Storage=none would be catastrophic for logging. Visual confirmation in vim is acceptable under exam conditions.

Permissions and ACLs (awareness)

Journal files are root-managed. Group systemd-journal (or adm on some distros) may read journals. For EX200 you operate as root via sudo; do not chmod 777 the journal directory. Keep standard ownership so journald can write.

sudo ls -la /var/log/journal
id

Common traps

1. Creating /var/log/journal but never restarting journald — old volatile runtime continues until restart/flush.
2. Setting Storage=persistent but /var is full — journald may fail to keep history; check df -h /var.
3. Assuming rsyslog files equal preserved journals — different subsystem.
4. Testing only with journalctl -b (current boot) — you must prove previous boot retention.
5. Deleting /var/log/journal to “free space” during the exam after enabling persistence — undoes the task.
6. Editing the wrong file under /usr/lib/systemd/ instead of /etc/systemd/ — vendor copies can be overwritten; use /etc.
7. Forgetting machine-id subdirectory and thinking persistence failed when files exist one level deeper.

Relationship to other operate-running-systems skills

• 10.1 Logs: Persistence makes multi-boot journalctl investigation possible.
• Boot/reboot: You often reboot to prove this objective.
• Disk space / storage chapters: Journals live on the filesystem that holds /var.
• Services: systemd-journald.service must stay active; do not mask it.

Optional: Forwarding and compress (do not over-scope)

Compress=yes
Seal=yes
ForwardToSyslog=yes

These appear in journald.conf but are not substitutes for Storage=persistent. Only change them if a task asks. Default exam focus is persistent storage on.

Section checkpoint

You should explain volatile /run vs persistent /var/log/journal, set Storage=persistent and/or create /var/log/journal, restart systemd-journald, verify journal files on disk, and prove with journalctl -b -1 / --list-boots after reboot that system journals are preserved. That meets the EX200 “preserve system journals” study point.