11.2 Linux and macOS Endpoint Incident Investigation

Linux and macOS Endpoint Incident Investigation

Investigating incidents across Unix-like systems—enterprise Linux hosting servers and macOS utilized on corporate workstations—demands specialized knowledge of system auditing architectures, process subsystems, and persistence mechanisms. Linux dominates cloud infrastructure and containers, making it a high-value target for web shells, cryptominers, ransomware, and rootkits. Concurrently, macOS endpoints present distinct targets for corporate espionage. Incident handlers must understand the logging subsystems, volatile memory artifacts, and configuration frameworks unique to Linux and macOS to identify, contain, and remediate intrusions.

Linux Log Repositories and Audit Subsystems

Linux systems store authentication and operational logs within /var/log/:

• System Event Logs: In Red Hat, CentOS, and Fedora, general system messages are captured in /var/log/messages. In Debian and Ubuntu, equivalent events are stored in /var/log/syslog. These logs record kernel errors, hardware faults, service startups, and daemon status changes.
• Authentication Logs: Authentication attempts, privilege shifts, and account management operations are logged to /var/log/secure on Red Hat/CentOS, and /var/log/auth.log on Debian/Ubuntu. Handlers examine these files for SSH authentications, PAM errors, sudo executions, and user additions.
• Linux Audit Daemon (auditd): The kernel auditing subsystem records configured system calls, file watches, and privilege events in /var/log/audit/audit.log. Local audit records are not inherently tamper-proof: a root-level adversary may disable audit rules, stop logging, or alter files. Forward records to a protected remote collector and alert on audit configuration changes or gaps. Handlers configure persistent rules in /etc/audit/rules.d/ and query events with dedicated tools. Analysts query audit records using specialized utilities:
  • ausearch: Searches audit logs for specific message types, syscalls, or timestamps (e.g., ausearch -m EXECVE -ts today).
  • aureport: Produces structured summary reports on executable runs, login attempts, syscall failures, and accounts.

User Activity, Session Tracking, and Shell Histories

Reconstructing user actions and logon activity requires analyzing binary accounting databases and user environment configurations:

• Binary Session Accounting Files: Unlike human-readable text logs, Linux maintains session records in binary structures:
  • utmp (/var/run/utmp): Tracks currently logged-in users, active terminal sessions, and system runlevels. Queried using w, who, and users.
  • wtmp (/var/log/wtmp): Binary historical database of logins, logouts, terminal sessions, and reboots. It is not inherently tamper-proof and may be rotated, truncated, or altered by privileged actors. Queried using the last command.
  • btmp (/var/log/btmp): Records failed login attempts. Queried using lastb, btmp provides critical telemetry for identifying brute-force attacks.
  • lastlog (/var/log/lastlog): Sparse database maintaining the most recent login timestamp, source terminal, and host IP for each user in /etc/passwd. Queried via the lastlog utility.
• User Account Databases: Handlers inspect /etc/passwd to identify unauthorized accounts, anomalous shells (such as /bin/sh on service accounts), and UID 0 accounts. The /etc/shadow file stores salted password hashes, password aging parameters, and account disable flags (denoted by ! or *).
• Shell Command Histories: User command histories are stored in home directories (e.g., ~/.bash_history, ~/.zsh_history). Handlers examine shell histories to reconstruct adversary commands. Adversaries deploy anti-forensic techniques: prepending commands with whitespace (when HISTCONTROL=ignorespace is set), running history -c, pointing HISTFILE to /dev/null, or terminating with kill -9 $$. Handlers treat missing or truncated histories as suspicious anomalies.

Persistence Mechanisms in Linux

Adversaries establish persistence across multiple configuration layers in Linux:

• Cron Scheduled Tasks: System-wide cron configurations reside in /etc/crontab, /etc/cron.d/, and hourly/daily directories (/etc/cron.hourly/, /etc/cron.daily/). User-specific schedules reside in spool directories (/var/spool/cron/crontabs/ or /var/spool/cron/). Attackers insert curl or bash reverse shells to regain access.
• systemd Services and Timers: Modern service management across Linux. Handlers inspect unit files in /etc/systemd/system/ (custom units) and /lib/systemd/system/ (system packages). Malicious services define ExecStart= commands pointing to backdoors with Restart=always. Attackers also configure .timer units to execute tasks on elapsed intervals.
• Initialization Scripts: Legacy startup scripts in /etc/rc.local, /etc/init.d/, /etc/profile.d/, and shell configurations (/etc/bash.bashrc, ~/.bashrc) execute unauthorized commands during boot or user logon.
• SSH Authorized Keys: A common backdoor involves appending an adversary public key into ~/.ssh/authorized_keys or /root/.ssh/authorized_keys, granting persistent, passwordless access.

Live Volatile Investigation and the /proc Filesystem

The /proc virtual filesystem presents real-time telemetry on running processes:

• Inspecting Process Subdirectories (/proc/[PID]/): For any suspicious PID, handlers inspect:
  • /proc/[PID]/exe: Symlink to the executable on disk. When an attacker deletes a running binary to thwart file scans, the link shows (deleted). Handlers can recover the intact binary by copying it directly: cp /proc/[PID]/exe /evidence/recovered_malware.bin.
  • /proc/[PID]/cmdline: Complete null-byte-delimited command invocation string and arguments.
  • /proc/[PID]/cwd: Symlink to the working directory.
  • /proc/[PID]/environ: Process environment variables, revealing injected dynamic libraries (LD_PRELOAD) or tokens.
  • /proc/[PID]/fd/: Open file descriptors, displaying open files, pipes, and sockets.
• Process and Network Commands: Handlers run ps auxef to display process hierarchies with full command lines, lsof -i to inspect open network sockets mapped to PIDs, and ss -tulpen to identify listening TCP/UDP sockets, process ownership, and inode numbers.

macOS Incident Investigation

macOS incorporates distinct Apple security architectures and logging subsystems:

• Apple Unified Logging System (ULS): Introduced in macOS 10.12, ULS stores telemetry in compressed tracev3 files in /var/db/diagnostics/ and /var/db/uuidtext/. Handlers query ULS using the log utility:
  • Historical query: log show --predicate 'process == "sudo"' --style syslog
  • Real-time stream: log stream --predicate 'eventMessage contains "failed"'
• LaunchDaemons and LaunchAgents: Apple launchd manages persistent processes via property lists (.plist):
  • LaunchDaemons (/Library/LaunchDaemons and /System/Library/LaunchDaemons): Run at boot as root, operating independently of user GUI logons. They represent primary targets for persistence.
  • LaunchAgents (/Library/LaunchAgents and ~/Library/LaunchAgents): Run only after a user logs into a graphical desktop session, executing within that user context.
• Quarantine Events: When a file is downloaded via browsers, email, or AirDrop, macOS attaches the extended attribute com.apple.quarantine. Handlers inspect this via xattr -p com.apple.quarantine [file]. Furthermore, macOS logs downloaded files in an SQLite database at ~/Library/Preferences/com.apple.LaunchServices.QuarantineEventsV*, recording file names, timestamps, and download URLs.
• Transparency, Consent, and Control (TCC): Regulates application access to sensitive resources (Camera, Microphone, Full Disk Access). Policies are stored in SQLite databases at /Library/Application Support/com.apple.TCC/TCC.db and per-user ~/Library/Application Support/com.apple.TCC/TCC.db. Handlers query the access table to identify unauthorized permission grants.
• FSEvents: Located in /.fseventsd/ on root volumes, FSEvents captures kernel file system transactions in gzip-compressed binary files, enabling reconstruction of file creations, modifications, and deletions.