5.3 Process Monitoring, Signals, Job Control & Scheduling Priorities
Key Takeaways
Check PID, start time, parent and state before doing anything to a process. A zombie has exited and waits for reaping by a parent or an adopter.
Use interval CPU and I/O measures to interpret a slowdown. Load averages include runnable and uninterruptibly sleeping tasks.
SIGKILL and SIGSTOP cannot be caught, blocked, or ignored. A task in an uninterruptible kernel wait can remain after a fatal signal is sent.
Bash job control uses process groups. Foreground placement retains existing redirections, and nohup only addresses hangup behavior.
GNU nice adjusts the inherited value. A running task’s nice value and resource limits depend on its scope, permissions and the installed tools.
5.3 Process Monitoring, Signals, Job Control & Scheduling Priorities
Before signaling or changing a process, verify its identity, its state and the service or interactive job that owns it. For a slowdown, compare samples over the interval in which the symptom occurred.
Process identity and states
A process has a process ID (PID) and can have multiple threads. Its parent process ID (PPID) identifies the parent visible in the current PID namespace. The first userspace process is PID 1 in the initial namespace. Its program depends on the system. A child may be created by fork() and then call execve() to replace its program while retaining its PID. Shell jobs can represent a whole pipeline in a process group, so a job number and a PID identify different things.
| State in Linux process reports | How to interpret it |
|---|---|
R | Running on a CPU or runnable and waiting for CPU time. |
S | Interruptible sleep while waiting for an event. An appropriate signal can interrupt the wait. |
D | Uninterruptible kernel sleep. Some waits involve storage or remote filesystems. A pending fatal signal can be delayed until a truly uninterruptible wait ends. Some kernel waits permit a fatal signal to interrupt them, so use the task's evidence to diagnose a persistent wait. |
T or t | Stopped by job control, or stopped during tracing, respectively, in current procps output. Use SIGCONT to resume a job-control stop. Trace stops are managed by the tracer. |
Z | A child has exited but a wait has yet to collect its exit status. A zombie has no program left to stop, although the kernel retains a process-table entry. |
An orphan can be alive. If a parent exits, Linux can reparent a child to the nearest living ancestor configured as a child subreaper, or to the applicable namespace init process. A zombie needs a responsible process to call a wait operation. For repeated zombies, inspect the PPID and service, and fix or coordinate recovery of the program that should wait for its children. Avoid a blanket parent kill. It can interrupt unrelated children or a critical service, and a replacement parent may be a subreaper.
Inspect processes and the affected resource
ps aux
ps -ef
ps -eo pid,ppid,stat,ni,%cpu,rss,comm --sort=-%cpu
pstree -p
Procps supports BSD-style options such as ps aux, Unix-style options such as ps -ef and GNU-style long options. With procps, ps CPU percentage summarizes CPU time relative to elapsed process lifetime. Use interval reports to investigate a current spike. RSS is resident memory attributed to the task and can include shared pages. It does not show the amount of RAM unique to that process.
| Evidence | Common tools and applications |
|---|---|
| Current system and task activity | top refreshes task and system metrics. In procps top, 1 toggles per-CPU views and P or M sorts by CPU or memory. htop offers an interactive view, including a process tree. Where collection is configured, atop can also read historical activity. |
| Interval CPU activity | mpstat -P ALL 2 5 samples per-CPU activity. pidstat -u 2 5 samples task CPU activity, pidstat -r reports memory statistics and page faults, and pidstat -d reports I/O statistics when available. |
| References, system calls and signals | lsof -p 4501 looks for opened objects visible to the caller. strace -p 4501 traces a selected process when permitted and can add overhead. Try a disposable command such as strace -c ls for a system-call summary. |
| Parent chain or ownership | pstree -p shows relationships and PIDs. Use ps -p 4501 -o pid,ppid,stat,lstart,comm to verify a candidate PID and its start time. Recheck a stale PID because it can be reused. |
Load averages summarize runnable and uninterruptibly sleeping tasks. If load is high and a process reports D, compare interval CPU, I/O and wait evidence before inferring a bottleneck. For example, multiple tasks in D and rising disk request latency suggest investigating the affected storage path. A load value by itself provides no CPU utilization percentage. Use the performance and storage troubleshooting procedures to distinguish a slow volume or remote mount from CPU contention.
With sufficient permission, inspect these entries for the same PID, shown here as 4501:
Path under /proc/4501/ | What it tells you |
|---|---|
status or stat | Sampled state, identifiers and related counters. For status, the UID/GID rows include real, effective, saved-set and filesystem IDs. |
cmdline, exe, cwd | NUL-delimited argument data, link to the running program image, and working-directory link. A program can change how its arguments appear. Executable and directory paths may refer to unlinked objects. |
fd/ | The file descriptors visible for this process. An open descriptor can hold a removed file's storage as discussed in storage troubleshooting. |
environ | Normally the initial environment at program execution. Later changes may not be reflected. Avoid collecting the values unnecessarily. |
limits | The current soft and hard limits for this process. |
Deliver a signal with the intended scope
Use kill -s TERM 4501 for a verified PID when termination is authorized. The return status from the send operation does not establish that the target exited. Recheck the process. Procps pgrep -x worker previews name matches. pkill -TERM -x worker signals visible matching tasks when permitted. Use an owner or other constraint where needed. The -f selector matches the full command line, which may produce additional matches. On Linux with psmisc, killall sends to processes selected by name. Inspect the target set before using a broad name selector.
| Signal | Default and practical interpretation |
|---|---|
SIGHUP (1 on Linux x86) | Terminates by default. It can indicate a hangup. A daemon reloads configuration on HUP only if it implements that behavior. |
SIGTERM (15) | Terminates by default. A service can handle it to implement orderly shutdown. Verify the outcome before escalating. |
SIGKILL (9) | Forces termination when it takes effect, with no process cleanup handler. The signal cannot be caught, blocked or ignored. A task in an uninterruptible kernel wait can remain visible. |
SIGSTOP and SIGCONT | STOP suspends execution when it takes effect and cannot be caught, blocked or ignored. CONT resumes a job-control stop. |
SIGINT and SIGTSTP | With ordinary terminal settings and signal processing enabled, Ctrl+C sends INT to the foreground process group and Ctrl+Z sends TSTP. The application can handle or ignore these signals. |
For a supervised workload, use the service manager's documented stop or reload operation as appropriate. If recovery requires a forced stop, confirm PID and workload, then check whether it exited and whether dependent services are healthy. For a stalled kernel wait, investigate the wait condition even if a termination signal has already been sent.
Bash job control and shell lifetime
In an interactive Bash shell with job control enabled, sleep 60 & starts a background job. jobs -l displays the shell's jobs and process IDs. If a foreground sleep 60 is stopped by Ctrl+Z, look up its job number. Assuming job 1 was stopped, bg %1 resumes it in the background and fg %1 puts it in the foreground. Foreground placement changes the controlling terminal's foreground process group. It preserves redirections already set up for the job.
Ctrl+D usually signals end of input when the input buffer is empty. A command reading the terminal can respond to EOF, and an interactive shell can exit unless configured to ignore it. exec program replaces the current shell process on success. In the calling interactive shell this also replaces its prompt and job management state, so practice inside a disposable shell.
nohup ./batch_processor.sh </dev/null > processor.log 2>&1 &
GNU nohup starts the command with HUP ignored and supplies default terminal redirections where needed. The command above explicitly chooses descriptor destinations and uses & to run asynchronously. In Bash, disown -h %1 marks a known job against Bash's HUP forwarding and disown %1 removes it from the job table. These tools address hangups from applicable sources. Check service or login-session policy separately for cleanup on logout.
Niceness and process limits
Linux nice values run from -20 through 19. Lower values can give a normal scheduled task more relative CPU weight under contention. A child inherits its parent's nice value. CPU placement, policy, cgroup weight and other activity also affect the observed outcome.
nice -n 10 ./compute.sh
ps -p 4501 -o pid,ni,comm
renice --priority 15 -p 4501
GNU nice -n 10 adds 10 to the inherited nice value. Starting at 0 gives 10, while starting at 5 gives 15. The last command uses a util-linux release with --priority, which sets the target value explicitly. Check your release for options. On util-linux with -n, the interpretation can change from absolute to relative when POSIXLY_CORRECT is set. An ordinary caller can normally increase its own task's nice value. Decreasing it depends on appropriate CAP_SYS_NICE privilege or a sufficient RLIMIT_NICE for the permitted target.
A soft resource limit is enforced. The corresponding hard limit bounds what an unprivileged process may set as its soft limit. For example, ulimit -Sn and ulimit -Hn show Bash's current open-file limits. A launched child inherits limits. Compare /proc/4501/limits with any systemd unit's settings for a service. Changing limits in a separate shell does not update an existing service.
Schedule one-time or recurring jobs
| Need | Tool and example |
|---|---|
| Run a job once | With an available atd, use at -f lab-report.sh 18:30 tomorrow. Inspect the queue with atq, and cancel your own scheduled lab job by its job number with atrm. |
| Run at a recurring time | Use crontab -e to edit your user schedule and crontab -l to inspect it. For a typical five-field user crontab, 15 2 * * * /home/jdoe/bin/report.sh schedules the script at 02:15 each day when the scheduler and host are available. The fields are minute, hour, day of month, month, and day of week. System crontabs commonly add a username field. |
| Catch up a periodic maintenance task | anacron checks day-based frequencies using timestamps. After the machine is available and anacron runs, a due task in /etc/anacrontab can run after its configured delay. It does not replay each missed day. |
Use absolute paths where practical and test the job with the intended account and scheduler environment. Review scheduler and service logs to verify execution.
An administrator is authorized to stop a verified maintenance process and already tried SIGTERM. Which signal terminates when it takes effect and cannot be caught, blocked, or ignored?
SIGHUP
SIGKILL
SIGSTOP
SIGQUIT
A managed Linux worker repeatedly leaves children in state Z under the same live parent. Which response addresses the cause while preserving service continuity?
Increase the CPU priority of each zombie
Send SIGCONT to the exited children
Reboot immediately because all zombies indicate filesystem corruption
Identify the parent and coordinate correcting its child-wait behavior
A normal Linux task starts at nice 0. Its owner has no CAP_SYS_NICE and no RLIMIT_NICE allowance to lower its nice value. Why does GNU nice -n -10 report permission denied for the adjustment?
The request would lower the inherited value to -10, which this caller lacks permission to do
The Linux nice range runs from 0 to 40
Only ELF programs can be run with changed niceness
The same caller can bypass this limit by using renice
Sections you finish are checked off in the contents.