6.1 Standard Streams, Redirection & File Descriptors (103.4)

6.1 Standard Streams, Redirection & File Descriptors

Quick Summary: Under the POSIX standard and the Unix philosophy, programs perform input and output operations through standardized byte streams. The Linux kernel assigns three default streams to every newly created process, represented by integer file descriptors: 0 for Standard Input (stdin), 1 for Standard Output (stdout), and 2 for Standard Error (stderr). Using shell redirection operators (<, >, >>, 2>, 2>&1, &>, <<, <<<), administrators can decouple programs from physical terminal hardware, routing data between files, processes, and virtual devices like /dev/null.

---

1. POSIX Standard Streams and File Descriptors (FD)

In Linux, the operating system treats virtually all resources—including regular files, directories, sockets, pipes, and hardware devices—as streams of bytes accessible via the filesystem. When a process opens a resource or is launched by the shell, the Linux kernel assigns it an unsigned integer handle known as a File Descriptor (FD).

By convention, every standard process starts execution with three file descriptors automatically opened and mapped to the controlling terminal session (/dev/pts/N or /dev/ttyN):

Default Process Stream Binding Architecture:
               ┌────────────────────────┐
               │     Linux Process      │
  Keyboard ───>│ FD 0 (stdin)           │
  (/dev/tty)   │                        │
               │ FD 1 (stdout) ────────>│ Display Screen
               │ FD 2 (stderr) ────────>│ (/dev/tty)
               └────────────────────────┘

Inspecting File Descriptors via /proc

The kernel exposes the active file descriptor bindings of every running process through the virtual procfs filesystem under /proc/<PID>/fd/. You can inspect the descriptor mappings of your current interactive shell (represented by the special variable $$):

$ ls -l /proc/$$/fd
total 0
lrwx------ 1 admin admin 64 Aug 29 10:00 0 -> /dev/pts/1
lrwx------ 1 admin admin 64 Aug 29 10:00 1 -> /dev/pts/1
lrwx------ 1 admin admin 64 Aug 29 10:00 2 -> /dev/pts/1
lrwx------ 1 admin admin 64 Aug 29 10:00 255 -> /dev/pts/1

All three descriptors point directly to the pseudo-terminal device /dev/pts/1. Redirection is the mechanism by which the shell alters these target pointers before launching an executable binary.

---

2. Standard Output and Error Redirection Operators

Redirection operators instruct the shell to intercept a stream and bind its file descriptor to a file or another descriptor instead of the default terminal display.

Output Redirection Operators Reference

Practical Demonstration: Separating stdout and stderr

Consider running find across /etc/ as an unprivileged user. Permitted directories yield valid paths (stdout), while restricted files produce permission warnings (stderr):

# 1. Default behavior: stdout and stderr intermingle on terminal display
$ find /etc -name "*.conf"
/etc/resolv.conf
find: ‘/etc/ssl/private’: Permission denied
/etc/sysctl.conf

# 2. Redirect standard output to a file, leaving errors on the screen:
$ find /etc -name "*.conf" > conf_files.txt
find: ‘/etc/ssl/private’: Permission denied

# 3. Redirect standard error to an error log, displaying successful results on screen:
$ find /etc -name "*.conf" 2> errors.log
/etc/resolv.conf
/etc/sysctl.conf

# 4. Simultaneously route stdout and stderr to two separate files:
$ find /etc -name "*.conf" > conf_files.txt 2> errors.log

---

3. Combining Streams: The Order of Evaluation Rule

A critical LPIC-1 exam objective tests how stdout and stderr are merged into a single destination file. The classic POSIX construct is:

command > file.log 2>&1

Step-by-Step Shell Evaluation of `command > file.log 2>&1`:
1. `> file.log`  --> Shell redirects FD 1 (stdout) to open `file.log`.
2. `2>&1`        --> Shell duplicates FD 2 (stderr) to point to the current target of FD 1 (`file.log`).
Result: Both stdout and stderr write into `file.log`.

⚠️ LPIC-1 Trap — Order of Redirection Evaluation Matters: If you invert the order and write:

command 2>&1 > file.log

The shell evaluates from left to right:

1. 2>&1: FD 2 (stderr) is redirected to where FD 1 points at that moment (the terminal display).
2. > file.log: FD 1 (stdout) is redirected to file.log. Result: stdout is written to file.log, but stderr continues to print directly to the screen! On the LPIC-1 exam, command > file 2>&1 is the correct portable POSIX syntax.

In modern Bash, the shorthand &> and &>> achieve identical results with fewer keystrokes:

# Truncate and write both streams to output.log:
$ systemctl restart nginx &> /var/log/nginx_init.log

# Append both streams to output.log:
$ systemctl restart nginx &>> /var/log/nginx_init.log

---

4. Standard Input Redirection: Files, Here Documents & Here Strings

While output redirection extracts data from programs, input redirection feeds data into a program's stdin (FD 0).

1. File Input Redirection (<)

Instead of reading keystrokes from the terminal keyboard, < binds FD 0 to an existing file on disk:

# Send email content from a file
$ mail -s "Daily Backup Report" admin@example.com < /var/log/backup.log

# Count lines using stdin redirection rather than passing an argument
$ wc -l < /etc/passwd
42

2. Here Documents (<< DELIMITER)

A Here Document (Heredoc) provides multi-line input directly within a shell script or interactive prompt until a designated delimiter token is encountered on a line by itself:

$ cat << EOF > /etc/motd
========================================
  Welcome to Enterprise Linux Server 9
  Authorized Access Only!
========================================
EOF

The <<- Tab Stripping Variant

When indenting shell scripts, leading tab characters can break formatting. Using <<- instructs the shell to strip all leading tab characters (not spaces) from each line of input and from the closing delimiter:

if [ -f /etc/config.json ]; then
	cat <<- 'CONFIG_EOF' > /tmp/parsed.json
		{
			"status": "active",
			"environment": "production"
		}
	CONFIG_EOF
fi

Exam Tip — Quoting the Heredoc Delimiter: If the delimiter is unquoted (<< EOF), variable expansion ($VAR) and command substitution ($(date)) are evaluated before writing. If the delimiter is quoted with single or double quotes (<< 'EOF' or << "EOF"), all text inside the heredoc is treated literally, suppressing all parameter and command expansions.

3. Here Strings (<<< "string")

Introduced in GNU Bash, a Here String feeds a single string variable or literal directly into stdin without requiring an external echo or printf pipe:

# Filter a string using bc arithmetic via Here String
$ bc <<< "scale=4; 355/113"
3.1415

# Pass variable text to tr
$ tr '[:lower:]' '[:upper:]' <<< "linux professional institute"
LINUX PROFESSIONAL INSTITUTE

---

5. Output Discarding and Clobber Protection

Discarding Output via /dev/null

/dev/null is a special character device in Linux known as the "bit bucket" or black hole. Any data written to /dev/null is immediately discarded by the kernel, and read operations return an immediate End-Of-File (EOF).

# Suppress both standard output and error messages silently:
$ ping -c 3 192.168.1.1 > /dev/null 2>&1

# Modern Bash equivalent:
$ ping -c 3 192.168.1.1 &> /dev/null

# Check only the exit status ($?) without cluttering terminal output:
$ if grep -q "root" /etc/passwd > /dev/null 2>&1; then echo "User exists"; fi

Clobber Protection: set -o noclobber and >|

By default, the > operator will overwrite and truncate an existing file without prompting, potentially causing catastrophic data loss if an administrator accidentally redirects output to a critical system configuration.

To prevent this, the shell provides the noclobber option:

• set -o noclobber (or set -C): Enables overwrite protection. If target file exists, redirection with > aborts with an error.
• set +o noclobber (or set +C): Disables protection (default shell state).

$ set -o noclobber
$ echo "Important Configuration" > /etc/important.conf
$ echo "New Line" > /etc/important.conf
-bash: /etc/important.conf: cannot overwrite existing file

The Force Override Operator: >|

When noclobber is active, an administrator can explicitly force the truncation and overwrite of an existing file using the >| operator:

$ echo "Forced New Configuration" >| /etc/important.conf
# Successfully overwrites the file despite noclobber being active

💡 LPIC-1 Exam Fill-in-the-Blank Alert: What shell operator forces output redirection to truncate and overwrite an existing file when the noclobber shell option is enabled? Answer: >|