5.3 Basic Text Filters: cat, head, tail, cut, paste, nl, od (103.2)

5.3 Basic Text Filters: cat, head, tail, cut, paste, nl, od (103.2)

Quick Summary: In the Unix philosophy, programs act as filters that accept text streams from standard input (stdin), transform the data, and emit results to standard output (stdout). Core utilities including cat, head, tail, cut, paste, nl, and od provide the primary toolchain for extracting, formatting, inspecting, and analyzing system configuration files and log streams on the LPIC-1 exam.

---

1. cat and tac: Concatenation and File Display

The cat (concatenate) utility reads files sequentially, writing their contents to standard output. It is also used to combine multiple files into a single destination file.

Essential cat Command-Line Flags

# Inspecting a configuration file with line numbers on non-empty lines and showing tabs:
$ cat -b -T /etc/hosts
     1  127.0.0.1^Ilocalhost
     2  ::1^Ilocalhost ip6-localhost ip6-loopback
     3  192.168.1.50^Iserver01.example.internal server01

# Locating problematic DOS/Windows CRLF carriage returns (^M):
$ cat -v /tmp/windows_script.sh
#!/bin/bash^M
echo "Running script"^M

The tac Command

tac is cat spelled backwards. It concatenates and displays files in reverse line order, printing the last line first and the first line last. It is invaluable for reviewing log files where the newest entries appear at the bottom.

---

2. head and tail: Header and Footer Slicing

The head Command

By default, head prints the first 10 lines of the specified file or standard input stream.

• -n <count> (or -<count>): Prints the first <count> lines (e.g., head -n 5 /etc/passwd or head -5 /etc/passwd).
• -n -<count> (Negative Count): Prints all lines of the file except the last <count> lines (e.g., head -n -20 log.txt prints everything except the final 20 lines).
• -c <bytes>: Prints the first <bytes> bytes of the file (e.g., head -c 512 /dev/urandom).
• -q (--quiet): Never prints file name header banners when multiple files are processed.
• -v (--verbose): Always prints file name header banners.

The tail Command

By default, tail prints the last 10 lines of the specified file.

• -n <count> (or -<count>): Prints the last <count> lines (e.g., tail -n 25 /var/log/syslog).
• -n +<count> (Positive Leading Count): Starts output at line number <count> and prints continuously to the end of the file (e.g., tail -n +2 data.csv skips the first CSV header row).
• -c <bytes>: Prints the last <bytes> bytes of the file.
• -f (--follow): Actively monitors and outputs appended data in real time as the file grows. Tracks the file by its underlying file descriptor.
• -F (--follow=name --retry): Follows the file by filename rather than file descriptor, actively retrying if the file is rotated, deleted, or recreated. Essential for tracking logs managed by logrotate.

# Real-time tracking of authentication logs across log rotations:
sudo tail -F /var/log/auth.log

# Combining head and tail to extract lines 20 through 30 of a file:
cat -n /etc/services | head -n 30 | tail -n 11

💡 LPIC-1 Exam Fill-in-the-Blank Alert: To follow the growth of an active log file in real time by file descriptor, the command and option is tail -f.

3. cut: Column and Field Extraction

The cut utility extracts sections, columns, or character ranges from each line of input files or stdin.

Essential cut Command-Line Flags

Field Selection Range Syntax

• N: The N-th field or character (1-indexed).
• N-M: From field N through field M inclusive.
• -M: From the beginning of the line through field M (e.g., cut -f -3).
• N-: From field N through the end of the line (e.g., cut -f 4-).
• 1,3,5: Discrete fields 1, 3, and 5.

# Extracting usernames (field 1) and default shells (field 7) from /etc/passwd:
$ cut -d: -f1,7 /etc/passwd
root:/bin/bash
daemon:/usr/sbin/nologin
student:/bin/bash

# Extracting all fields EXCEPT the encrypted password placeholder (field 2):
$ cut -d: --complement -f2 /etc/passwd

# Changing output delimiter to a tab:
$ cut -d: -f1,7 --output-delimiter=$'	' /etc/passwd

⚠️ LPIC-1 Trap: cut cannot reorder fields. Running cut -d: -f7,1 /etc/passwd still outputs field 1 followed by field 7 in their original stream order. To reorder columns, use awk or a while read shell loop.

---

4. paste: Horizontal Stream Merging

While cat merges files vertically (end-to-end), paste merges files horizontally, joining corresponding lines side-by-side separated by delimiter characters.

Horizontal Merging Mechanics with paste:
File 1 (users.txt):    File 2 (uids.txt):     paste -d: users.txt uids.txt
┌──────────┐           ┌──────────┐           ┌──────────────────┐
│ root     │           │ 0        │           │ root:0           │
│ bin      │     +     │ 1        │    =>     │ bin:1            │
│ student  │           │ 1000     │           │ student:1000     │
└──────────┘           └──────────┘           └──────────────────┘

Essential paste Flags

• -d '<delims>': Specifies custom delimiter character(s) (default is TAB). If multiple delimiters are given (e.g., -d ',:'), paste cycles through them across columns.
• -s (--serial): Pastes all lines of one file at a time serially into a single horizontal line, rather than merging lines in parallel.

# Converting a vertical list of hostnames into a single comma-separated line:
$ cat hosts.txt
web01
web02
web03
$ paste -s -d, hosts.txt
web01,web02,web03

# Grouping stdin stream into three parallel columns using hyphen placeholders:
$ seq 1 9 | paste - - -
1	2	3
4	5	6
7	8	9

---

5. nl: Advanced Line Numbering

The nl utility numbers lines from files or stdin with rich support for header, body, and footer logical page formatting.

# Numbering lines starting at 100 with step increments of 5:
$ nl -v 100 -i 5 -s ": " sample.txt
   100: First entry
   105: Second entry
   110: Third entry

---

6. od: Octal, Hexadecimal & Binary Inspection

The od (Octal Dump) utility dumps input files or streams in unambiguous octal, hexadecimal, decimal, or ASCII character representations. It is essential for detecting non-printing characters, null bytes (\0), BOM markers, and Windows CRLF line terminations.

Common od Invocation Formats

• od -c: Dumps input as standard ASCII characters or backslash escapes (\n, \r, \t, \0).
• od -x: Dumps input in 2-byte units formatted as hexadecimal (shorthand for od -t x2).
• od -o: Dumps input in 2-byte units formatted as octal (shorthand for od -t o2).
• od -b: Dumps input as single-byte octal values (shorthand for od -t o1).
• od -t <type> (--format=<type>): Explicit format type specifier:
  • c: Printable character or backslash escape.
  • x1, x2, x4, x8: Hexadecimal in 1, 2, 4, or 8-byte integers.
  • u1, u2, u4: Unsigned decimal.
  • d1, d2, d4: Signed decimal.
  • fF, fD: Single or double precision floating point.

# Detecting carriage returns (\r) and newlines (\n) in a text string:
$ printf "Hello\r\nWorld\n" | od -c
0000000   H   e   l   l   o  \r  \n   W   o   r   l   d  \n
0000015

# Inspecting byte stream in 1-byte hexadecimal format with address offsets:
$ printf "LPIC-1" | od -t x1 -A d
0000000  4c  50  49  43  2d  31
0000006