5.4 Advanced Text Filters: sort, uniq, wc, tr, split, less & Checksums (103.2)

5.4 Advanced Text Filters: sort, uniq, wc, tr, split, expand (103.2)

Quick Summary: Analyzing logs and processing system data streams requires advanced pipeline filtering. The LPIC-1 exam tests multi-key sorting with sort, duplicate suppression and frequency analysis with uniq (which strictly requires pre-sorted input), word and line metrics with wc, character translation and deletion with tr, file chunking with split, and tab whitespace expansion with expand and unexpand.

---

1. sort: Sorting Text Streams

The sort command rearranges lines of text from files or standard input into a sorted sequence. By default, sort evaluates lines lexicographically according to the system locale (ASCII collation where numbers precede uppercase letters, which precede lowercase letters).

Essential sort Command-Line Flags

Multi-Column Key Specification (-k)

The -k option takes a starting column position and an optional ending column position, along with individual column modifier flags: $\text{-k } \text{start}[\text{.char}][\text{flags}][,\text{end}[\text{.char}][\text{flags}]]$

# Sorting /etc/passwd numerically by UID (field 3), using ':' delimiter:
$ sort -t: -k3,3n /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
student:x:1000:1000:student:/home/student:/bin/bash

# Multi-key sort: sort first by department (field 2 alphabetically), then by salary (field 3 numerically descending):
$ sort -t, -k2,2 -k3,3nr employees.csv

# Sorting directly in-place without shell redirection data corruption:
$ sort -n -o numbers.txt numbers.txt

⚠️ LPIC-1 Trap: Running sort file.txt > file.txt with standard shell redirection empties the file before sort can read it because the shell truncates the file during tokenization. Always use sort -o file.txt file.txt for in-place sorting.

---

2. uniq: Duplicate Line Filtering & Reporting

The uniq filter compares adjacent lines and removes or reports duplicate occurrences.

CRITICAL PIPELINE RULE FOR uniq:
Unsorted Stream:                 Sorted Stream (sort):            Deduplicated (uniq):
┌──────────┐                     ┌──────────┐                     ┌──────────┐
│ apple    │                     │ apple    │                     │ apple    │
│ banana   │  ==> uniq fails!    │ apple    │  ==> uniq succeeds! │ banana   │
│ apple    │      (Non-adjacent) │ banana   │                     │ orange   │
│ orange   │                     │ orange   │                     └──────────┘
└──────────┘                     └──────────┘

The Fundamental Rule of uniq: uniq detects duplicates ONLY among adjacent, consecutive lines. To filter or count all duplicate entries across an entire file, you MUST pass the input through sort first: sort input.txt | uniq.

Essential uniq Command-Line Flags

# Counting frequency of failed login attempts by IP address:
$ awk '{print $1}' /var/log/auth_failures.log | sort | uniq -c | sort -nr
     42 192.168.1.105
     15 10.0.0.50
      3 172.16.4.12

# Extracting only duplicate usernames from a raw list:
$ sort raw_users.txt | uniq -d

# Extracting entries that occur exactly once (strictly unique):
$ sort raw_users.txt | uniq -u

3. wc: Word, Line, Character and Byte Counting

The wc (word count) utility analyzes text files or standard input streams to compute line, word, byte, and character metrics.

# Default wc invocation prints: [Lines] [Words] [Bytes] [Filename]
$ wc /etc/passwd
  45   85 2450 /etc/passwd

Essential wc Command-Line Flags

# Counting total active user accounts on the system:
$ cut -d: -f1 /etc/passwd | wc -l
45

# Inspecting byte size vs UTF-8 character count:
$ printf "München" | wc -c
8
$ printf "München" | wc -m
7

---

4. tr: Character Translation and Deletion

The tr (translate) utility transforms, compresses, or deletes individual characters from standard input and writes the result to standard output.

⚠️ LPIC-1 Trap: tr NEVER accepts file paths as command arguments. Running tr a-z A-Z file.txt results in a fatal syntax error. tr operates strictly on standard input via shell redirection or pipelines (tr 'a-z' 'A-Z' < file.txt or cat file.txt | tr ...).

Essential tr Syntax & Flags

$\text{tr [options] SET1 [SET2]}$

POSIX Character Classes for tr

• [:lower:]: All lowercase letters (a-z)
• [:upper:]: All uppercase letters (A-Z)
• [:digit:]: All numeric digits (0-9)
• [:alpha:]: All alphabetic characters ([:lower:] + [:upper:])
• [:alnum:]: All alphanumeric characters ([:alpha:] + [:digit:])
• [:space:]: All horizontal and vertical whitespace (spaces, tabs, newlines, carriage returns)
• [:blank:]: Horizontal whitespace (spaces and tabs only)

# Converting lowercase text to uppercase:
$ echo "linux professional institute" | tr '[:lower:]' '[:upper:]'
LINUX PROFESSIONAL INSTITUTE

# Stripping DOS carriage returns (\r) from a text file:
$ tr -d '\r' < dos_file.txt > unix_file.txt

# Squeezing multiple consecutive spaces into a single space:
$ echo "This    has    too   many   spaces" | tr -s ' '
This has too many spaces

# Generating a frequency list of unique words in a document:
$ tr -cs '[:alnum:]' '\n' < document.txt | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -nr

---

5. split: Splitting Files into Chunks

The split utility partitions a large file into smaller, fixed-size segments. By default, output chunks are named with prefix x followed by alphabetical suffixes (xaa, xab, xac...).

Essential split Command-Line Flags

# Splitting a 500MB tar archive into 100MB pieces with numeric extensions:
$ split -b 100M -d database.tar.gz db_part_
# Generates: db_part_00, db_part_01, db_part_02, db_part_03, db_part_04

# Reassembling split pieces into the original archive:
$ cat db_part_* > database_restored.tar.gz

---

6. expand and unexpand: Tab and Space Normalization

• expand: Converts TAB characters in files or stdin into spaces.
  • -t <tabstop> (or -<tabstop>): Sets tab stop width (default is 8 spaces). Example: expand -t 4 code.py > code_spaces.py.
  • --initial: Converts only leading tabs at the beginning of lines.
• unexpand: Converts spaces back into TAB characters.
  • -a (--all): Converts all sequences of spaces to tabs where possible (by default, unexpand only converts leading whitespace).
  • -t <tabstop>: Sets tab stops.

---

7. Paging Long Output with less

less is named in the 103.2 Terms and Utilities list, and it is the pager the exam assumes you use when a filter's output overflows one screen. Unlike more, less can scroll backwards and does not read the entire file before displaying the first page — which is why it opens a multi-gigabyte log instantly.

# Page through a filter pipeline
$ sort -k3 -n /etc/passwd | less

# Keep ANSI colours readable and allow line-by-line control
$ less -R /var/log/syslog

# Search from the start without wrapping the terminal
$ less -S wide_report.csv

Exam Rule: less is the pager man itself uses. The mnemonic LPI likes is that "less is more than more" — less is the superset with backward scrolling.

---

8. Integrity Checksums: md5sum, sha256sum and sha512sum

The 103.2 Terms and Utilities list includes three digest utilities: md5sum, sha256sum and sha512sum. They are filters in exactly the same sense as wc — they read a stream (or named files) and emit a fixed-size summary line — and the exam uses them to test whether a downloaded ISO or package payload is byte-identical to the publisher's original.

All three share one output format: <digest> <filename>, separated by two spaces.

# Generate a digest for one file
$ sha256sum debian-12.5.0-amd64-netinst.iso
a4c8f3...c91d  debian-12.5.0-amd64-netinst.iso

# Read from standard input (note the '-' filename in the output)
$ echo -n "linux" | md5sum
70f8c327d3f6bbfd... -

# Record digests for a whole directory
$ sha512sum /srv/release/*.tar.gz > SHA512SUMS

# Verify against a published list: -c reads the digest file and re-computes each entry
$ sha256sum -c SHA256SUMS
debian-12.5.0-amd64-netinst.iso: OK
broken-download.iso: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match

Verification flags shared by all three tools:

• -c (--check): Read digests from a file and verify each listed target. This is the flag exam items ask for.
• --quiet: With -c, print only the failures.
• --ignore-missing: With -c, do not fail on files listed but absent.
• -b / -t: Binary or text read mode (identical behaviour on Linux; the distinction exists for portability).

Exam Rule: A digest proves integrity, not authenticity. Anyone who can replace the ISO can replace the SHA256SUMS file next to it. Authenticity requires a GPG signature over the digest file — which is why distributions ship both SHA256SUMS and SHA256SUMS.gpg.

---

9. Reading Compressed Text Without Decompressing: zcat, bzcat, xzcat

Log files rotate into compressed archives, but you still need to filter them. The 103.2 objective lists three decompressing readers that write plain text to standard output, leaving the compressed file on disk untouched:

# Count error lines inside a rotated, gzip-compressed log
$ zcat /var/log/syslog.2.gz | grep -c ' ERROR '

# Chain a bzip2 archive straight into a field extractor
$ bzcat /var/log/audit/audit.log.1.bz2 | cut -d' ' -f1,3 | sort | uniq -c

# Feed an xz-compressed dump into a pager
$ xzcat backup.sql.xz | less

Exam Rule: The z* family extends to other filters as well — zgrep, zless, zdiff, and their bz/xz counterparts. None of these tools modify or delete the compressed source file; they only stream its decompressed contents.