3.4 Backup, Compression, Archiving & Data Transfer Operations

Key Takeaways

  • Archiving packages files and metadata. Compression changes the byte representation; a tool can combine both operations.

  • GNU tar supports gzip, bzip2 and xz through -z, -j and -J. List the archive and test a restore into a separate directory.

  • GNU cpio reads filenames for copy-out and pass-through, and reads the archive for copy-in. Newc and crc are distinct formats.

  • In rsync, a trailing slash on the source selects its contents. Archive mode excludes ACLs, extended attributes and hard-link preservation unless requested separately.

  • Use dd for a healthy block-copy source and GNU ddrescue with a mapfile for failing media. A recovery image can have gaps where data could not be read.

Last updated: November 2026

Backup, Compression, Archiving & Data Transfer Operations

Data preservation, systems imaging, and efficient synchronization represent critical operational competencies for Linux systems administrators. This chapter details essential operational fluency in archiving utilities (tar, cpio), compression trade-offs (gzip, bzip2, xz, 7z), remote synchronization via rsync, and low-level block imaging with dd and ddrescue.


1. Archiving with tar and cpio

A critical distinction in Linux storage administration is the difference between archiving and compression:

  • Archiving: Packaging multiple discrete files, directories, and their associated metadata (for example, ownership, permissions and timestamps) into a single contiguous byte stream or container file.
  • Compression: Encoding a data stream to reduce storage where possible. Compressed output can be larger for data with little remaining redundancy.

The Tape Archive Utility: tar

Originally developed to stream backups onto physical magnetic tape drives, tar is the ubiquitous archiving standard in Linux. Operating tar requires specifying exactly one primary operation mode, accompanied by optional operational modifiers:

Mode FlagLong OptionDescription
-c--createCreate a brand-new archive file.
-x--extractExtract files from an existing archive.
-t--listList archive contents without extracting files to disk.
-r--appendAppend new files to the end of an existing uncompressed archive.
-u--updateAppend files absent from the archive or newer than their archived copies, on a supported uncompressed archive.
-A--concatenateAppend individual tar archives into an existing tar file.

Crucial Modifiers and Options

  • -f <file>: Specifies the archive filename or device node. Supply the archive name as its argument, for example tar -cvf backup.tar data.
  • -v: Verbose output displaying files as they are processed.
  • -p: Preserves file permissions and mode bits during extraction (enabled by default for the superuser).
  • -C <dir>: Changes to the specified directory for the associated file arguments or extraction.
  • --exclude=<pattern>: Excludes files or directories matching a glob pattern.
  • --one-file-system: During recursive collection, avoids crossing filesystem boundaries. Review explicit sources and other paths before assuming everything on a mount is excluded.
# Create a gzip-compressed archive using relative member names.
tar --exclude='*.gz' -cvzf /backup/logs.tar.gz -C /var log

# Review the member names.
tar -tzf /backup/logs.tar.gz

# Restore into an existing, empty test directory.
tar -xzf /backup/logs.tar.gz -C /opt/restore

Absolute vs. Relative Paths in tar

By default, GNU tar strips the leading slash from member names. For predictable restores, archive from a known parent directory and keep member names relative. For example, archiving /etc/nginx stores records as etc/nginx:

  • Safety Rationale: An absolute destination name can target a live system file if absolute names are enabled during extraction and the caller can write that path. Review member names and extract into a separate empty directory before restoring to a live system.
  • Overriding Safety: The -P (or --absolute-names) flag explicitly forces tar to retain the leading forward slash, and permits absolute paths during extraction.

Copy In and Out with cpio

GNU cpio has three modes. Copy-out reads paths and writes an archive. Copy-in reads an archive for listing or extraction. Pass-through reads paths and copies files to a destination directory without writing an intermediate archive. The Linux kernel’s initramfs format uses the newc or crc cpio format.

# Collect relative names from a known directory; NUL separates paths.
(cd / && find etc -type f -name '*.conf' -print0) \
  | cpio -0 -o -H newc > /backup/etc_conf.cpio

# Inspect the members, then extract in an existing empty directory.
cpio -it < /backup/etc_conf.cpio
(cd /opt/restore && cpio -id --no-absolute-filenames < /backup/etc_conf.cpio)

# Copy a tree to an existing backup directory without an archive.
(cd /opt && find app -depth -print0) | cpio -0 -pd /mnt/storage/app_backup

-H newc selects the portable SVR4 format without a data checksum. The separate crc variant adds a data checksum. The -d option creates needed directories.

2. Compression Algorithms & Direct Inspection

Choose a compressor the receiving tools support. Compare size, memory use and time on representative data when performance matters. Results depend on the content, options and hardware.

UtilityUsual filenameUse
gzip and gunzip.gzCompress or decompress a stream. GNU tar selects gzip with -z.
bzip2 and bunzip2.bz2Compress or decompress a stream. GNU tar selects bzip2 with -j.
xz and unxz.xzCompress or decompress XZ streams, typically with LZMA2. GNU tar selects xz with -J.
7z or 7zz.7zArchive multiple files. The 7z format supports AES-256 encryption. Protect credentials and check that recovery is possible.
unzip.zipList with unzip -l backup.zip, test with unzip -t backup.zip, or extract into an existing directory with unzip backup.zip -d /opt/restore.

Read compressed text as a stream

These commands decompress suitable text to a stream without writing uncompressed copies to disk. List archive members with the archiver when you have a container holding multiple files:

# Concatenate and display compressed files to stdout
zcat /var/log/syslog.2.gz
bzcat /backup/audit.log.bz2
xzcat /backup/kernel.log.xz

# Search compressed text files directly using regular expressions
zgrep -i "segfault" /var/log/messages.1.gz

# Paginate through compressed files
zless /var/log/nginx/access.log.3.gz

3. Remote Synchronization with rsync

rsync copies files locally or remotely, for example through SSH. Its usual quick check uses size and modification time to decide whether to transfer a file. A remote transfer can use an existing destination file as a basis, sending unmatched data rather than retransmitting the whole file. Local transfers default to sending whole files. A path such as /backup/mirror/ is local from rsync’s perspective; use a host-qualified path for an SSH destination.

Syntax & Command Flags

rsync [OPTIONS] SOURCE DESTINATION
FlagLong OptionFunction
-a--archiveArchive mode: Equivalent to -rlptgoD, subject to privileges and support. Preservation of ACLs, extended attributes and hard links needs -A, -X and -H respectively when supported.
-v--verboseIncreases verbosity. Use --progress for transfer progress.
-z--compressCompresses file data streams in transit across the network.
-h--human-readableOutputs transfer numbers and sizes in human-readable units (K, M, G).
-P--partial --progressCombines --progress (shows real-time transfer bar and transfer speed) with --partial (keeps partial files for use by a later transfer).
--delete--deleteDeletes extraneous receiver entries in the synchronized directories, subject to filter and deletion options.
-e--rshSpecifies the remote shell transport (e.g., -e 'ssh -p 2222' for non-standard ports).
-n--dry-runSimulates the synchronization without modifying destination storage.

The Crucial Trailing Slash Semantics

For the directory examples below, assume the destination already exists. The source trailing slash selects its contents. Without the slash, rsync copies the named directory into the destination:

# Scenario A: Trailing slash on source path
rsync -av /data/src/ /backup/dest/
# Result: Copies the CONTENTS of /data/src/ directly into /backup/dest/
# Files appear at: /backup/dest/file1.txt, /backup/dest/file2.txt

# Scenario B: No trailing slash on source path
rsync -av /data/src /backup/dest/
# Result: Copies the DIRECTORY /data/src itself into /backup/dest/
# Files appear at: /backup/dest/src/file1.txt, /backup/dest/src/file2.txt

Safety Rule: When pairing rsync with --delete, always run a simulation first using -n (or --dry-run) combined with -v. Check both paths and the deletion list, preferably with -ni for itemized dry-run output. Rerun only after confirming that the selected transfer scope and deletion list match your intention.


4. Block-Level Imaging & Recovery: dd and ddrescue

Where file tools copy file contents and selected metadata, a disk image can include partition metadata and the bytes of filesystems and free space. Prepare a separate destination with enough capacity. For a consistent image, use an unmounted source or a suitable storage snapshot; a live image can combine data from different moments.

Readable source: GNU dd

# On a lab machine, verify /dev/sda and a separate mounted backup location first.
dd if=/dev/sda of=/backup/lab-disk.img bs=4M status=progress

# Copy the first 512 bytes for inspection; this does not back up a GPT.
dd if=/dev/sda of=/backup/lab-first-512.bin bs=512 count=1

if= sets the input, of= sets the output, bs= controls block size, and count= limits input blocks. status=progress shows transfer statistics in GNU dd. Check all paths before writing. Restoring the first 512 bytes of an image to a disk can overwrite partition or boot metadata.

conv=noerror,sync tells GNU dd to continue after read errors and pad short input blocks. Padding keeps place in the output under the documented rescue-copy conditions; it does not recreate unread bytes. A large block size can leave a sizeable region of padding after a read failure. Use a rescue tool designed to track bad regions if the source is failing.

Failing source: GNU ddrescue

# Use an unmounted source. Keep the image and mapfile on separate healthy storage.
ddrescue -n /dev/sdb /backup/lab-rescue.img /backup/lab-rescue.map
# Optionally retry later using the same paths and mapfile.
ddrescue -r3 /dev/sdb /backup/lab-rescue.img /backup/lab-rescue.map

GNU ddrescue prioritizes readable regions, then uses phases such as trimming and scraping to work closer to unreadable areas. The first command skips scraping to prioritize collection, and the second permits up to three retry passes after the other applicable phases. The mapfile records which regions are known to be copied or still need attention. Reuse it with the same source and destination when resuming, and inspect it before calling the image complete. Treat unrecovered regions as missing data. Diagnose and restore from a copy of the image when practical.

5. Verify a restore

For a file-level backup, list its contents, test the archive format when supported and extract into a clean staging directory. Check representative files, intended ownership and permissions, and whether the dependent service can use the restored state in a test environment. Copy to the live target only after confirming the destination and rollback plan.

Test Your Knowledge

The local directory /backup/mirror/ exists and is empty. An administrator runs rsync -av --delete /var/www /backup/mirror/. Where does the synchronized tree appear?

A

Directly under /backup/mirror/ without a www directory

B

The command stops because --delete requires a trailing slash on the source

C

Under /backup/mirror/www/

D

The command deletes the source tree after copying

Test Your Knowledge

A failing unmounted disk has unreadable sectors. The administrator needs to image as much as possible, resume a later pass, and track which byte ranges remain missing. The destination is healthy storage. Which command provides this workflow?

A

dd if=/dev/sdc of=/mnt/rescue/sdc.img bs=4M conv=noerror,sync

B

tar -cf /mnt/rescue/sdc.img /dev/sdc

C

rsync -a /dev/sdc /mnt/rescue/sdc.img

D

ddrescue -n /dev/sdc /mnt/rescue/sdc.img /mnt/rescue/sdc.map

Test Your Knowledge

Which utility searches gzip-compressed text logs for "Connection refused" without writing uncompressed copies to disk?

A

tar -xzf

B

zgrep

C

gunzip -c | grep -v

D

xzcat

Sections you finish are checked off in the contents.