7.3 File Signatures, Magic Bytes, Hex Headers/Footers & File Carving Techniques

7.3 File Signatures, Magic Bytes, Hex Headers/Footers & File Carving Techniques

Quick Answer: File signatures (magic bytes) are standardized byte sequences placed at fixed offsets (usually byte 0x00) that authoritatively identify a file's format regardless of its file extension. Malicious actors routinely disguise malware or exfiltrate sensitive data by altering extensions (e.g., renaming a Windows executable malware.exe to invoice.pdf). Forensic investigators expose extension mismatches by analyzing raw hex headers. Critical magic numbers include JPEG (FF D8 FF / trailer FF D9), PNG (89 50 4E 47 0D 0A 1A 0A / trailer 49 45 4E 44 AE 42 60 82), PDF (25 50 44 46 / trailer 25 25 45 4F 46), ZIP/Office XML (50 4B 03 04), and Windows PE (4D 5A with 50 45 00 00 at offset 0x3C). When file system metadata is wiped or corrupted, file carving reconstructs files from unallocated clusters using Header-Footer, File Length, or Bifragment Gap Carving with tools like foremost, scalpel, photorec, and bulk_extractor.

---

The Concept of File Signatures (Magic Bytes)

In modern operating systems, graphical file managers (such as Windows Explorer or macOS Finder) use file extensions (.docx, .jpg, .pdf) as superficial hints to select the default application for opening a file. However, file extensions provide zero cryptographic or forensic guarantee of file integrity.

Extension Spoofing vs. Magic Byte Analysis

Adversaries exploit operating system extension reliance through extension masquerading:

• Renaming a malicious Portable Executable (PE) payload.exe to employee_roster.xlsx allows it to bypass basic email attachment filters.
• Renaming an exfiltrated ZIP archive containing stolen intellectual property to system_diagnostic.log conceals outbound data transfer.
• Digital forensics suites (Autopsy, EnCase, FTK) execute automated File Extension Mismatch Analysis, flagging any file whose declared extension conflicts with the binary magic bytes extracted from its file header.

+---------------------------------------------------------------------------------------------------+
|                             EXTENSION SPOOFING DETECTION                                          |
+---------------------------------------------------------------------------------------------------+
| File Name:             quarterly_financials.xlsx                                                  |
| Declared Type:         Microsoft Excel Worksheet                                                  |
| Hex Bytes at 0x00:     4D 5A 90 00 03 00 00 00 ... (ASCII 'MZ')                                   |
| Hex Bytes at e_lfanew: 50 45 00 00                 (ASCII 'PE' followed by 0x00 0x00)             |
| Forensic Finding:      CRITICAL MISMATCH -> Windows Executable masquerading as an Excel workbook! |
+---------------------------------------------------------------------------------------------------+

---

Master Table of Critical CHFI File Signatures

The following hex headers, trailers, and offset rules represent core factual knowledge tested on the EC-Council CHFI exam:

---

Deep-Dive: Executable and Document Signatures

1. Windows Portable Executable (PE) Internal Offsets

Every Windows executable (.exe), dynamic link library (.dll), kernel driver (.sys), and Control Panel applet (.cpl) follows the Portable Executable format:

• Offset 0x00 - 0x01: DOS Header Magic 4D 5A (ASCII MZ, honoring Mark Zbikowski, architect of MS-DOS).
• Offset 0x3C - 0x3F (e_lfanew): A 4-byte little-endian pointer specifying the exact byte offset where the true NT PE File Header begins.
• PE Header Signature: At the offset specified by e_lfanew, the file MUST contain the 4-byte signature 50 45 00 00 (ASCII PE followed by two null bytes 0x00 0x00).
• Machine Architecture (Offset e_lfanew + 4): 0x014C indicates 32-bit x86; 0x8664 indicates 64-bit AMD64/x86_64.

+---------------------------------------------------------------------------------------------------+
|                         WINDOWS PORTABLE EXECUTABLE (PE) ANATOMY                                  |
+---------------------------------------------------------------------------------------------------+
| 0x00: DOS Header Magic -> 4D 5A ('MZ')                                                            |
| ...                                                                                               |
| 0x3C: e_lfanew Pointer -> E0 00 00 00 (Points to byte offset 0xE0)                                |
| ...                                                                                               |
| 0xE0: NT PE Signature  -> 50 45 00 00 ('PE' + 0x00 0x00)                                         |
| 0xE4: Machine Field    -> 64 86 (0x8664 = 64-Bit x86_64 Executable)                              |
+---------------------------------------------------------------------------------------------------+

2. Microsoft Office: OLE Compound Documents vs. OpenXML

• Legacy Office (DOC, XLS, PPT): Begin with the OLE CF (Compound File) Header: D0 CF 11 E0 A1 B1 1A E1 (colloquially remembered as "DocFile"). This is a structured virtual filesystem inside a single file.
• Modern Office (DOCX, XLSX, PPTX): Built upon Microsoft Office Open XML standards. Because OpenXML files are standard ZIP archives containing XML structures and media, they begin identically with the PK zip header: 50 4B 03 04.

3. Java Bytecode vs. Mach-O Fat Binary Collision

Notice that the hex sequence CA FE BA BE ("Babe Cafe") represents both:

1. A compiled Java Class File (.class).
2. A macOS Mach-O Fat / Universal Binary.

• Forensic Disambiguation: Examiners distinguish between them by inspecting bytes 4–7: in a Java class file, bytes 4–7 specify the Java minor/major version (e.g., 00 00 00 34 for Java 8); in a Mach-O fat binary, bytes 4–7 specify the number of fat architecture headers (fat_arch count).

---

File Carving Methodologies & Algorithms

File carving is the forensic process of reassembling and extracting files from raw unstructured data streams (such as unallocated space, drive slack, or raw memory dumps) without relying on file system metadata (such as MFT records, FAT directories, or ext4 inodes).

+---------------------------------------------------------------------------------------------------+
|                              FILE CARVING METHODOLOGIES                                           |
+---------------------------------------------------------------------------------------------------+
| 1. Header-Footer (SOF/EOF) -> Scans for Start-of-File and stops at End-of-File marker            |
| 2. File Length Carving      -> Reads internal length field in header and extracts exact byte count |
| 3. Bifragment Gap Carving   -> Carves fragmented clusters across non-contiguous gaps              |
+---------------------------------------------------------------------------------------------------+

1. Header-Footer (SOF / EOF) Carving

• Mechanism: The carver scans sector by sector for a known Start of File (SOF) signature (e.g., FF D8 FF for JPEG). Once detected, the carver writes subsequent contiguous sectors to an output file until it encounters the designated End of File (EOF) trailer (e.g., FF D9 for JPEG or 49 45 4E 44 AE 42 60 82 for PNG).
• Forensic Challenge - False Trailers: Embedded data can fool naive header-footer carvers. For example, a digital camera JPEG frequently contains a smaller thumbnail JPEG inside its EXIF metadata. A naive carver encountering the thumbnail's FF D9 trailer prematurely halts, yielding a corrupt, truncated image.

2. File Length / Structure Carving

• Many common file formats do not possess a static footer marker (e.g., BMP, WAV, AVI, ZIP). Instead, they define their file size within internal header structures:
  • Bitmap (BMP): Bytes 0x02 - 0x05 specify the total file size as a 32-bit little-endian integer.
  • PNG: Operates via sequential chunks (IHDR, IDAT, PLTE, IEND). Each chunk starts with a 4-byte length field. The carver jumps from chunk to chunk until the IEND chunk is processed.
  • ZIP / Office Documents: The carver locates the End of Central Directory (EOCD) signature 50 4B 05 06, parses the Central Directory offset, and computes the archive boundary.

3. Fragmentation Carving & Bifragment Gap Carving

On heavily utilized or aging storage drives, files are rarely written into contiguous physical sectors. When a file is fragmented, standard linear carving captures unrelated cluster data (gap data), producing corrupted files.

• Bifragment Gap Carving Algorithm:
  1. Locates the header (Fragment A) and potential footers (Fragment B) across unallocated space.
  2. Employs hypothesis testing: assumes the file is split into two fragments with an arbitrary intervening gap.
  3. Validates candidate reassemblies using semantic parsers, entropy transitions, or format-specific decompression checks (e.g., verifying that zlib/DEFLATE blocks decompress without CRC checksum errors).

---

Practical Forensic Carving Tools & Workflows

Digital forensics laboratories utilize specialized open-source and commercial utilities to carve unallocated space.

+---------------------------------------------------------------------------------------------------+
|                         FORENSIC CARVING TOOL COMPARISON                                          |
+-------------------+--------------------+------------------------+---------------------------------+
| Tool              | Author / Origin    | Operational Model      | Primary Forensic Strength       |
+-------------------+--------------------+------------------------+---------------------------------+
| **foremost**      | US Air Force OSI   | Rule-based linear scan | Lightweight, config-driven      |
| **scalpel**       | Golden G. Richard  | Two-pass indexed scan  | High-speed on massive TB images |
| **photorec**      | Christophe Grenier | Signature & structural | 480+ formats, ignores filesys   |
| **bulk_extractor**| Simson Garfinkel   | Feature stream parser  | Extracts URLs, PII, EXIF, zlib  |
+-------------------+--------------------+------------------------+---------------------------------+

1. foremost

Originally developed by the United States Air Force Office of Special Investigations (AFOSI). Reads configuration parameters from /etc/foremost.conf.

• Configuration Syntax: extension case_sensitive max_size header footer

  # Foremost configuration file syntax (hex values preceded by \x)
  # Format: extension  case_sensitive  max_size  header_signature  trailer_signature
  jpg   y   20000000   \xff\xd8\xff   \xff\xd9
  png   y   20000000   \x89\x50\x4e\x47\x0d\x0a\x1a\x0a   \x49\x45\x4e\x44\xae\x42\x60\x82
  pdf   y   50000000   \x25\x50\x44\x46   \x25\x25\x45\x4f\x46
  

• Command Execution:

  # Carve all supported image and document formats from unallocated space
  foremost -t jpg,png,pdf,docx -i /evidence/unallocated.raw -o /cases/carved_output/
  

2. scalpel

A total rewrite of foremost designed to eliminate excessive memory consumption and redundant disk I/O on massive storage volumes.

• Two-Pass Architecture: Pass 1 scans the entire raw disk image to build an in-memory index of all header and footer locations. Pass 2 resolves fragment boundaries and writes carved files to disk.
• Requires un-commenting target file signatures in scalpel.conf prior to execution:

  # Execute scalpel against raw disk image
  scalpel -c /etc/scalpel/scalpel.conf -o /cases/scalpel_output/ /evidence/disk_image.dd
  

3. photorec

Maintained by CGSecurity, photorec is an exceptionally robust carving utility bundled with testdisk. It recognizes over 480 file extensions and handles non-contiguous fragments by checking cluster continuity and internal file structures.

4. bulk_extractor

Unlike traditional carvers that output complete reconstructed files, bulk_extractor operates as a stream-based feature extraction tool.

• It scans every byte of raw disk images, pagefiles, or memory captures, completely ignoring file systems and partition tables.
• Carves and indexes high-value forensic tokens: email addresses, URLs, domain names, credit card numbers, phone numbers, GPS EXIF coordinates, and compressed zlib streams.
• Automatically decompresses internal raw zlib streams to carve nested artifacts (such as SQLite databases or browser caches embedded in RAM):

  # Run bulk_extractor across memory dump to extract telemetry and network tokens
  bulk_extractor -o /cases/bulk_results/ /evidence/memory.dmp
  

---

Carving Validation & False Positive Reduction

Carving algorithms inevitably produce false positives—random byte sequences that coincidentally match a 3-byte header signature.

• Validation Parsers: Forensic pipelines execute post-carve validation scripts (e.g., verifying that carved JPEGs can be rendered by libjpeg, or testing that carved ZIP archives pass unzip -t integrity validation).
• CRC32 Checksum Validation: Formats like PNG incorporate 4-byte CRC32 checksums at the end of each chunk. If the calculated CRC32 does not match the stored checksum, the fragment is discarded.
• Entropy Heuristics: Unallocated clusters containing encrypted volumes (BitLocker, FileVault) display near-maximum Shannon entropy (~7.99 bits/byte). Carvers must suppress header matching on high-entropy blocks to prevent carving garbled noise.