16.2 Python for Digital Forensics: Parsing Raw Artifacts, Custom Scapy Scripts & Evidence Automation

16.2 Python for Digital Forensics: Parsing Raw Artifacts, Custom Scapy Scripts & Evidence Automation

Quick Answer: Python serves as the primary scripting language in digital forensics and incident response (DFIR). It enables investigators to bridge gaps left by commercial suites, parse bespoke and corrupted artifacts, and automate repetitive triage tasks. Using the standard library, examiners utilize hashlib to compute multi-hash verification streams across forensic images, struct to unpack binary disk and memory headers using explicit endianness specifiers (< little-endian, > big-endian), and datetime to convert 64-bit Windows FILETIME timestamps (100-nanosecond intervals since Jan 1, 1601) and Unix epochs. For specialized analysis, third-party libraries like pefile extract Portable Executable metadata and compute ImpHash for malware clustering, Scapy dissects network PCAPs to isolate DNS and HTTP indicators, and python-registry parses offline Windows Registry hives.

---

The Role of Scripting in Modern DFIR Workflows

Commercial forensic platforms (such as EnCase, FTK, and Axiom) provide broad coverage for recognized file systems and standard operating system artifacts. However, enterprise forensic investigations routinely encounter operational challenges that commercial monolithic suites cannot resolve:

1. Proprietary and Bespoke Artifacts: Malware authors, specialized enterprise applications, and cloud services frequently write data in proprietary binary formats, custom structured databases, or encrypted logs for which no off-the-shelf parser exists.
2. Corrupted or Incomplete Artifacts: File system corruption, partial file carving, or damaged Volume Shadow Copies can cause commercial software to crash, skip records, or discard valid data blocks.
3. Triage Scale and Rapid Ingestion: During enterprise incident response engagements involving thousands of endpoints, loading complete disk images into heavy GUI tools creates severe operational bottlenecks. Lightweight Python triage scripts can rapidly ingest targeted artifacts (such as Prefetch files, Amcache hives, Shimcache entries, and Event Logs) and generate consolidated timelines within minutes.
4. Scientific Repeatability and Defensibility: Commercial tools operate largely as closed-source 'black boxes'. In contrast, a well-documented, open Python script can be inspected, mathematically validated, peer-reviewed, and presented in court under Federal Rules of Evidence with complete procedural transparency.

Core Engineering Tenets of Forensic Scripting

• Non-Destructive Operations: Scripts must open files strictly in read-only binary mode (open(filename, 'rb')). Write-access handles or in-place modifications violate basic evidence preservation rules.
• Deterministic Output: Given the identical input data, a forensic script must always generate the exact same output, hash values, and parsed fields across different runs and host operating systems.
• Cryptographic Stream Verification: Forensic scripts must verify evidence integrity by calculating hashes (MD5, SHA-1, SHA-256) before and after parsing operations.
• Granular Exception Handling: Incomplete records, malformed byte sequences, or unallocated slack space must not cause the script to abort. Unparseable bytes should be logged as anomalies while execution continues across subsequent records.

---

Python Standard Library: Low-Level Forensic Foundations

1. os and sys: Safe File System Traversal

The os module allows examiners to traverse evidence directory structures recursively without altering file system metadata:

import os

def scan_evidence_directory(evidence_root):
    for root, dirs, files in os.walk(evidence_root):
        for file in files:
            file_path = os.path.join(root, file)
            try:
                stat_info = os.stat(file_path)
                # Extracting file size and timestamps (seconds since Unix epoch)
                print(f"File: {file_path} | Size: {stat_info.st_size} bytes | Modified: {stat_info.st_mtime}")
            except (PermissionError, FileNotFoundError) as e:
                print(f"[ERROR] Could not access {file_path}: {e}")

2. hashlib: Multi-Hash Stream Generation

Forensic standards require calculating multiple cryptographic hashes (e.g., MD5 and SHA-256) to confirm evidence integrity. Reading a 4 TB forensic image from disk multiple times to compute each hash separately is inefficient and strains disk I/O. Using hashlib, an investigator can compute multiple cryptographic hashes simultaneously in a single read pass using a chunked buffer:

import hashlib

def compute_multi_hash(file_path, chunk_size=1048576):  # 1 MB buffer
    md5 = hashlib.md5()
    sha1 = hashlib.sha1()
    sha256 = hashlib.sha256()
    
    with open(file_path, 'rb') as f:
        while chunk := f.read(chunk_size):
            md5.update(chunk)
            sha1.update(chunk)
            sha256.update(chunk)
            
    return {
        'MD5': md5.hexdigest(),
        'SHA1': sha1.hexdigest(),
        'SHA256': sha256.hexdigest()
    }

3. struct: Binary Header Unpacking

Low-level forensic artifacts (such as Master File Table $MFT records, Windows Prefetch files, and binary registry hives) store metadata as raw bytes. The struct module unpacks binary buffers into native Python data types (integers, strings, floats) according to explicit format strings.

Endianness and Format Character Reference

• Endianness Specifiers:
  • <: Little-endian (least significant byte stored first—standard for x86/x64 architectures, Windows file systems, NTFS, and FAT).
  • >: Big-endian (most significant byte stored first—standard for network byte order, TCP/IP headers, and older PowerPC/SPARC architectures).
  • =: Native byte order.
• Type Characters:
  • b / B: Signed / Unsigned 1-byte integer (char).
  • h / H: Signed / Unsigned 2-byte integer (short, 16-bit).
  • i / I: Signed / Unsigned 4-byte integer (int / long, 32-bit).
  • q / Q: Signed / Unsigned 8-byte integer (long long, 64-bit).
  • s: Byte string (e.g., 4s unpacks a 4-byte ASCII string like FILE or INDX).

import struct

# Unpacking an NTFS MFT Record Header (First 16 bytes)
# Format: 4s (Signature 'FILE'), H (Offset to Fixup), H (Entries in Fixup), Q ($LogFile Sequence Number)
mft_header_sample = bytes.fromhex("46494c45 3000 0300 123456789abcdef0")
magic, fixup_offset, fixup_count, lsn = struct.unpack('<4sHHQ', mft_header_sample)

print(f"Signature: {magic.decode('ascii')} | Fixup Offset: {fixup_offset} | LSN: {hex(lsn)}")

4. datetime: Timestamp Transformations

Forensic artifacts store temporal metadata across incompatible epoch systems. Converting these timestamps into standardized UTC ISO-8601 strings is essential for constructing master event timelines.

The Windows FILETIME Epoch (NTFS / Registry / Prefetch)

• A 64-bit unsigned integer representing the number of 100-nanosecond intervals that have elapsed since January 1, 1601, at 00:00:00 UTC.
• Conversion Logic: Divide the FILETIME integer by 10,000,000 to convert 100-nanosecond intervals into seconds, and subtract the difference between the 1601 epoch and the 1970 Unix epoch (11,644,473,600 seconds).

import datetime

def filetime_to_datetime(filetime_int):
    """Converts a 64-bit Windows FILETIME integer into a timezone-aware UTC datetime object."""
    if filetime_int == 0:
        return None
    # 100-nanoseconds to microseconds is // 10
    return datetime.datetime(1601, 1, 1, tzinfo=datetime.timezone.utc) + datetime.timedelta(microseconds=filetime_int // 10)

# Example: 133400000000000000 represents a Windows FILETIME
print("Converted FILETIME:", filetime_to_datetime(133400000000000000))

Unix Epoch vs. Apple Epochs

• Unix Epoch: 32-bit or 64-bit integer representing seconds since January 1, 1970 UTC (datetime.datetime.fromtimestamp(ts, tz=datetime.timezone.utc)).
• Apple HFS+ Epoch: Seconds elapsed since January 1, 1904 UTC.
• Apple Mac Absolute Time (Cocoa / WebKit): Seconds (or floating-point seconds) elapsed since January 1, 2001 UTC.

---

Portable Executable Analysis with pefile & ImpHash

In incident response, analyzing suspicious Windows executables (.exe, .dll, .sys) requires parsing the Portable Executable (PE) structure. The third-party Python library pefile parses these internal data directories and headers.

+-------------------------------------------------------------+
|                      DOS Header (e_magic = 'MZ')            |
+-------------------------------------------------------------+
|                      DOS Stub Program                       |
+-------------------------------------------------------------+
|                      PE Header (Signature = 0x4550 'PE')    |
|  - Machine Architecture, Number of Sections, TimeDateStamp  |
+-------------------------------------------------------------+
|                      Optional Header                        |
|  - AddressOfEntryPoint, ImageBase, Subsystem, DataDirectory |
+-------------------------------------------------------------+
|                      Section Headers                        |
|  - .text (Code), .data (Globals), .rsrc (Resources)         |
|  - VirtualAddress, VirtualSize, PointerToRawData, Entropy   |
+-------------------------------------------------------------+
|                      Import Address Table (IAT)             |
|  - KERNEL32.dll -> CreateProcessA, VirtualAlloc, WriteProc..|
+-------------------------------------------------------------+

Critical PE Forensic Artifacts

1. Compilation Timestamp (TimeDateStamp): Located in the COFF File Header. Indicates when the compiler built the binary. However, this value can be altered by attackers using timestomping utilities.
2. Section Entropy (Shannon Entropy): Measures the randomness of data within each section on a scale from 0.0 to 8.0.
   • Standard compiled code (.text) typically exhibits entropy between 5.5 and 6.8.
   • Sections with entropy exceeding 7.0 indicate heavily packed, obfuscated, or encrypted payloads (such as UPX, VMProtect, or embedded shellcode).
3. Import Address Table (IAT): Lists the dynamic-link libraries (DLLs) and API functions requested by the executable. Identifying suspicious imports (e.g., VirtualAllocEx, WriteProcessMemory, CreateRemoteThread) provides immediate indicators of process injection capabilities.
4. Import Hash (ImpHash):
   • Developed by Mandiant, the ImpHash is an MD5 hash computed across the executable's imported functions and their parent DLLs in normalized order.
   • Forensic Significance: When malware authors modify a payload, recompile code, or change variable strings, the cryptographic hash of the file (MD5/SHA-256) changes entirely. However, unless the author alters the imported API functions or their link order, the ImpHash remains identical. This enables forensic investigators to cluster and attribute related malware variants across global threat campaigns.

import pefile

def analyze_portable_executable(pe_path):
    pe = pefile.PE(pe_path)
    
    print(f"PE Analysis: {pe_path}")
    print(f"Compilation Timestamp (Epoch): {pe.FILE_HEADER.TimeDateStamp}")
    print(f"Calculated ImpHash: {pe.get_imphash()}")
    
    print("\n--- Section Entropy Analysis ---")
    for section in pe.sections:
        name = section.Name.decode('utf-8', errors='ignore').strip(chr(0))
        entropy = section.get_entropy()
        print(f"Section: {name:<8} | Size: {section.SizeOfRawData:<8} | Entropy: {entropy:.2f}")
        if entropy > 7.0:
            print(f"  [!] WARNING: High entropy detected in {name}. Possible packed or encrypted payload.")
            
    print("\n--- Suspicious API Imports ---")
    if hasattr(pe, 'DIRECTORY_ENTRY_IMPORT'):
        for entry in pe.DIRECTORY_ENTRY_IMPORT:
            dll_name = entry.dll.decode('utf-8', errors='ignore')
            for imp in entry.imports:
                if imp.name:
                    func_name = imp.name.decode('utf-8', errors='ignore')
                    if func_name in ['VirtualAllocEx', 'WriteProcessMemory', 'CreateRemoteThread', 'QueueUserAPC']:
                        print(f"  [ALERT] Code injection API identified: {dll_name}!{func_name}")

---

Network Forensics & Protocol Dissection with Scapy

Scapy is a Python library used for network packet manipulation, forging, and dissection. In digital forensics, Scapy is applied defensively to dissect Packet Capture (PCAP) files, reconstruct network communications, and extract indicators of compromise (IoCs).

Handling Large PCAPs: rdpcap vs. PcapReader

• rdpcap("traffic.pcap"): Reads the entire packet capture into system memory at once. If an investigator attempts to open a 10 GB PCAP using rdpcap, the host system will exhaust RAM and crash.
• PcapReader("traffic.pcap"): Operates as an iterator / streaming generator, reading packets one at a time from disk. This allows scripts to process multi-gigabyte PCAP files with minimal memory overhead.

Automated DNS Query Extraction and Tunneling Detection

Attackers often use DNS tunneling for data exfiltration and command-and-control communications. Because recursive DNS queries are routed through internal corporate resolvers to reach external authoritative nameservers, DNS bypasses standard firewall egress rules. Python scripts utilizing Scapy can parse PCAP files, extract DNS queries, and identify suspicious high-entropy subdomains:

from scapy.all import PcapReader, DNS, DNSQR
import math

def calculate_entropy(data_str):
    """Calculates Shannon entropy of a string to identify base32/hex encoded exfiltration."""
    prob = [float(data_str.count(c)) / len(data_str) for c in dict.fromkeys(list(data_str))]
    return -sum([p * math.log(p) / math.log(2.0) for p in prob])

def extract_dns_indicators(pcap_path):
    suspicious_queries = set()
    
    with PcapReader(pcap_path) as reader:
        for packet in reader:
            # Verify packet contains a DNS layer with a Query Record
            if packet.haslayer(DNS) and packet.haslayer(DNSQR):
                query_name = packet[DNSQR].qname.decode('utf-8', errors='ignore')
                # Remove trailing root dot
                query_name = query_name.rstrip('.')
                
                # Inspect the subdomain label for data exfiltration signatures
                subdomain = query_name.split('.')[0]
                if len(subdomain) > 25:
                    entropy = calculate_entropy(subdomain)
                    if entropy > 3.8:
                        suspicious_queries.add((query_name, entropy))
                        
    for domain, ent in suspicious_queries:
        print(f"[DNS TUNNEL SUSPECT] Domain: {domain} (Entropy: {ent:.2f})")

---

Offline Registry Parsing with python-registry

During live forensic acquisitions, Windows locks registry hives, preventing standard file access. In offline dead-disk acquisitions, investigators extract hive files (SYSTEM, SOFTWARE, SAM, NTUSER.DAT, UsrClass.dat) from the disk image.

The python-registry library allows cross-platform, offline navigation of Windows Registry structures without requiring native Windows APIs.

Extracting Persistence and Execution Records

1. Run and RunOnce Keys: Located in SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Run and user NTUSER.DAT hives. Used by adversaries to establish persistence across user logons.
2. UserAssist Artifacts: Located in NTUSER.DAT\\Software\\Microsoft\\Windows\\CurrentVersion\\Explorer\\UserAssist\\{GUID}\\Count.
   • Records GUI-based program executions launched via Windows Explorer.
   • Value names are obfuscated using the ROT13 cipher.
   • The binary data buffer contains execution counts, focus times, and the last execution timestamp stored as an 8-byte Windows FILETIME.

import codecs
import struct
from Registry import Registry

def parse_userassist_keys(ntuser_path):
    reg = Registry.Registry(ntuser_path)
    userassist_path = r"Software\Microsoft\Windows\CurrentVersion\Explorer\UserAssist"
    
    try:
        ua_key = reg.open(userassist_path)
    except Registry.RegistryKeyNotFoundException:
        print("[!] UserAssist key not found in hive.")
        return
        
    for guid_subkey in ua_key.subkeys():
        try:
            count_key = guid_subkey.subkey("Count")
            for val in count_key.values():
                # Decode ROT13 obfuscated program path
                raw_name = val.name()
                program_name = codecs.decode(raw_name, 'rot_13')
                raw_data = val.value()
                
                # Windows 7 through 11: 72-byte buffer; Execution count at byte offset 4
                if len(raw_data) >= 72:
                    exec_count = struct.unpack('<I', raw_data[4:8])[0]
                    filetime_raw = struct.unpack('<Q', raw_data[60:68])[0]
                    print(f"Application: {program_name} | Run Count: {exec_count} | Last Run: {filetime_raw}")
        except Registry.RegistryKeyNotFoundException:
            continue

[!IMPORTANT] When presenting evidence derived from custom Python scripts in legal proceedings, examiners must preserve the script source code, document the exact interpreter version, list all external library dependencies (via pip freeze), and validate parsing accuracy against known test datasets.

---

Email and Multimedia Forensics in Python

The blueprint lists email forensics using Python and multimedia forensics using Python alongside the Windows, Linux, malware, cloud, and mobile targets.

Email Header and Container Parsing

Python's standard library parses RFC 5322 natively, which makes header analysis scriptable across a whole mailbox rather than message by message.

import email, mailbox
from email import policy

def trace_hops(raw_bytes):
    msg = email.message_from_bytes(raw_bytes, policy=policy.default)
    # Received headers are PREPENDED by each receiving MTA -> reverse for chronological order
    hops = list(reversed(msg.get_all('Received', [])))
    return {
        'from': msg['From'],
        'reply_to': msg['Reply-To'],          # divergence from From is a phishing indicator
        'return_path': msg['Return-Path'],
        'message_id': msg['Message-ID'],
        'auth_results': msg.get_all('Authentication-Results', []),  # SPF / DKIM / DMARC verdicts
        'hops': hops,
        'attachments': [p.get_filename() for p in msg.walk() if p.get_filename()],
    }

for message in mailbox.mbox('/evidence/inbox.mbox'):
    print(trace_hops(message.as_bytes()))

Scripted parsing scales the Section 13.3 methodology: run the bottom-up hop trace across ten thousand messages, then cluster on originating IP, Reply-To divergence, and DMARC failure to isolate a campaign.

Multimedia Triage

The image, video, and audio techniques are covered in Section 18.3; the scripted form is what makes them usable on a seized corpus.

from PIL import Image
from PIL.ExifTags import TAGS
import imagehash, hashlib, pathlib

def media_triage(folder):
    for p in pathlib.Path(folder).rglob('*.jp*g'):
        img = Image.open(p)
        exif = {TAGS.get(k, k): v for k, v in (img._getexif() or {}).items()}
        yield {
            'path': str(p),
            'sha256': hashlib.sha256(p.read_bytes()).hexdigest(),
            'phash': str(imagehash.phash(img)),   # groups re-saved / resized duplicates
            'model': exif.get('Model'),
            'software': exif.get('Software'),     # names any editor that touched the file
            'captured': exif.get('DateTimeOriginal'),
            'has_gps': 'GPSInfo' in exif,
        }

pyexiftool extends coverage to maker notes and XMP, ffmpeg-python and pymediainfo read video container atoms and extract frames, and librosa supports audio spectral work. Perceptual hashing is the key scaling primitive: a cryptographic hash cannot group two saves of the same photograph, while a perceptual hash clusters them, which is how a corpus of hundreds of thousands of images is reduced to a reviewable set.