11.3 Malware Analysis Techniques for SOC Responders

The Five Stages of Malware Analysis

In a Security Operations Center, malware analysis is not a monolithic activity; it is a structured, risk-managed discipline structured into progressive tiers of analytical depth. As an investigation escalates from initial alert validation to in-depth threat intelligence profiling, the analyst applies increasingly complex and time-intensive techniques.

Malware Analysis Maturity Pyramid:
           [Stage 5: Advanced Dynamic] ── Manual Kernel Debugging (WinDbg, x64dbg)
         [Stage 4: Advanced Static]  ── Disassembly & Decompilation (Ghidra, IDA Pro)
       [Stage 3: Dynamic Analysis]   ── Behavioral Sandboxing (Procmon, Regshot, INetSim)
     [Stage 2: Static Analysis]    ── PE Headers, Strings, Imports, Hashing, Entropy
   [Stage 1: Automated Analysis]   ── VirusTotal, Hybrid-Analysis, Any.Run, Joe Sandbox

The Five Analysis Stages

1. Stage 1: Automated Analysis: Rapid, automated submission of suspicious binaries to online or on-premises sandbox appliances (such as VirusTotal, Hybrid-Analysis, Any.Run, or Joe Sandbox). The system executes the binary automatically and generates a high-level report containing multi-antivirus detection ratios, observed network connections, dropped files, and MITRE ATT&CK technique tags. Automated analysis provides immediate triage within minutes, though sophisticated malware may detect the automated hypervisor environment and suppress its malicious behavior.
2. Stage 2: Static Analysis (Static Triage): Examining the file's properties, headers, metadata, strings, and structure without executing any code. Static triage identifies file hashes, confirms true file formats via magic bytes, extracts human-readable text strings, analyzes Portable Executable (PE) headers, inspects imported Windows API functions, and calculates entropy to detect packing. Static analysis reduces execution risk because the analyst does not intentionally run the binary, but malformed files, vulnerable parsers, accidental launches, and active content still justify an isolated analysis environment.
3. Stage 3: Dynamic Analysis (Behavioral Analysis): Executing the malware specimen inside a tightly controlled, instrumented, and isolated virtual machine sandbox. The analyst monitors the real-time behavioral impact of the binary on the operating system: newly spawned child processes, injected threads, modified registry autostart keys, dropped filesystem payloads, and generated network socket traffic.
4. Stage 4: Advanced Static Analysis (Reverse Engineering): Loading the binary into disassemblers and decompilers (such as NSA Ghidra or IDA Pro) to convert machine bytecode into human-readable assembly instructions (x86/x64) or pseudo-C code. The reverse engineer analyzes cryptographic routines, proprietary network protocol encoding, internal algorithm logic, and dormant backdoor capabilities that never trigger during dynamic sandbox execution.
5. Stage 5: Advanced Dynamic Analysis (Debugging): Attaching a user-mode or kernel-mode debugger (such as x64dbg, x32dbg, or WinDbg) to the running malware process. The analyst sets memory breakpoints, steps through execution instructions single-cycle at a time, manipulates register values, bypasses anti-debugging checks, and captures unpacked code directly from memory buffers as the malware self-decrypts.

Malware Analysis Stages Comparison Table

---

Static Analysis Tools and Techniques

Static triage provides immediate insight into a specimen's intended functionality while preserving absolute host safety. Analysts follow a structured static analysis checklist.

Cryptographic Hashing and Fuzzy Matching

• MD5 and SHA-256: Generates unique cryptographic fingerprints for exact-match IoC queries across SIEM, EDR, and threat intelligence platforms.
• SSDEEP (Context Triggered Piecewise Hashing): Unlike standard cryptographic hashes where altering a single bit changes the entire hash, SSDEEP generates a fuzzy hash based on rolling byte sequences. It computes percentage similarity between two files, allowing analysts to instantly recognize when a new sample is a minor recompile or variant of a known malware family.
• Imphash (Import Hash): Calculates an MD5 hash of the imported functions and their specific sequence within the binary's Import Address Table (IAT). Malware developers frequently modify strings, icons, and variable names to evade file hash detection, but reuse standard software frameworks and import structures. Binaries exhibiting identical Imphash values almost certainly belong to the same threat group or development pipeline.

File Type Identification & Magic Numbers

Adversaries deliberately disguise executable files by appending benign file extensions (e.g., naming a malicious PE executable invoice.pdf or photo.jpg). Operating systems rely on file extensions, but file parsers rely on magic numbers—specific hexadecimal byte sequences located at offset zero (the very start) of the file.

Strings Extraction: strings vs. FLOSS

Extracting human-readable character sequences embedded within a binary reveals IP addresses, URLs, command-line arguments, registry keys, error messages, and file paths. However, standard utilities like GNU strings only extract plaintext ASCII and 16-bit Unicode sequences.

Threat actors routinely obfuscate strings using XOR loops, stack strings (constructed byte-by-byte in CPU registers right before use), or custom encoding to defeat standard strings extraction. To solve this, Mandiant developed FLOSS (FireEye Labs Obfuscated String Solver). FLOSS uses advanced static heuristics and automated emulation to detect decoding tight-loops, deobfuscate stack strings, and extract concealed ASCII/Unicode strings automatically without requiring manual reverse engineering.

Portable Executable (PE) Header Analysis

Windows executables conform to the PE format, structured into headers and sections:

• .text: Contains the executable machine code instructions executed by the CPU. Configured with Read and Execute (RX) permissions.
• .data: Contains initialized global and static variables. Configured with Read and Write (RW) permissions.
• .rdata: Contains read-only data, string literals, and critical tables: the Import Table (APIs the binary imports from the OS) and the Export Table (APIs the binary exposes to external callers).
• .rsrc: Contains system resources: icons, dialog boxes, version information, language manifests, and frequently, hidden embedded secondary binaries or scripts.

Suspicious Windows API Imports Matrix

By reviewing the Import Address Table using tools like PEview, PE-bear, or CFF Explorer, an analyst infers the binary's underlying architectural intent.

Packing Detection and Shannon Entropy

To defeat antivirus scanners and static analysis, malware authors utilize packers (such as UPX, Themida, ASPack, or VMProtect). A packer compresses or encrypts the original executable and bundles it with a lightweight decompression stub. When executed, the stub unpacks the payload directly into memory and transfers CPU execution to the unpacked entry point.

Analysts detect packed binaries using two key indicators:

1. Import Address Table Scarcity: A standard Windows program imports dozens or hundreds of functions across multiple DLLs. A packed binary typically imports only two to four functions—specifically LoadLibraryA and GetProcAddress from kernel32.dll—which the stub needs to dynamically resolve imports after unpacking.
2. Shannon Entropy Analysis: In information theory, entropy measures the degree of randomness in a dataset on a scale from 0.0 (completely uniform) to 8.0 (completely random, maximum uncertainty). Plaintext source code and standard compiled machine instructions typically measure between 5.5 and 6.8. Compressed or strongly encrypted data exhibits an entropy score approaching 8.0. A very high entropy score in an executable section—especially alongside a sparse import table—is a packing, compression, or encryption indicator, not definitive proof. Legitimate compressed data and specialized binaries can also have high entropy.

---

Dynamic Analysis Tools and Techniques

Dynamic analysis reveals the true behavioral actions of a specimen as it executes in real time.

Sandbox Architecture and INetSim Network Simulation

A dynamic analysis sandbox should use layered isolation and controlled simulation to reduce the risk that malware reaches production networks or the public internet. No lab design makes escape risk literally zero, so teams also use access controls, monitoring, disposable hosts, validated snapshots, and documented procedures.

• Host-Only Virtual Network: The guest analysis virtual machine is attached to an internal, host-only virtual switch with no default gateway leading to the physical network card.
• INetSim (Internet Network Services Simulation): Because many malware specimens immediately terminate or refuse to unpack if internet connectivity is missing, analysts pair the guest detonation machine with a secondary Linux VM hosting INetSim. INetSim emulates standard internet application protocols: DNS, HTTP, HTTPS, SMTP, FTP, NTP, TFTP, and IRC. When the malware attempts to resolve a C2 domain (e.g., evil-c2.com), INetSim's DNS service resolves the query to its own local IP. When the malware issues an HTTP GET request to download a secondary payload, INetSim returns an HTTP 200 OK response with a benign dummy executable. This convinces the malware that it is online, prompting it to proceed with subsequent operational stages while safely trapping all network telemetry locally.

Endpoint Behavioral Monitoring Tools

• Sysinternals Process Monitor (Procmon): Captures real-time operating system calls across four core subsystems: Process/Thread activity, File System activity, Registry activity, and Network connections. Because Procmon captures tens of thousands of benign events per minute, analysts apply strict display filters: Process Name is malware.exe, Operation is RegSetValue, or Operation is CreateFile.
• Sysinternals Process Explorer: Provides an interactive view of running processes, parent-child lineages, loaded DLL modules, and operating system handles. It highlights newly spawned processes in green and terminating processes in red, enabling rapid visual identification of process hollowing.
• Regshot: An open-source utility that takes an initial snapshot (Shot 1) of the entire Windows Registry before malware execution, and a second snapshot (Shot 2) post-execution. Regshot computes a text-based differential report listing every key added, deleted, or modified, immediately pinpointing persistence mechanisms and system configuration tampering.
• Wireshark: Running on the host network interface, Wireshark captures all raw socket communications between the detonation VM and INetSim, recording full HTTP headers, user-agent strings, and raw payload requests.

---

Safe Handling, Defanging, and Sample Storage

Handling live malware carries inherent operational risk. SOC procedures enforce strict containment controls to prevent accidental detonation:

• Defanging Observables: Before sharing malicious domains, URLs, or IP addresses in tickets, emails, or reports, analysts must defang them to prevent recipients or email clients from accidentally hyperlinking or triggering network calls:
  • Replace http:// with hxxp:// and https:// with hxxps://
  • Bracket dots in domains and IPs: malicious[.]com, 198[.]51[.]100[.]24
  • Example: http://bad.org/worm.exe becomes hxxp://bad[.]org/worm[.]exe
• Protected Sample Handling: Do not place live malware unencrypted on ordinary shared drives or send it through standard messaging channels. Security controls may quarantine it, but the larger risks are accidental execution, unauthorized access, uncontrolled replication, and policy violation. When policy permits transferring a live specimen, store it in an approved encrypted container with least-privilege access and communicate the password out of band. Values such as infected are common conventions, not universal requirements; follow the receiving laboratory’s procedure. File extensions should be altered (e.g., renaming malware.exe to malware.bin or malware.sample) to prevent accidental execution by double-clicking.