4.2 Malware Analysis and Detection

Static Analysis: Examine Without Executing

CEH frames malware analysis as a defensive skill: understanding a sample so you can detect, contain, and hunt it. The two pillars are static and dynamic analysis.

Static analysis inspects the file without running it, which is safe but can be defeated by obfuscation. Core static techniques the exam expects:

• File fingerprinting — compute a cryptographic hash (MD5, SHA-1, SHA-256) to uniquely identify the sample and look it up on threat-intel services like VirusTotal.
• Strings analysis — extract printable strings to reveal URLs, IP addresses, registry keys, error messages, or commands embedded in the binary.
• PE header / import analysis — examine the Portable Executable header and the imported APIs (Import Address Table); calls to CreateRemoteThread, URLDownloadToFile, or crypto APIs hint at intent. Tools: PEiD, PEview, Dependency Walker.
• Packer detection — identify whether the binary is packed/encrypted (e.g., UPX). A packed file shows few imports and high entropy, signaling the real code is hidden until runtime.
• Disassembly — convert machine code to assembly with IDA Pro, Ghidra, or a debugger to study logic.

Static analysis is fast and low-risk, but packers, crypters, and polymorphism mutate the binary so its hash and signature change every infection, blunting static detection. That limitation is exactly why dynamic analysis exists.

Dynamic Analysis, Sandboxing, and IoCs

Dynamic (behavioral) analysis runs the sample in a controlled, isolated environment and watches what it actually does: files written, processes spawned, registry keys set, and — critically — network connections to command-and-control servers. Key monitoring tools CEH references include Process Monitor and Process Explorer (Sysinternals) for process/registry/file activity, Regshot for before/after registry diffs, Wireshark for captured C2 traffic, and automated sandboxes (Cuckoo, Any.Run, Joe Sandbox).

The sandbox must be isolated from production (no route to the real network, snapshot/rollback enabled) because executing live malware is dangerous. Sophisticated malware practices sandbox evasion: it checks for VM artifacts (MAC address ranges, VMware tools, low CPU/RAM), looks for debuggers (anti-debugging), or simply sleeps for minutes to outlast automated analysis. A behaving-too-nicely sample may simply be detecting your lab.

The output of analysis is a set of Indicators of Compromise (IoCs) — concrete artifacts you feed into detection systems to find the threat elsewhere:

IoCs are often shared in STIX/TAXII format and matched in tools like YARA rules. Behavioral IoCs map to attacker techniques in the MITRE ATT&CK framework, which CEH increasingly references.

Detection Methods and Defense in Depth

Anti-malware uses several detection strategies, each with a blind spot:

• Signature-based — matches files against a database of known-malware hashes/byte patterns. Fast and accurate for known threats, but blind to new, packed, polymorphic, and fileless malware.
• Heuristic — flags suspicious code constructs or behavior patterns that resemble malware, catching some unknown variants at the cost of false positives.
• Behavioral / EDR — monitors runtime actions (process injection, mass file encryption, C2 beaconing) and is the only reliable way to catch fileless attacks. Endpoint Detection and Response (EDR) records and correlates this telemetry.
• Sandbox detonation — runs unknown attachments/files in a sandbox before delivery.

No single control stops everything, so CEH emphasizes defense in depth for malware:

1. Patch management to close the exploits worms and droppers use.
2. Application allowlisting so only approved binaries run (blocks unknown executables outright).
3. Least privilege so a compromise can't trivially install kernel rootkits.
4. Email/web filtering and macro disabling to stop the most common delivery path.
5. Network segmentation and egress filtering to break lateral movement and C2 callbacks.
6. Tested, offline (immutable) backups — the definitive ransomware countermeasure, because paying is never guaranteed to restore data.
7. User awareness training, since most malware arrives through a person.

On the exam, when a question asks for the best defense against ransomware specifically, tested offline backups outrank antivirus; when it asks about fileless malware, behavioral/EDR detection beats signatures.

The Triage Workflow and Threat Hunting

A practical malware-analysis workflow ties the techniques together in order:

1. Isolate the sample and the infected host (pull from the network, snapshot for evidence).
2. Fingerprint — hash the file (SHA-256) and check threat-intel/VirusTotal for prior sightings.
3. Static triage — strings, PE header, imports, and packer detection to form a hypothesis cheaply.
4. Dynamic detonation — run it in an isolated sandbox while capturing process, file, registry, and network activity.
5. Extract IoCs — record hashes, C2 domains/IPs, dropped files, registry keys, mutexes, and scheduled tasks.
6. Hunt and remediate — push IoCs and YARA rules across the fleet to find other infected hosts, then rebuild, patch, and rotate credentials.

Threat hunting uses the resulting IoCs and ATT&CK techniques to proactively search telemetry for the same adversary elsewhere — for example, searching EDR logs for the C2 domain or for Office spawning PowerShell. CEH frames anti-malware controls as layers in a broader program, not a silver bullet.

Common evasion the analyst must anticipate:

The takeaway for the exam: a single control fails to a determined adversary, so detection must combine signatures (known threats) + heuristics (suspicious constructs) + behavioral/EDR (runtime actions) + sandboxing (pre-delivery), all backed by patching, least privilege, allowlisting, segmentation, and tested backups. When a scenario describes malware that changed its appearance to slip past antivirus, the analyst should pivot from static signatures to behavioral analysis and IoC-driven hunting.