5.3 Evading IDS and IPS (Defensive Focus)

IDS vs. IPS and Detection Methods

An Intrusion Detection System (IDS) monitors traffic and raises alerts when it observes malicious activity, but it sits passively out-of-band (often on a SPAN/mirror port) and does not stop the packet. An Intrusion Prevention System (IPS) sits inline in the traffic path and can actively drop, reset, or quarantine offending flows. IDS/IPS are further classified by placement: a NIDS/NIPS is network-based (monitors a segment), while a HIDS is host-based (watches a single system's logs, files, and processes).

Detection methods CEH tests:

Snort, Suricata, and Zeek are common network sensors; Snort is the canonical CEH example and uses a rule language (alert tcp any any -> $HOME_NET 80 ...).

Insertion and Evasion: The Core Concept

Ptacek and Newsham's foundational work defined two complementary attacks on a sensor:

• Insertion — the attacker sends a packet the IDS accepts but the target rejects. The sensor's reconstructed stream now contains extra bytes the victim never processes, so a signature can be "diluted" or broken (e.g., inserting characters with a bad checksum or low TTL that die before reaching the target).
• Evasion — the attacker sends a packet the IDS rejects but the target accepts. The sensor misses bytes that the victim does process, so the malicious payload never matches a signature.

Both attacks exploit the same root cause: the IDS and the end host disagree on how to interpret/reassemble the same packet stream. Whenever an IDS makes a different decision than the actual target about TTL, fragment overlap, checksum validity, or segmentation, an attacker can hide an exploit in the gap. This is why modern defenders favor target-based reassembly — the sensor models how the specific destination OS reassembles traffic — rather than a generic guess.

Fragmentation, TTL, and Encoding Techniques

Fragmentation attacks split the payload across many small IP fragments or TCP segments so no single packet matches a signature; only after reassembly is the attack visible, and if the IDS reassembles differently from the host, it misses the match. Variants include tiny fragments, overlapping fragments (offsets crafted so different OSes keep older vs. newer data), and out-of-order delivery. Nmap's -f flag and fragroute automate fragmentation.

TTL manipulation sets a Time-To-Live just high enough to reach the IDS but expire before the target (insertion) or vice-versa, desynchronizing the two views. Bad checksums are another insertion trick: the IDS may process a packet that the host silently drops.

Encoding and obfuscation defeat signature matching by transforming the payload into a form the target decodes but the IDS does not:

• URL/hex/Unicode encoding of HTTP requests (e.g., %2e%2e%2f for ../).
• Polymorphic shellcode — each instance is XOR-encoded with a unique key plus a small decoder stub, so the byte pattern is never the same.
• Case manipulation, double-encoding, and tabs/whitespace in protocol commands.
• Encryption/tunneling (HTTPS, SSH) so the IDS sees only ciphertext.

Defensive Countermeasures

As CEH frames it, understanding evasion is what lets a defender close the gap. Countermeasures:

Additional defenses: deploy the IPS inline so it can drop rather than merely alert; tune signatures to reduce false positives (alert fatigue is itself an evasion enabler); place sensors at multiple choke points; and feed alerts into a SIEM for correlation. In Snort, defenders enable preprocessors such as frag3 (target-based IP defragmentation) and stream5 (stateful TCP reassembly) precisely to defeat fragmentation and insertion/evasion.

A further evasion class is denial-of-service against the sensor itself: flooding the IDS with crafted alerts (false positives) or with more traffic than it can inspect at line rate causes fail-open behavior, where packets pass uninspected. Defenders size sensors for peak throughput, deploy them fail-closed on critical segments, and watch for inspection-drop counters. Attackers also try session splicing — delivering the payload one byte per packet with long delays so a sensor with a short reassembly timeout discards state before the full signature forms; the fix is long-lived stateful reassembly windows.

A final point: because signature-based sensors only catch known patterns, defenders pair them with anomaly-based detection and threat-intelligence feeds, and they regularly update rule sets so newly disclosed exploits are recognized. The key exam takeaway: evasion works by desynchronizing the sensor from the target (or overwhelming it), so normalization, target-aware reassembly, updated signatures, and adequate capacity are the structural fixes.