3.3 Intrusion Detection & Prevention Systems (IDS/IPS)
Key Takeaways
- Intrusion Detection Systems (IDS) operate out-of-band (OOB) on mirrored network traffic copies, generating alerts without introducing network latency, but cannot prevent initial exploit packets from reaching targets.
- Intrusion Prevention Systems (IPS) sit inline (in-band) directly in physical network paths, inspecting traffic in real time to actively drop malicious packets before reaching destinations.
- Signature-based detection matches payloads against known attack patterns (e.g., Cisco Snort rules) with high accuracy, but remains blind to zero-day vulnerabilities.
- Anomaly-based (behavioral) detection establishes statistical baselines of normal network activity to detect novel zero-day exploits, but produces higher false positive rates.
- Hardware Network TAPs guarantee 100% passive frame capture without switch CPU consumption, outperforming software SPAN/RSPAN port mirroring during high-volume traffic bursts.
3.3 Intrusion Detection & Prevention Systems (IDS/IPS)
Quick Summary: Intrusion Detection Systems (IDS) passively monitor network traffic copies out-of-band to generate alerts without adding network latency. Intrusion Prevention Systems (IPS) sit inline directly in the physical traffic path to actively block malicious traffic in real time. Detection engines rely on Signature-based rules for known threats and Anomaly/Behavioral-based baselines for zero-day detection.
While firewalls dictate which IP addresses and port numbers are permitted to cross network boundaries, they generally permit all traffic matching open firewall ports (e.g., TCP port 443). Intrusion Detection and Prevention Systems inspect packet payloads deeply to identify and neutralize malicious content—such as exploit payloads, SQL injections, buffer overflows, and malware command-and-control beacons—that enter over legitimate open ports.
1. IDS vs. IPS Architecture & Deployment Models
The fundamental distinction between an IDS and an IPS lies in their physical deployment architecture and their operational capability to respond to active security threats.
Intrusion Detection Systems (IDS) - Out-of-Band
An IDS is deployed Out-of-Band (OOB). It does not sit directly in the primary physical traffic path. Instead, a network switch or physical hardware TAP sends a duplicated copy of traffic to the IDS sensor monitoring interface.
- Operational Response: When an IDS detects an attack signature in a mirrored frame, it cannot stop the original frame because that frame has already traversed the switch to the victim. To mitigate ongoing attacks, the IDS sends alerts to a SIEM dashboard and can inject spoofed TCP Reset (TCP RST) packets to both source and destination hosts to tear down the connection.
- Advantages: Zero impact on network latency; network performance is completely unaffected if the IDS sensor fails or experiences processing overload.
- Disadvantages: Cannot prevent single-packet exploits or the initial attack packet of a connection stream.
Intrusion Prevention Systems (IPS) - Inline
An IPS is deployed Inline (In-Band). All physical network traffic must enter one physical interface of the IPS sensor and exit out another interface before reaching the destination network.
- Operational Response: The IPS inspects packets in real time before forwarding them. If an attack signature is matched, the IPS executes inline remediation actions immediately:
drop-packet(silently discards the packet),block-source(temporarily blocks all traffic from the attacker's IP), orreset-connection. - Advantages: Prevents malicious packets from reaching victim endpoints; actively stops zero-day exploit execution.
- Disadvantages: Adds minor processing latency to every packet; if the IPS hardware fails, it creates a network outage unless equipped with hardware fail-open (bypass) NICs.
2. Detection Engine Methodologies, Snort Rules & Accuracy Matrix
IDS/IPS appliances analyze traffic streams using complementary detection engines:
1. Signature-Based Detection & Snort Rules
Compares network traffic payloads against a database of pre-defined attack signatures (such as Cisco Snort rules).
- Snort Rule Anatomy: A Snort rule consists of a Rule Header and Rule Options:
alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS 80 (msg:"SQL Injection - UNION SELECT"; content:"UNION SELECT"; nocase; sid:1000001; rev:1;) - Strengths: Highly accurate; exceptionally low False Positive rate for known attacks; low CPU overhead.
- Weaknesses: Completely blind to Zero-Day vulnerabilities, obfuscated scripts, and polymorphic malware.
2. Anomaly-Based (Behavioral) Detection
Establishes a statistical baseline of normal network activity over a training period (typical bandwidth usage, connection rates, protocol distribution, login hours).
- Operational Logic: Flags traffic that deviates significantly from baseline norms (e.g., a workstation initiating 5,000 outbound UDP connections per minute).
- Strengths: Capable of detecting zero-day attacks, novel malware variants, and insider data exfiltration.
- Weaknesses: Prone to high False Positive rates when legitimate business network behavior shifts.
3. Heuristic & Protocol Analysis
Inspects traffic against RFC protocol specifications, flagging protocol violations and abnormal packet formatting (e.g., Xmas tree scan with FIN, URG, and PSH flags set simultaneously).
Detection Accuracy Decision Matrix
Security operations teams evaluate IDS/IPS effectiveness using a four-quadrant truth matrix:
| Operational Decision | Attack Actually Present | Traffic Benign / Normal |
|---|---|---|
| Alert / Block Executed | True Positive (TP) - Correct Security Action | False Positive (FP) - Operational Error (Causes Alert Fatigue) |
| No Action Executed | False Negative (FN) - CRITICAL FAILURE (Undetected Breach) | True Negative (TN) - Correct Operational State |
3. Host-Based vs Network-Based Systems & Traffic Acquisition
Complete infrastructure protection requires coordinating host-based and network-based sensor deployments.
NIDS/NIPS vs. HIDS/HIPS
- Network-Based IDS/IPS (NIDS/NIPS): Deployed at key network chokepoints (e.g., perimeter switch or DMZ gateway). Inspects all unencrypted traffic traversing that network segment. Cannot inspect host-internal process memory or encrypted TLS payloads unless paired with SSL decryption proxies.
- Host-Based IDS/IPS (HIDS/HIPS): Installed directly on individual servers or workstations (e.g., OSSEC, Wazuh, Tripwire). Monitors host system log files, critical registry keys, system calls, memory allocations, and File Integrity Monitoring (FIM) hashes. Can inspect encrypted payloads after local OS decryption.
Traffic Acquisition Mechanics: SPAN vs. RSPAN vs. Network TAP
For a NIDS sensor to inspect traffic, it must receive clean copies of network frames:
- SPAN (Switch Port Analyzer): Software feature on switches that mirrors traffic from source ports/VLANs to a destination monitor port. Low cost, but drops frames during high switch CPU load or port oversubscription.
- RSPAN (Remote SPAN): Mirrors traffic across a dedicated RSPAN VLAN across multiple campus switches to a centralized collector.
- Network TAP (Test Access Point): Hardware device spliced directly into physical copper or fiber cabling. Guarantees 100% frame capture with zero switch CPU impact and full passive fault tolerance.
What is the primary operational trade-off when deploying an out-of-band Intrusion Detection System (IDS) instead of an inline Intrusion Prevention System (IPS)?
Which detection engine methodology establishes a statistical baseline of normal network activity to identify previously unknown 'Zero-Day' attack vectors?
Which classification in the IDS detection matrix represents the most critical failure scenario, where an active malicious attack occurs but the system takes no action and issues no alert?
Which traffic capture mechanism uses a dedicated hardware device spliced directly into physical cabling to guarantee 100% packet capture without placing CPU processing overhead on switches?