3.3 Intrusion Prevention System (IPS) Configuration

3.3 Intrusion Prevention System (IPS) Configuration

FortiOS Intrusion Prevention System (IPS) provides network-based threat protection against protocol exploits, software vulnerabilities, buffer overflows, and unauthorized access attempts. Operating within the flow-based IPS engine and accelerated by specialized Fortinet hardware, IPS sensors continuously inspect sessions passing through the FortiGate.

---

IPS Engine Architecture

The FortiGate IPS engine consists of three core scanning components:

1. Protocol Decoders: Parse application-layer protocols (HTTP, SMB, FTP, SSH, DNS, DCE/RPC) to normalize traffic streams and eliminate evasion techniques such as URL encoding, packet fragmentation, and obfuscated headers.
2. Signature Pattern Matcher: Compares normalized payload data against thousands of threat signatures supplied by FortiGuard. High-performance matching is accelerated using Content Processors (CP9/CP10) and SPU instructions.
3. Anomaly Detection Engine: Monitors network traffic metrics for protocol standard deviations (e.g., malformed TCP header flags, illegal ICMP packet sizes, or out-of-spec RPC calls).

---

IPS Sensors, Signatures, and Filters

An IPS Sensor is a collection of IPS rules and filters applied to firewall policies. An IPS sensor contains two rule construction methods: IPS Filters and Signature Overrides.

1. IPS Filter Construction

IPS Filters group signatures dynamically using five core metadata attributes:

• Severity: Critical, High, Medium, Low, Info.
• Target: Server (protects internal servers against incoming exploits) or Client (protects internal users against malicious drive-by responses).
• OS: Windows, Linux, macOS, Android, iOS, Cisco, etc.
• Protocol: HTTP, FTP, SMB, DNS, SSH, TELNET, IMAP, etc.
• Application Service: Specifies exact application contexts.

Advantage of IPS Filters: When FortiGuard releases new signature updates daily, any new signature matching an existing filter criteria (e.g., Target: Server, Severity: Critical) is automatically added to the sensor and assigned the configured action without requiring administrator intervention.

2. Signature Overrides

Signature Overrides explicitly select an individual signature by its unique Rule ID (e.g., ID 41532 for a specific Apache vulnerability). Overrides allow administrators to custom-tune actions for specific rules, such as changing an action from default to block, pass, or reset.

---

Rate-Based IPS Filters

Rate-based IPS filters defend against Denial of Service (DoS) attacks, port scans, and protocol brute-force attempts (such as SSH, FTP, or RDP password guessing) by monitoring event frequency.

Key Parameters

• Rate: Maximum allowed connection or packet count within the duration window.
• Duration: Measurement window in seconds.
• Track By: Tracks event counters per Source IP or Destination IP.
• Block Period: Time duration (in seconds) to quarantine and drop all traffic from the offending IP address once the rate threshold is breached.

Example: Setting Rate: 10, Duration: 60, Track By: Source IP, Block Period: 3600 on an SSH rate filter will block any source IP that initiates more than 10 SSH connection attempts within 60 seconds, preventing further access for 1 hour (3600 seconds).

---

IPS Engine Optimization & Fail-Open Tuning

During high-volume network spikes or DDoS events, IPS inspection can exhaust CPU cycles or system RAM buffers. FortiOS provides tuning controls to manage system behavior under load:

• Fail-Open Setting (set fail-open {enable | disable}):
  • enable (Default): If the IPS engine runs out of buffer memory, uninspected packets bypass the IPS engine and are allowed through the FortiGate. This prioritizes network availability over security enforcement.
  • disable: If the IPS engine runs out of buffer memory, uninspected packets are dropped. This prioritizes strict security enforcement over network availability.
• Hardware Offload (nTurbo): Ensures IPS stream scanning for established flows is offloaded to SPU hardware, bypassing CPU interrupt bottlenecks.

---

Overview Comparison

---

CLI Configuration Examples

Configuring a Comprehensive IPS Sensor

config ips sensor
    edit "Enterprise_Server_IPS"
        set comment "IPS Sensor protecting corporate web and database servers"
        config entries
            edit 1
                set location server
                set severity critical high
                set action block
                set status enable
            next
            edit 2
                set location client
                set severity critical
                set action block
                set status enable
            next
        end
        config override
            edit 41532
                set rule-id 41532
                set action block
                set status enable
            next
        end
    next
end

Configuring a Rate-Based IPS Filter for SSH Brute-Force Defense

config ips sensor
    edit "Enterprise_Server_IPS"
        config filter
            edit 1
                set name "SSH_BruteForce_Protection"
                set service "SSH"
                set action block
                set rate-count 10
                set rate-duration 60
                set rate-track src-ip
                set rate-block-period 3600
            next
        end
    next
end

Configuring Global IPS Fail-Open Setting

config ips global
    set fail-open disable
    set database extended
end