6.2 IPsec VPN Phase 1 & Phase 2 Fundamentals

6.2 IPsec VPN Phase 1 & Phase 2 Fundamentals

Internet Protocol Security (IPsec) is a suite of standards that secures IP communications by authenticating and encrypting each IP packet in a data stream. In FortiOS 7.6, IPsec VPNs form the backbone of site-to-site connectivity, branch office interconnectivity, SD-WAN overlays, and remote-access remote worker infrastructure. IPsec establishes secure tunnels using the Internet Key Exchange (IKE) protocol, which operates in two distinct phases: IKE Phase 1 and IKE Phase 2.

---

Route-Based vs. Policy-Based IPsec Architecture

FortiOS supports two distinct architectural implementations of IPsec VPNs: Route-Based and Policy-Based.

Exam Key Point: Route-based IPsec is the recommended best practice in FortiOS 7.6. Virtually all modern features—including SD-WAN, Auto-Discovery VPN (ADVPN), and dynamic routing protocols (BGP/OSPF)—require route-based IPsec virtual interfaces.

---

IKE Phase 1: Control Channel Negotiation

The primary objective of IKE Phase 1 is to authenticate the peer gateways and establish a secure, encrypted management channel (the IKE Security Association / SA) to protect subsequent Phase 2 negotiations.

IKE Protocol Versions (IKEv1 vs. IKEv2)

• IKEv1: Legacy protocol. Uses separate exchanges for Phase 1 (Main/Aggressive Mode) and Phase 2 (Quick Mode).
• IKEv2: Modern standard in FortiOS 7.6. Reduces negotiation overhead (4 messages total for Phase 1 and initial Phase 2), natively supports EAP authentication, handles dynamic IP changes (MOBIKE), and improves resilience against Denial-of-Service attacks.

IKEv1 Negotiation Modes

When using IKEv1, Phase 1 can be configured in one of two negotiation modes:

1. Main Mode (6 Messages):

   • Exchanges 6 packets to negotiate algorithms, exchange Diffie-Hellman keys, and verify identities.
   • Identity Protection: Identity payloads (IP addresses/certificates) are encrypted because authentication occurs after DH key exchange.
   • Requires static IP addresses on both peer gateways.

2. Aggressive Mode (3 Messages):

   • Exchanges only 3 packets to achieve faster tunnel establishment.
   • No Identity Protection: Identity payloads and pre-shared key hashes are sent unencrypted in the initial exchange.
   • Required in IKEv1 dial-up scenarios where the remote peer has a dynamic IP address and pre-shared key authentication is used without certificates.

Phase 1 Parameters & Proposals

During Phase 1, peers must agree on four critical parameters:

• Encryption Algorithm: AES-128, AES-256, AES-GCM-128, AES-GCM-256.
• Authentication Hash: SHA-256, SHA-384, SHA-512 (or Null for GCM modes).
• Diffie-Hellman (DH) Group: Determines the strength of the key exchange (e.g., Group 14 [2048-bit MODP], Group 19 [256-bit Elliptic Curve], Group 20 [384-bit EC], Group 21 [521-bit EC]).
• Authentication Method: Pre-Shared Key (PSK) or PKI Digital Certificates (RSA/ECDSA signatures).

config vpn ipsec phase1-interface
    edit "To_Branch_Office"
        set interface "port1"
        set ike-version 2
        set peertype any
        set net-device enable
        set proposal aes256-sha256 aes128-sha256
        set dhgrp 14 19
        set remote-gw 198.51.100.25
        set psksecret ENC dXNlcnNlY3JldA==
    next
end

---

IKE Phase 2: Data Channel Negotiation (Quick Mode)

Once Phase 1 establishes the secure IKE SA, IKE Phase 2 negotiates child security associations (IPsec SAs) used to encrypt and authenticate actual user payload traffic.

Key Phase 2 Parameters

• Encapsulation Security Payload (ESP) Proposals: Defines encryption and authentication algorithms specifically for user data (e.g., set proposal aes256-sha256).
• Phase 2 Selectors (local-subnet & remote-subnet):
  • Defines the source and destination traffic subnets authorized to enter the IPsec tunnel.
  • In Route-Based IPsec, selectors are configured as 0.0.0.0/0 (Local) to 0.0.0.0/0 (Remote). This allows any traffic directed into the IPsec virtual interface by the routing table to be encrypted.
  • Both peers must have matching inverse selectors (Local Subnet on Gateway A must match Remote Subnet on Gateway B).

Perfect Forward Secrecy (PFS)

• When PFS is enabled (set dhgrp), Phase 2 forces a brand-new, independent Diffie-Hellman key exchange for every Quick Mode SA creation.
• Security Benefit: If an attacker compromises the long-term Phase 1 secret key in the future, they cannot retroactively decrypt past Phase 2 session traffic because Phase 2 keys were generated independently via PFS DH exchanges.

config vpn ipsec phase2-interface
    edit "To_Branch_Office_P2"
        set phase1name "To_Branch_Office"
        set proposal aes256-sha256
        set dhgrp 14 19
        set pfs enable
        set auto-key-keep-alive enable
        set src-subnet 0.0.0.0 0.0.0.0
        set dst-subnet 0.0.0.0 0.0.0.0
    next
end

---

Site-to-Site vs. Dial-Up IPsec Tunnels

• Site-to-Site (type static): Connects two fixed locations with known static public IP addresses. Both gateways actively initiate negotiation.
• Dial-Up (type dynamic): Used when remote gateways (spokes or FortiClient users) have dynamic public IP addresses. The Hub acts as a responder (type dynamic) and waits for incoming connections, identifying peers via Peer ID (set peerid) or digital certificates.

---

Diagnostic & Troubleshooting CLI Reference

Troubleshooting IPsec issues requires inspecting real-time IKE daemon events using diagnose debug application ike -1.

# 1. Enable real-time IKE debug output
diagnose debug reset
diagnose debug application ike -1
diagnose debug enable

# 2. Trigger tunnel renegotiation
diagnose vpn tunnel up To_Branch_Office_P2

# 3. View active IPsec SA summary
get vpn ipsec tunnel summary

# 4. View detailed Phase 1 & Phase 2 SA state
diagnose vpn tunnel list name To_Branch_Office_P2

Common Debug Log Indicators

• no SA proposal chosen: Indicates a mismatch in Phase 1 or Phase 2 encryption/hash/DH algorithms.
• PSK mismatch / authentication failed: Incorrect pre-shared key configured on one of the gateways.
• selector mismatch: Local and remote subnets in Phase 2 do not match reciprocally between peers.