5.1 IPsec VPN

Why IPsec VPN Matters for NSE 4

IPsec (Internet Protocol Security) is the standard for permanently connecting two sites — a branch office to headquarters, for example — over an untrusted network such as the internet. On the Fortinet NSE 4 exam, VPN is roughly 20% of the question pool, and IPsec questions reliably test the two IKE phases, the difference between route-based and policy-based tunnels, and which firewall policies and routes make a tunnel pass traffic. Memorizing the negotiation order is the single highest-value preparation step for this section.

IPsec is a framework, not a single protocol. On FortiGate it combines three pieces:

• IKE (Internet Key Exchange) — negotiates and authenticates the tunnel. It runs over UDP port 500, and UDP port 4500 when NAT-Traversal (NAT-T) is in use.
• ESP (Encapsulating Security Payload) — IP protocol 50; encrypts and authenticates the user data once the tunnel is up.
• Security Associations (SAs) — the agreed set of keys, algorithms, and lifetimes that each direction of the tunnel uses.

The Two IKE Phases

IPsec is built in two negotiation stages. Confusing them is the most common NSE 4 mistake.

Phase 1 — Build the IKE SA

Phase 1 establishes a secure, authenticated channel between the two FortiGates themselves. It authenticates each peer (by pre-shared key or digital certificate), performs a Diffie-Hellman key exchange, and produces a single bidirectional IKE SA. No user traffic crosses Phase 1 — it exists only to protect the Phase 2 negotiation.

Phase 2 — Build the IPsec SAs

Phase 2 negotiates the parameters that actually encrypt user data. It runs inside the protected Phase 1 channel and produces a pair of IPsec SAs — one inbound, one outbound — because IPsec SAs are unidirectional. Phase 2 also defines the selectors (the local and remote subnets, sometimes called the quick mode selectors) that decide which traffic enters the tunnel. If Perfect Forward Secrecy (PFS) is enabled, Phase 2 runs its own Diffie-Hellman exchange so that compromise of one key never exposes past or future keys.

Phase 1 vs Phase 2 Comparison

Because Phase 2 SAs have a shorter lifetime, they are re-keyed more often than Phase 1 — that re-keying happens silently inside the existing IKE channel without re-authenticating the peers.

IKEv1 Main Mode vs Aggressive Mode

When Phase 1 uses IKE version 1, it runs in one of two modes:

Main Mode

Main Mode exchanges six messages. The peer identities are sent after the Diffie-Hellman exchange completes, so they travel encrypted and are hidden from anyone watching the link. Main Mode requires that each peer can be identified by a fixed IP address when authenticating with a pre-shared key, because the key is tied to the source address.

Aggressive Mode

Aggressive Mode compresses the same work into three messages. It is faster, and it allows a peer to be identified by a local ID string instead of an IP address — which is why it is commonly used for dial-up VPN where the remote peer has a dynamic address. The trade-off is security: the peer identity and a hash of the pre-shared key are sent in the clear, exposing the deployment to identity disclosure and offline PSK-cracking attacks.

IKEv2 is the modern choice and the FortiGate default for new wizard-built tunnels. It removes the Main/Aggressive distinction entirely, completes Phase 1 in a single four-message exchange, has built-in NAT-T, and uses fewer resources. On the exam, Main vs Aggressive is strictly an IKEv1 concept.

Tunnel Mode vs Transport Mode

IPsec can encapsulate traffic in two ways:

• Tunnel mode encrypts and encapsulates the entire original IP packet inside a new IP packet with new outer headers. The original source and destination addresses are hidden. This is what site-to-site VPNs use, and it is the FortiGate default.
• Transport mode encrypts only the payload and leaves the original IP header intact. It is used for host-to-host communication, not gateway-to-gateway tunnels.

For NSE 4, remember: site-to-site IPsec on FortiGate uses tunnel mode, which is why the remote private subnet stays unreadable to anyone inspecting the link.

Route-Based vs Policy-Based IPsec

FortiGate offers two ways to model an IPsec tunnel.

Route-Based (Interface Mode)

Route-based IPsec is the FortiGate default and the recommended approach. When the tunnel is created, FortiOS automatically generates a virtual IPsec tunnel interface. You then treat that interface like any other:

• Add a static route that sends the remote subnet toward the tunnel interface.
• Create firewall policies in both directions (local-to-tunnel and tunnel-to-local) to permit the traffic.

Route-based tunnels support dynamic routing (OSPF, BGP) across the VPN, multiple Phase 2 selectors, and redundant tunnels — which is why production deployments favor them.

Policy-Based (Tunnel Mode)

Policy-based IPsec has no tunnel interface. Instead, a single firewall policy with the special IPsec action both selects the interesting traffic and encrypts it. It is simpler for a one-off tunnel but cannot run dynamic routing and is considered legacy.

The IPsec VPN Wizard

The FortiGate GUI provides an IPsec Wizard that builds common tunnels from templates: Site to Site, Hub-and-Spoke, Remote Access (dial-up for FortiClient users), and a Custom option. The wizard collects the remote gateway, authentication method, and protected subnets, then automatically creates the Phase 1, Phase 2, firewall policies, static route, and any address objects in one pass. The Custom template exposes every Phase 1 and Phase 2 parameter for manual tuning.

Dial-Up VPN

A dial-up VPN is an IPsec tunnel where the remote peers have dynamic or unknown IP addresses. The FortiGate is configured as the responder only — its remote-gateway type is set to Dialup User — and it waits for peers to initiate. Many spokes or remote clients can connect to one dial-up Phase 1 definition. Because the responder cannot know the peer address in advance, dial-up tunnels rely on a peer ID (a local ID string) or certificates rather than IP-bound pre-shared keys.

Dead Peer Detection

Dead Peer Detection (DPD) lets a FortiGate notice when the peer at the other end of a tunnel has gone away — a reboot, a circuit failure, or a crash. The FortiGate sends periodic DPD keepalive probes; if the peer fails to answer within the configured retries, the FortiGate tears down the stale IKE and IPsec SAs so a fresh tunnel can be built (and so a backup tunnel can take over). On FortiOS, DPD can run on demand (probe only when traffic is idle and waiting) or on idle (probe whenever the link is quiet).

Firewall Policies and Routes a Tunnel Needs

Building the Phase 1 and Phase 2 definitions only negotiates the encryption — it does not make traffic flow. For a route-based tunnel you must also:

1. Add a route to the remote protected subnet pointing at the IPsec tunnel interface.
2. Create a firewall policy from the internal/LAN interface to the tunnel interface for outbound traffic.
3. Create a firewall policy from the tunnel interface back to the LAN for return traffic.

A classic NSE 4 troubleshooting scenario is a tunnel that shows up in the monitor but passes no data — the cause is almost always a missing route or a missing firewall policy, not a Phase 1/Phase 2 problem.