5.3 GRE & IPsec Tunnel Virtualization Mechanics

Key Takeaways

  • Generic Routing Encapsulation (GRE / RFC 2784) is an unencrypted, stateless tunneling protocol (IP protocol 47) that encapsulates multi-protocol payload packets inside an outer IP header, enabling dynamic routing protocols (OSPF, EIGRP, BGP) and multicast traffic across unicast IP networks.
  • A standard GRE/IP tunnel introduces exactly 24 bytes of overhead (20-byte outer IPv4 header + 4-byte GRE header), reducing the default 1500-byte interface MTU to an IP MTU of 1476 bytes and requiring TCP MSS clamping (`ip tcp adjust-mss 1436`) to prevent Path MTU Discovery black holes.
  • IPsec operates in two modes: Transport Mode (encrypts only the payload and preserves the original IP header, adding 36-56 bytes of overhead) and Tunnel Mode (encapsulates and encrypts the entire original IP packet within a new outer IP header, adding 50-70+ bytes of overhead).
  • IPsec Security Protocols comprise Encapsulating Security Payload (ESP / IP protocol 50), which provides data confidentiality (encryption), integrity, data origin authentication, and anti-replay, and Authentication Header (AH / IP protocol 51), which provides integrity and authentication without encryption and fails when traversing NAT.
  • Internet Key Exchange version 2 (IKEv2 / RFC 7296) streamlines Phase 1 and Phase 2 negotiations into 4 initial messages (IKE_SA_INIT and IKE_AUTH), natively integrates NAT-Traversal (UDP port 4500), supports asymmetric authentication methods, and provides built-in Dead Peer Detection (DPD) and DoS cookie protection.
Last updated: August 2026

5.3 GRE & IPsec Tunnel Virtualization Mechanics

Core Blueprint Focus: Cisco 350-401 ENCOR v1.2 topic 2.2.b (GRE and IPsec tunneling) requires candidates to configure and verify GRE tunnels, analyze encapsulation overhead and MTU/MSS clamping, differentiate between IPsec Transport and Tunnel modes, evaluate ESP versus AH security protocols, and implement modern IKEv2 site-to-site Virtual Tunnel Interfaces (VTI) on Cisco IOS-XE platforms.

Enterprise WANs and cloud architectures require logical overlay networks to interconnect geographically dispersed sites across untrusted or multi-provider transport backbones. Tunnel virtualization technologies decouple the overlay logical topology from the underlying physical transport, enabling dynamic routing, multicast distribution, and end-to-end cryptographic protection.


1. Generic Routing Encapsulation (GRE / RFC 2784)

Generic Routing Encapsulation (GRE) is a lightweight, stateless tunneling protocol developed by Cisco and standardized in RFC 2784 and RFC 1701. GRE encapsulates a wide variety of network-layer payload protocols (such as IPv4, IPv6, and MPLS) inside an outer IP transport header.

+-------------------------------------------------------------------------+
|                    GRE PACKET ENCAPSULATION ANATOMY                     |
|                                                                         |
|   |<---------------------- Outer IP Packet (1500 B) ------------------>| |
|   |                                                                   | |
|   +-------------------+---------------+-------------------------------+ |
|   |  Outer IP Header  |  GRE Header   |     Original Passenger IP     | |
|   |    (Delivery)     |  (RFC 2784)   |    Packet (Inner IP + Data)   | |
|   |     20 Bytes      |    4 Bytes    |          1476 Bytes           | |
|   +-------------------+---------------+-------------------------------+ |
|   | Protocol: 47      | EtherType:    | Src/Dst: 10.1.1.1 -> 10.2.2.2 | |
|   | Src: 198.51.100.1 | 0x0800 (IPv4) | Payload: TCP/UDP Data         | |
|   | Dst: 203.0.113.1  |               |                               | |
|   +-------------------+---------------+-------------------------------+ |
+-------------------------------------------------------------------------+

Key Technical Characteristics of GRE

  • IP Protocol Number 47: GRE packets operate directly on top of IP using IP protocol number 47.
  • Multicast and Broadcast Transport: Unlike native IPsec tunnel mode (which cannot encapsulate multicast without specialized extensions), GRE natively encapsulates IP multicast and broadcast packets. This allows dynamic routing protocols (e.g., OSPF 224.0.0.5/224.0.0.6, EIGRP 224.0.0.10) to establish neighbor adjacencies across the tunnel.
  • Stateless Operation: GRE tunnels have no keepalive state machine by default. An unconfigured GRE tunnel interface remains in up/up line protocol state as long as a valid route exists to the tunnel destination IP address in the local routing table. Enabling keepalive sends periodic GRE keepalives to detect remote endpoint failure.
  • No Native Encryption: GRE provides zero data confidentiality. Payloads are transmitted in cleartext unless encapsulated inside an IPsec security association (GRE over IPsec).

Standard GRE Header Fields (RFC 2784)

  • C (Checksum Present - Bit 0): Set to 1 if a Checksum field is present.
  • Flags & Version (Bits 1-15): Must be set to zero in RFC 2784 compliance (Version = 0).
  • Protocol Type (Bits 16-31, 2 Bytes): Specifies the EtherType of the encapsulated passenger payload (e.g., 0x0800 for IPv4, 0x86DD for IPv6).

2. Tunnel Overhead, MTU Fragmentation & TCP MSS Clamping

When encapsulating packets inside GRE or IPsec tunnels, the added headers increase the total packet size. If the resulting packet exceeds the physical path Maximum Transmission Unit (MTU), fragmentation occurs, leading to performance degradation, retransmissions, or dropped packets.

+-------------------------------------------------------------------------+
|                   GRE TUNNEL MTU & TCP MSS CALCULATION                  |
|                                                                         |
|   |<------------------- Physical Link MTU: 1500 Bytes ---------------->| |
|   |                                                                   | |
|   +------------------+--------------+---------------------------------+ |
|   | Outer IPv4 (20B) |   GRE (4B)   |     Inner IP Payload (1476B)    | |
|   +------------------+--------------+---------------------------------+ |
|   | <--- GRE Overhead: 24B ---> |   |                                 |
|                                     |                                 |
|                                     |<------ IP MTU: 1476 Bytes ----->| |
|                                     |                                 |
|                                     +------------+-----------+--------+ |
|                                     | Inner IPv4 |    TCP    | TCP    | |
|                                     | Header(20B)|Header(20B)| Payload| |
|                                     +------------+-----------+--------+ |
|                                     | <-- Header Overhead: 40B --> |  |
|                                     |                                 |
|                                     |<----- TCP MSS: 1436 Bytes ----->| |
+-------------------------------------------------------------------------+

Mathematical Calculations

  1. Standard GRE Overhead: GRE Overhead=Outer IPv4 Header (20 Bytes)+GRE Header (4 Bytes)=24 Bytes\text{GRE Overhead} = \text{Outer IPv4 Header (20 Bytes)} + \text{GRE Header (4 Bytes)} = 24\text{ Bytes}

  2. Tunnel Interface IP MTU: IP MTU=Physical Interface MTU (1500 Bytes)GRE Overhead (24 Bytes)=1476 Bytes\text{IP MTU} = \text{Physical Interface MTU (1500 Bytes)} - \text{GRE Overhead (24 Bytes)} = 1476\text{ Bytes}

  3. TCP Maximum Segment Size (MSS) Clamping: TCP MSS=Tunnel IP MTU (1476 Bytes)[Inner IP Header (20B)+TCP Header (20B)]\text{TCP MSS} = \text{Tunnel IP MTU (1476 Bytes)} - \left[ \text{Inner IP Header (20B)} + \text{TCP Header (20B)} \right] TCP MSS=147640=1436 Bytes\text{TCP MSS} = 1476 - 40 = 1436\text{ Bytes}

Path MTU Discovery (PMTUD) & The "Black Hole" Problem

When an endpoint sends a packet with the Don't Fragment (DF) bit set in the IP header and the packet size exceeds an intermediate router's MTU:

  1. The router drops the packet.
  2. The router transmits an ICMP Type 3, Code 4 message (Destination Unreachable: Fragmentation Needed and DF Set) containing the next-hop MTU back to the sender.
  3. The Black Hole Problem: In many enterprise WANs and internet connections, intermediate firewalls or security gateways drop all inbound ICMP messages. As a result, the sender never receives the ICMP error, never reduces its packet size, and continues retransmitting dropped packets indefinitely.
  4. The Solution (TCP MSS Clamping): By applying ip tcp adjust-mss 1436 to the tunnel interface, the router intercepts TCP SYN and SYN-ACK packets during the initial 3-way handshake and rewrites the advertised MSS option value to 1436 bytes. This forces both client and server to limit their TCP segments, completely eliminating PMTUD black holes and fragmentation.

3. IPsec Security Architecture: Transport vs. Tunnel Mode

IP Security (IPsec) is an open architectural framework (RFC 4301) providing network-layer security services: Confidentiality (encryption), Data Integrity, Data Origin Authentication, and Anti-Replay Protection.

+-------------------------------------------------------------------------+
|                     IPSEC TRANSPORT VS. TUNNEL MODE                     |
|                                                                         |
|   CLEARTEXT IP PACKET:                                                  |
|   +-----------------------+-------------------------------------------+ |
|   |    Orig IP Header     |               TCP / Payload               | |
|   +-----------------------+-------------------------------------------+ |
|                                                                         |
|   IPSEC TRANSPORT MODE (Host-to-Host / GRE-over-IPsec / VTI):           |
|   +---------------+------------+----+-------------------+-----+-------+ |
|   | Orig IP Header| ESP Header | IV |   TCP / Payload   | ESP |  ESP  | |
|   | (Preserved)   | (Prot 50)  |    |    (ENCRYPTED)    |Trlr |  Auth | |
|   +---------------+------------+----+-------------------+-----+-------+ |
|                   |<-------------- ENCRYPTED ---------->|             | |
|                   |<----------------- AUTHENTICATED ----------------->| |
|                                                                         |
|   IPSEC TUNNEL MODE (Site-to-Site LAN-to-LAN VPN):                      |
|   +---------------+------------+----+-------------------+-----+-------+ |
|   | New IP Header | ESP Header | IV |  Original Packet  | ESP |  ESP  | |
|   | (New Gateway) | (Prot 50)  |    | (IP Header + Data)|Trlr |  Auth | |
|   +---------------+------------+----+-------------------+-----+-------+ |
|                   |<-------------- ENCRYPTED ---------->|             | |
|                   |<----------------- AUTHENTICATED ----------------->| |
+-------------------------------------------------------------------------+

Transport Mode vs. Tunnel Mode Comparison

Technical AttributeIPsec Transport ModeIPsec Tunnel Mode
Header PreservationPreserves original IP header; inserts ESP after IP headerEncapsulates entire original IP packet inside a New Outer IP header
Payload ProtectionEncrypts upper-layer payload (Transport/Application)Encrypts entire original packet (Inner IP header + Payload)
Addressing VisibilityOriginal source/destination IP addresses are exposed on wireOriginal source/destination IP addresses are completely hidden
Protocol OverheadLower overhead (typically 36–56 bytes)Higher overhead (typically 50–74 bytes due to extra IP header)
Typical DeploymentEnd-to-end host communication; GRE-over-IPsec; VTITraditional Site-to-Site Gateway-to-Gateway VPNs

4. IPsec Security Protocols: ESP (Protocol 50) vs. AH (Protocol 51)

IPsec provides two distinct Layer 3 security protocols:

+-------------------------------------------------------------------------+
|                         ESP VS. AH PROTOCOL MATRIX                      |
|                                                                         |
|   SECURITY SERVICE               ESP (Protocol 50)    AH (Protocol 51)  |
|   -------------------------------------------------------------------   |
|   Confidentiality (Encryption)   YES (AES-GCM/CBC)    NO (Cleartext)    |
|   Data Integrity (HMAC/Hashing)  YES (SHA-256/512)    YES (SHA-256/512) |
|   Data Origin Authentication     YES (Pre-Shared/Cert)YES (Pre-Shared)  |
|   Anti-Replay Protection         YES (Seq Numbers)    YES (Seq Numbers) |
|   NAT Traversal (NAT-T UDP 4500) YES                  NO (NAT Incompatible)
+-------------------------------------------------------------------------+

Encapsulating Security Payload (ESP - IP Protocol 50)

  • Complete Security Suite: ESP provides confidentiality via symmetric encryption algorithms (AES-128, AES-256, AES-GCM), integrity and authentication via HMAC hashes (SHA-256, SHA-384, SHA-512), and anti-replay protection via 32-bit/64-bit sequence numbers.
  • NAT-Traversal (NAT-T / RFC 3948): When a router or firewall detects a Network Address Translation (NAT/PAT) device along the path, standard ESP fails because ESP contains no Layer 4 port numbers for PAT translation. NAT-T solves this by encapsulating ESP inside UDP port 4500, allowing encrypted packets to traverse PAT gateways seamlessly.

Authentication Header (AH - IP Protocol 51)

  • Integrity Without Confidentiality: AH calculates an Integrity Check Value (ICV) across the payload and the outer IP header (excluding mutable fields like TTL and TOS).
  • Zero Encryption: All data is transmitted in cleartext.
  • NAT Incompatibility: When a NAT router translates IP addresses or ports, the altered IP header invalidates the AH ICV hash at the destination peer, causing immediate packet drops. AH cannot operate across NAT.

5. Key Exchange Protocols: IKEv1 vs. IKEv2 Architecture

Internet Key Exchange (IKE) dynamically authenticates peers, negotiates cryptographic algorithms, and manages Security Associations (SAs) using UDP port 500 (and UDP port 4500 for NAT-T).

+-------------------------------------------------------------------------+
|                       IKEv2 4-MESSAGE NEGOTIATION                       |
|                                                                         |
|   INITIATOR                                              RESPONDER      |
|                                                                         |
|   [1. IKE_SA_INIT Request]  ============================>               |
|       - Security Proposal (Encryption, Hash, DH Group)                  |
|       - Diffie-Hellman Public Value (g^x) & Nonce (Ni)                  |
|                                                                         |
|                             <===========================  [2. Response] |
|                                 - Chosen Proposal & DH Value (g^y)      |
|                                 - Nonce (Nr) & Optional NAT Detection   |
|                                                                         |
|   ================== ENCRYPTED CHANNEL ESTABLISHED ===================  |
|                                                                         |
|   [3. IKE_AUTH Request]     ============================>               |
|       - Peer Identity (IDi: IP/FQDN/Cert)                               |
|       - Authentication (PSK Hash or Digital Signature)                  |
|       - First Child SA Proposal (Traffic Selectors / TSi, TSr)          |
|                                                                         |
|                             <===========================  [4. Response] |
|                                 - Responder Identity (IDr) & Auth Data  |
|                                 - First Child SA Accepted Proposal      |
+-------------------------------------------------------------------------+

IKEv2 vs. IKEv1 Architectural Improvements

Technical AttributeIKEv1 (RFC 2409)IKEv2 (RFC 7296)
Message ExchangesPhase 1: 6 packets (Main Mode) or 3 (Aggressive) + Phase 2: 3 packets (Quick Mode) = 6 to 9 packets4 initial packets (IKE_SA_INIT + IKE_AUTH) to establish both IKE SA and first Child SA
NAT-Traversal SupportVendor-specific draft extension; requires post-discovery renegotiationNatively integrated in IKE_SA_INIT; automatic port float to UDP 4500
Authentication FlexibilitySymmetric only (both peers must use PSK or both use Certs)Asymmetric authentication (e.g., PSK on one peer, RSA certificate on other, or EAP)
Dead Peer Detection (DPD)Optional proprietary keepalive extensionNative standard keepalive mechanism
DoS / Anti-SpoofingVulnerable to half-open state exhaustionAnti-DoS Cookie Challenge mechanism under attack conditions
Bandwidth EfficiencyHigher control-plane overheadSignificantly reduced round-trip latency and CPU overhead

6. Cisco IOS-XE GRE & IKEv2 SVTI Configuration & Verification

The following configuration illustrates establishing an IKEv2 Static Virtual Tunnel Interface (SVTI) over IPv4 with AES-256-GCM encryption and TCP MSS clamping on Cisco IOS-XE.

! Step 1: Configure IKEv2 Proposal and Policy
crypto ikev2 proposal IKEV2-PROP-ENTERPRISE
 encryption aes-gcm-256
 prf sha384
 group 19 20
!
crypto ikev2 policy IKEV2-POL-ENTERPRISE
 proposal IKEV2-PROP-ENTERPRISE
!
! Step 2: Configure IKEv2 Keyring (Pre-Shared Keys)
crypto ikev2 keyring IKEV2-KEYRING-BRANCH
 peer BRANCH-ROUTER-1
  address 203.0.113.2
  pre-shared-key CiscoSecureKey2026!
!
! Step 3: Configure IKEv2 Profile
crypto ikev2 profile IKEV2-PROF-BRANCH
 match identity remote address 203.0.113.2 255.255.255.255
 identity local address 198.51.100.1
 authentication remote pre-share
 authentication local pre-share
 keyring local IKEV2-KEYRING-BRANCH
 dpd 10 2 on-demand
!
! Step 4: Configure IPsec Transform Set and IPsec Profile
crypto ipsec transform-set TS-AES-GCM esp-gcm 256
 mode transport
!
crypto ipsec profile IPSEC-PROF-VTI
 set transform-set TS-AES-GCM
 set ikev2-profile IKEV2-PROF-BRANCH
!
! Step 5: Configure the Static Virtual Tunnel Interface (SVTI)
interface Tunnel100
 description Secure Site-to-Site IKEv2 VTI to Branch 1
 ip address 10.254.1.1 255.255.255.252
 ip mtu 1400
 ip tcp adjust-mss 1360
 tunnel source GigabitEthernet0/0/0
 tunnel destination 203.0.113.2
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile IPSEC-PROF-VTI
 no shutdown

Verification and Diagnostic Commands

! Verify IKEv2 Phase 1 Security Association
Router# show crypto ikev2 sa
IPv4 Crypto IKEv2  SA
Tunnel-id Local                 Remote                fvrf/ivrf            Status
1         198.51.100.1/500      203.0.113.2/500       none/none            READY
      Encr: AES-GCM, keysize: 256, PRF: SHA384, Hash: None, DH Grp: 19, Auth: PRE-SHARED
      Life/Active Time: 86400/1245 sec

! Verify IPsec Phase 2 Child Security Association (Encaps/Decaps counters)
Router# show crypto ipsec sa interface Tunnel100
interface: Tunnel100
    Crypto map tag: Tunnel100-head-0, local addr 198.51.100.1

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   current_peer 203.0.113.2 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 48291, #pkts encrypt: 48291, #pkts digest: 0
    #pkts decaps: 48280, #pkts decrypt: 48280, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts no encapsulation: 0, #pkts encapsulation failed: 0
    #send errors 0, #recv errors 0

     local crypto endpt.: 198.51.100.1, remote crypto endpt.: 203.0.113.2
     plaintext mtu 1400, path mtu 1500, ip mtu 1400, ip mtu idb GigabitEthernet0/0/0
     current outbound spi: 0x8F4A12BC(2403996348)
     inbound esp sas:
      spi: 0x3C910B8D (1016138637)
        transform: esp-gcm 256 , in use settings ={\Transport, }
        in use sas: conn_id=2001, flow_id=1
        sa timing: remaining key lifetime (sec): 3420
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE(sa:ready)
Loading diagram...
GRE vs IPsec Packet Layouts & IKEv2 State Machine
Test Your Knowledge

A network engineer configures a point-to-point standard GRE tunnel across an Ethernet physical infrastructure with a standard MTU of 1500 bytes. To prevent packet fragmentation and eliminate Path MTU Discovery (PMTUD) black holes for TCP traffic traversing the tunnel, what IP MTU and TCP MSS values must be configured on the tunnel interface?

A
B
C
D
Test Your Knowledge

Why does the Authentication Header (AH / IP protocol 51) fail when traversing a Network Address Translation (NAT/PAT) boundary, while Encapsulating Security Payload (ESP) with NAT-Traversal succeeds?

A
B
C
D
Test Your Knowledge

When contrasting IKEv1 and IKEv2 protocols for site-to-site IPsec VPN deployments, what is a primary operational advantage of IKEv2?

A
B
C
D
Test Your Knowledge

A network architect is designing an encrypted tunnel between two corporate data centers over a public transit provider. The design specifies configuring a Virtual Tunnel Interface (VTI) with IPsec protection. Why should IPsec Transport Mode be configured instead of IPsec Tunnel Mode on the transform set?

A
B
C
D