6.2 VXLAN & EVPN Overlay Architecture

6.2 VXLAN & EVPN Overlay Architecture

Traditional enterprise data centers and campus networks built on classical Layer 2 technologies face severe architectural limitations. The IEEE 802.1Q standard provides a 12-bit VLAN identifier, capping total network segments at 4,094 VLANs—insufficient for modern multi-tenant cloud environments. Furthermore, classical Layer 2 networks rely on Spanning Tree Protocol (STP) to prevent loops, which intentionally disables redundant uplinks and limits bandwidth utilization to 50%.

Virtual Extensible LAN (VXLAN) (RFC 7348) combined with Multiprotocol BGP Ethernet VPN (MP-BGP EVPN) (RFC 7432 and RFC 8365) solves these scalability bottlenecks by creating a flexible Layer 2/Layer 3 overlay across an all-routed Layer 3 Spine-Leaf underlay.

+-----------------------------------------------------------------------------------+
|                         CLASSICAL VLAN VS. VXLAN OVERLAY                          |
+-----------------------------------------------------------------------------------+
|  CLASSICAL 802.1Q LAYER 2:                                                        |
|  - 12-bit VLAN ID -> Maximum 4,094 network segments.                              |
|  - Spanning Tree Protocol (STP) blocks redundant uplinks (50% wasted bandwidth).  |
|  - Broadcast/Unknown Unicast flooding extends across entire physical broadcast domains.|
|                                                                                   |
|  VXLAN + MP-BGP EVPN OVERLAY:                                                     |
|  - 24-bit VNI -> Over 16.7 Million (16,777,216) isolated overlay segments.       |
|  - Layer 3 Routed Spine-Leaf underlay with 100% ECMP bandwidth utilization.       |
|  - Control plane MAC/IP distribution via BGP; data-plane broadcast flooding eliminated.|
+-----------------------------------------------------------------------------------+

1. VXLAN Fundamentals & Data Plane Encapsulation

VXLAN is a MAC-in-UDP encapsulation protocol. It takes a complete Layer 2 Ethernet frame generated by an endpoint, prepends a VXLAN header, wraps it in a standard UDP/IP transport header, and routes it across the underlay network.

+-----------------------------------------------------------------------------------+
|                         VXLAN PACKET ENCAPSULATION FORMAT                         |
+-----------------------------------------------------------------------------------+
| Outer Ethernet Header (14B):  Src MAC = Leaf-1 Uplink, Dst MAC = Spine-1 Interface|
+-----------------------------------------------------------------------------------+
| Outer IP Header (20B):        Src IP = Ingress VTEP, Dst IP = Egress VTEP IP       |
|                               Protocol = 17 (UDP)                                 |
+-----------------------------------------------------------------------------------+
| Outer UDP Header (8B):        Src Port = Hash(Inner 5-Tuple), Dst Port = 4789     |
+-----------------------------------------------------------------------------------+
| VXLAN Header (8B):            [ Flags: 8b (I=1) ] [ Reserved: 24b ]               |
|                               [ 24-bit VNI ] [ Reserved: 8b ]                     |
+-----------------------------------------------------------------------------------+
| Inner Ethernet Frame:         Original Host MACs (Src/Dst), 802.1Q (optional),    |
|                               Inner IPv4/IPv6 Header, User Data Payload & FCS     |
+-----------------------------------------------------------------------------------+

Detailed Encapsulation Fields

1. Outer Ethernet Header (14 Bytes):

   • Contains the Layer 2 MAC addresses for point-to-point physical transmission between directly connected underlay devices (e.g., Leaf to Spine).
   • Rewritten at every routed hop across the underlay fabric.

2. Outer IP Header (20 Bytes IPv4 / 40 Bytes IPv6):

   • Source IP: IP address of the source VXLAN Tunnel Endpoint (VTEP) (usually a Loopback interface).
   • Destination IP: IP address of the destination remote VTEP (for unicast traffic) or an underlay Multicast Group IP (for multicast-based BUM traffic).

3. Outer UDP Header (8 Bytes):

   • Destination Port: 4789 (Standard IANA-assigned port for VXLAN per RFC 7348). (Note: Legacy pre-standard Cisco implementations used port 8472).
   • Source Port: Calculated dynamically by the ingress VTEP as a hash of the inner frame's Layer 2/3/4 headers. This ensures that different host flows generate distinct UDP source ports, enabling intermediate Spine switches to distribute traffic evenly across all Equal-Cost Multi-Path (ECMP) routed paths without inspecting inner payload headers.

4. VXLAN Header (8 Bytes):

   • Flags (8 bits): The I-bit (Instance bit) must be set to 1 to indicate a valid VNI. The remaining 7 bits are reserved and set to 0.
   • VXLAN Network Identifier (VNI / VNID) (24 bits): Designates the individual overlay segment. A 24-bit field allows $2^{24} = 16,777,216$ unique broadcast domains.

MTU Considerations and Overhead

• VXLAN encapsulation adds 50 bytes of overhead for standard IPv4 underlay transport (Outer Ethernet 14B + Outer IP 20B + UDP 8B + VXLAN 8B) or 54 bytes if an outer 802.1Q tag is present.
• If an endpoint transmits a standard 1,500-byte MTU frame, the encapsulated packet will reach 1,550+ bytes. To prevent underlay fragmentation or dropped packets, all switch interfaces in the routed underlay must be configured for Jumbo Frames (typically an MTU of 9000 to 9216 bytes).

2. MP-BGP EVPN Control Plane Architecture

Early VXLAN deployments relied on Data-Plane Flood-and-Learn (using underlay PIM multicast groups or ingress replication) to discover remote MAC addresses and map them to remote VTEP IP addresses. This caused significant control plane overhead, broadcast storm vulnerabilities, and slow convergence.

Multiprotocol BGP Ethernet VPN (MP-BGP EVPN) (RFC 7432 / RFC 8365) introduces a standardized, scalable control plane using BGP Address Family Identifier (AFI) 25 (L2VPN) and Subsequent Address Family Identifier (SAFI) 70 (EVPN).

+-----------------------------------------------------------------------------------+
|                         MP-BGP EVPN CONTROL PLANE BENEFITS                        |
+-----------------------------------------------------------------------------------+
| 1. Control Plane Learning: MAC and IP addresses are advertised proactively via    |
|    BGP update messages rather than learned from data plane broadcast flooding.    |
| 2. ARP Suppression: Leaf switches answer local host ARP requests directly from     |
|    their BGP EVPN database, eliminating fabric-wide ARP broadcast flooding.       |
| 3. Dual-Stack Host Routing: Advertises host /32 (IPv4) and /128 (IPv6) routes      |
|    directly into VRF routing tables for optimal inter-subnet forwarding.          |
| 4. Multi-Tenancy Isolation: Uses Route Distinguishers (RD) and Route Targets (RT) |
|    to enforce complete cryptographic separation across overlapping tenant VRFs.   |
+-----------------------------------------------------------------------------------+

Route Distinguishers (RD) and Route Targets (RT)

• Route Distinguisher (RD) (8 Bytes): Prepended to an EVPN route (e.g., 10.255.1.1:10010) to create a globally unique prefix across BGP peers, preventing route collisions when multiple tenants use identical overlapping private IP or MAC addresses.
• Route Target (RT): BGP Extended Communities attached to EVPN routes that control import and export policies. Leaf switches use RTs to import specific Layer 2 VNIs (MAC-VRFs) and Layer 3 VNIs (IP-VRFs) into their local hardware tables.

3. MP-BGP EVPN Route Types Deep Dive

MP-BGP EVPN defines several distinct Network Layer Reachability Information (NLRI) Route Types. The ENCOR 350-401 blueprint emphasizes understanding Route Types 2, 3, and 5.

+-----------------------------------------------------------------------------------+
|                             MP-BGP EVPN ROUTE TYPES                               |
+-----------------------------------------------------------------------------------+
| Type 1: Ethernet Auto-Discovery (A-D) Route   - Fast convergence & multi-homing   |
| Type 2: MAC / IP Advertisement Route         - Advertises host MAC & /32 host IP  |
| Type 3: Inclusive Multicast Ethernet Tag (IMET) - Discovers VTEPs for BUM traffic |
| Type 4: Ethernet Segment Route               - Multi-homing DF election & loops   |
| Type 5: IP Prefix Route                      - Inter-subnet & external WAN routing|
+-----------------------------------------------------------------------------------+

Summary of Primary EVPN Route Types

4. Distributed Anycast Gateway & ARP Suppression

In traditional Layer 2 designs, all default gateways resided on centralized core or distribution switches (e.g., via HSRP/VRRP). When two servers attached to different access switches communicated across subnets, traffic was forced to travel up to the core gateway and back down to the access switch—a phenomenon called traffic hair-pinning or tromboning.

Distributed Anycast Gateway (DAG)

EVPN replaces centralized gateways with a Distributed Anycast Gateway (DAG):

• Identical IP and MAC across All Leaves: The identical virtual default gateway IP address (e.g., 10.1.10.1/24) and virtual MAC address (e.g., 0000.5e00.0101 or fabric anycast MAC) are configured on every leaf switch hosting that subnet/VNI.
• Local First-Hop Routing: When any host transmits an ARP request for its default gateway, the directly connected leaf switch responds immediately in hardware. Inter-subnet routing occurs locally at line rate on the first-hop leaf switch.
• Seamless VM Mobility: When a virtual machine moves to a different physical leaf switch, its configured default gateway IP and MAC remain 100% identical. The VM continues communicating without updating its ARP table or default route.

+-----------------------------------------------------------------------------------+
|                TRADITIONAL HAIRPINNING VS. DISTRIBUTED ANYCAST GATEWAY            |
+-----------------------------------------------------------------------------------+
|  TRADITIONAL GATEWAY (HAIRPINNING):                                               |
|  [VM-1] ---> [Leaf-1] ---> [Core/Dist HSRP Gateway] ---> [Leaf-2] ---> [VM-2]     |
|  (Traffic traverses the core twice for inter-subnet communication).               |
|                                                                                   |
|  DISTRIBUTED ANYCAST GATEWAY (DAG):                                               |
|  [VM-1] ---> [Leaf-1 (Routes Locally via DAG)] ===VXLAN===> [Leaf-2] ---> [VM-2]  |
|  (Routed at first hop; transported directly across the underlay).                 |
+-----------------------------------------------------------------------------------+

Local ARP Suppression Mechanics

Broadcast ARP requests for other endpoints create massive broadcast overhead in large fabrics. EVPN solves this with ARP Suppression:

1. When Host B attaches to Leaf-2, Leaf-2 learns Host B's MAC and IP and advertises an EVPN Route Type 2 to all leaf switches.
2. When Host A (on Leaf-1) broadcasts an ARP request for Host B's IP (10.1.10.12), Leaf-1 intercepts the ARP broadcast.
3. Leaf-1 checks its local EVPN ARP suppression cache. Finding the entry learned from the Type 2 BGP route, Leaf-1 replies directly to Host A with Host B's MAC address in proxy.
4. The broadcast ARP frame is never flooded across the VXLAN fabric, eliminating broadcast storms.

5. Integrated Routing and Bridging (IRB): Asymmetric vs. Symmetric IRB

When traffic must travel between two different subnets / VNIs (inter-VNI routing), the fabric implements Integrated Routing and Bridging (IRB). There are two primary architectural models for IRB:

+-----------------------------------------------------------------------------------+
|                         ASYMMETRIC IRB VS. SYMMETRIC IRB                          |
+-----------------------------------------------------------------------------------+
|  ASYMMETRIC IRB:                                                                  |
|  [Ingress VTEP] : Performs ROUTING (VNI 10 -> VNI 20) + BRIDGING onto VNI 20.     |
|  [Overlay Core] : Transports packet encapsulated with Destination L2VNI (VNI 20).  |
|  [Egress VTEP]  : Performs ONLY BRIDGING out to destination host.                 |
|  - Limitation   : Ingress VTEP MUST configure and maintain all destination VNIs.   |
|                                                                                   |
|  SYMMETRIC IRB:                                                                   |
|  [Ingress VTEP] : Performs ROUTING (VNI 10 -> Transit L3VNI 50001).               |
|  [Overlay Core] : Transports packet encapsulated with Tenant L3VNI (VNI 50001).   |
|  [Egress VTEP]  : Receives on L3VNI, performs ROUTING into destination VNI 20.     |
|  - Advantage    : VTEPs only instantiate locally connected VNIs + common L3VNI.    |
+-----------------------------------------------------------------------------------+

Detailed Comparison: Asymmetric vs. Symmetric IRB

6. Cisco IOS-XE EVPN VXLAN Configuration & Verification

1. Cisco IOS-XE Leaf Switch Configuration

! Step 1: Configure Tenant VRF and associate Layer 3 VNI (L3VNI)
Leaf-1(config)# vrf definition TENANT-A
Leaf-1(config-vrf)# rd 10.255.1.1:50001
Leaf-1(config-vrf)# address-family ipv4
Leaf-1(config-vrf-af)# route-target export 65000:50001
Leaf-1(config-vrf-af)# route-target import 65000:50001
Leaf-1(config-vrf-af)# exit-address-family

! Step 2: Configure Layer 2 EVPN Instance (EVI) for VLAN 10
Leaf-1(config)# l2vpn evpn
Leaf-1(config-evpn)# replication-type static
Leaf-1(config-evpn)# l2vpn evpn instance 10 vlan-based
Leaf-1(config-evpn-evi)# encapsulation vxlan
Leaf-1(config-evpn-evi)# route-target export 65000:10010
Leaf-1(config-evpn-evi)# route-target import 65000:10010

! Step 3: Configure the NVE (Network Virtualization Edge) Interface
Leaf-1(config)# interface nve1
Leaf-1(config-if-nve)# no shutdown
Leaf-1(config-if-nve)# source-interface Loopback0
Leaf-1(config-if-nve)# host-reachability protocol bgp
Leaf-1(config-if-nve)# member vni 10010
Leaf-1(config-if-nve-vni)# ingress-replication
Leaf-1(config-if-nve)# member vni 50001 vrf TENANT-A

! Step 4: Configure Distributed Anycast Gateway on SVI 10
Leaf-1(config)# interface Vlan10
Leaf-1(config-if)# vrf forwarding TENANT-A
Leaf-1(config-if)# ip address 10.1.10.1 255.255.255.0
Leaf-1(config-if)# fabric forwarding mode-anycast
Leaf-1(config-if)# no shutdown

! Step 5: Activate BGP L2VPN EVPN Neighbor Sessions
Leaf-1(config)# router bgp 65000
Leaf-1(config-router)# neighbor 10.255.255.1 remote-as 65000
Leaf-1(config-router)# neighbor 10.255.255.1 update-source Loopback0
Leaf-1(config-router)# address-family l2vpn evpn
Leaf-1(config-router-af)# neighbor 10.255.255.1 activate
Leaf-1(config-router-af)# neighbor 10.255.255.1 send-community both

2. Verification Commands & Operational Output

! Verify NVE interface status and VNI mapping
Leaf-1# show nve vni
Interface  VNI        Type  Mode  VLAN  VRF        State
---------  ---------  ----  ----  ----  ---------  -----
nve1       10010      L2CP  BGP   10    n/a        UP
nve1       50001      L3CP  BGP   n/a   TENANT-A   UP

! Check active VTEP peer discovery
Leaf-1# show nve peers
Interface  VTEP IP          State  LearnType  Uptime
---------  ---------------  -----  ---------  --------
nve1       10.255.1.2       UP     BGP        04:22:18
nve1       10.255.1.3       UP     BGP        04:22:18

! Inspect BGP EVPN Route Type 2 (MAC/IP) table
Leaf-1# show bgp l2vpn evpn route-type 2
Route Distinguisher: 10.255.1.1:10010
 *> [2][10.255.1.1:10010][0][48][0000aaaa0001][32][10.1.10.11]/24
     Path: Local, Next-Hop: 10.255.1.1, VNI: 10010/50001
 *>i[2][10.255.1.2:10010][0][48][0000aaaa0002][32][10.1.10.12]/24
     Path: Internal, Next-Hop: 10.255.1.2, VNI: 10010/50001

! Verify local ARP suppression cache on the leaf
Leaf-1# show ip arp suppression-cache
IP Address       MAC Address      VNI    Source  Uptime
---------------  ---------------  -----  ------  --------
10.1.10.12       0000.aaaa.0002   10010  EVPN    03:15:02