4.2 NSX Virtual Network Fabrics, Transport Nodes, & Geneve Overlays

4.2 NSX Virtual Network Fabrics, Transport Nodes, & Geneve Overlays

Executive Overview: The virtual network fabric in VMware Cloud Foundation 9.0 represents the programmatic substrate that interconnects heterogeneous workloads across physical server clusters. Modern VCF networking achieves unprecedented efficiency by converging NSX virtual switching directly onto the foundational vSphere Distributed Switch (VDS), retiring legacy N-VDS switch implementations. By defining logical boundaries through Transport Zones and encapsulating tenant frames within Geneve tunnels, NSX enables multi-tier network topologies to span across physical Layer 3 boundaries without reconfiguring physical hardware. This section analyzes the packet encapsulation mechanics of Geneve, multi-TEP teaming architectures, and the cluster-wide automation delivered by Transport Node Profiles (TNPs).

---

The Converged vSphere Distributed Switch (VDS) Architecture

A pivotal architectural evolution in modern VMware Cloud Foundation is the convergence of the NSX virtual switch onto the native vSphere Distributed Switch (VDS) (introduced in vSphere 7 and standardized across vSphere 8 and 9.0).

Legacy N-VDS vs. Converged VDS

In earlier NSX-T releases, preparing an ESXi host as a transport node required installing a proprietary, standalone switch called the N-VDS (NSX Virtual Distributed Switch). The N-VDS operated in parallel with, but completely separate from, vCenter Server's native VDS:

• Physical NIC Partitioning: Because an ESXi physical network adapter (pNIC) could only bind to one virtual switch at a time, a host with four physical NICs had to dedicate two NICs to the vSphere VDS (for Management, vMotion, and vSAN traffic) and two separate NICs to the N-VDS (for NSX overlay traffic). Hosts equipped with only two physical NICs faced severe architectural compromises.
• Management Fragmentation: Administrators were forced to maintain dual management planes: vCenter managed the VDS, while NSX Manager managed the N-VDS. Troubleshooting host networking required querying multiple interfaces and CLIs.

LEGACY N-VDS MODEL:                              MODERN CONVERGED VDS MODEL:
┌──────────────────────────────────────────┐     ┌──────────────────────────────────────────┐
│         ESXi Host Physical NICs          │     │         ESXi Host Physical NICs          │
│  vmnic0    vmnic1    vmnic2    vmnic3    │     │           vmnic0          vmnic1         │
│    │         │         │         │       │     │             │               │            │
│ ┌──┴─────────┴───┐  ┌──┴─────────┴───┐   │     │ ┌───────────┴───────────────┴──────────┐ │
│ │   vSphere VDS  │  │    NSX N-VDS   │   │     │ │   Converged vSphere VDS (VCF 9.0)    │ │
│ │ (Mgmt, vSAN,   │  │ (Geneve Overlay│   │     │ │ (Mgmt, vSAN, vMotion, NSX Overlays,  │ │
│ │   vMotion)     │  │   & VLANs)     │   │     │ │  Edge Uplinks on Unified Uplinks)    │ │
│ └────────────────┘  └────────────────┘   │     │ └──────────────────────────────────────┘ │
└──────────────────────────────────────────┘     └──────────────────────────────────────────┘

Architectural Superiority of Converged VDS

In VCF 9.0, NSX operates directly inside the standard vSphere Distributed Switch (VDS version 7.0/8.0/9.0):

• Unified Uplink Architecture: A single pair of high-speed physical adapters (e.g., dual 25 GbE or 100 GbE interfaces) carries all infrastructure traffic (ESXi management, vSAN storage, vMotion) alongside NSX overlay and VLAN segments.
• Centralized Operational Visibility: All port groups and NSX segments appear natively in the vSphere Client interface. Network administrators gain unified visibility into link health, LLDP/CDP neighbor discovery, and performance metrics across both virtual and infrastructure traffic.
• Enhanced Network Offloads: Converged VDS natively integrates with modern Data Processing Units (DPUs / SmartNICs) via the vSphere Distributed Services Engine (DSE), offloading NSX virtual switching, Geneve encapsulation, and distributed firewall processing directly to hardware accelerators.

---

Transport Zones: Scope, Types, and Architectural Roles

A Transport Zone (TZ) defines the reach, scope, and span of logical networks across transport nodes. Transport zones dictate which ESXi hosts and Edge nodes can participate in specific virtual segments.

┌─────────────────────────────────────────────────────────────────────────────────┐
│                     OVERLAY TRANSPORT ZONE (GENEVE ENCAPSULATED)                │
│  - Workload Segments (Web, App, DB) span across Host A, Host B, and Edge Nodes  │
│  - Uses Virtual Network Identifiers (VNIs: 50000 - 16777215)                    │
│  - Decoupled from physical switch VLAN databases                                │
├─────────────────────────────────────────────────────────────────────────────────┤
│                       VLAN TRANSPORT ZONE (802.1Q TAGGED)                       │
│  - Directly maps logical segments to physical underlay VLAN IDs (e.g., VLAN 100)│
│  - Used for Edge Uplink BGP Peering and physical workload bridging             │
│  - NO Geneve encapsulation overhead; standard Ethernet framing                  │
└─────────────────────────────────────────────────────────────────────────────────┘

1. Overlay Transport Zones

• Encapsulation: Segments created within an Overlay Transport Zone utilize Geneve encapsulation. Tenant packets are wrapped with an outer IP/UDP header and transported across the physical routed underlay.
• Logical Span: Overlay segments can span across multiple clusters, physical racks, and Layer 3 network boundaries. A virtual machine on Host A in Rack 1 can communicate with a virtual machine on Host B in Rack 5 on the same Layer 2 overlay subnet without any VLAN configuration on intermediate physical switches.
• Virtual Network Identifier (VNI): Each overlay segment is assigned a unique 24-bit VNI (supporting over 16 million unique segments), completely eliminating the 4,094 VLAN scalability limit of physical IEEE 802.1Q networks.

2. VLAN Transport Zones

• Native Framing: Segments created within a VLAN Transport Zone do not encapsulate packets with Geneve. Traffic traversing a VLAN segment is tagged with standard IEEE 802.1Q VLAN headers matching the specified VLAN ID.
• Primary Use Cases:
  1. Edge Uplinks: Connecting Tier-0 Gateway Services Routers running on Edge nodes to physical Top-of-Rack switches for external dynamic BGP peering.
  2. Physical Workload Integration (Bridging): Connecting virtual machines directly to legacy bare-metal database servers or physical appliances residing on existing datacenter VLANs without routing.

Multi-Zone Membership Rule: A single Transport Node can belong to multiple transport zones simultaneously. In a standard VCF architecture, ESXi Host Transport Nodes join the default Overlay Transport Zone (and an optional VLAN Transport Zone for bridging), while Edge Transport Nodes must belong to both an Overlay Transport Zone (to exchange traffic with ESXi host TEPs) and a VLAN Transport Zone (to establish northbound BGP uplinks to physical routers).

---

Geneve Encapsulation Protocol Deep-Dive

The IETF standard Geneve (Generic Network Virtualization Encapsulation - RFC 8926) is the foundational wire protocol for all NSX overlay communication.

Geneve vs. Legacy Overlay Protocols (VXLAN and NVGRE)

In earlier generations of network virtualization, protocols such as VXLAN (RFC 7348) and NVGRE (RFC 7637) were widely deployed. However, VXLAN suffered from a rigid, non-extensible 8-byte fixed header that could only transmit a 24-bit VNI. It lacked the ability to carry dynamic metadata.

Geneve solves this limitation by introducing a modular Type-Length-Value (TLV) architecture:

• Dynamic Contextual Metadata: Geneve allows NSX to insert variable-length TLV option fields into the packet header. NSX uses these TLVs to carry tenant identifiers, security group tags, ingress interface metadata, flow-tracing telemetry, and distributed firewall context directly within the packet wire format.
• Extensible Architecture: As new SDN capabilities, encryption standards, or telemetry requirements emerge, Geneve accommodates them without requiring a fundamental protocol redesign or physical switch firmware replacement.

Detailed Geneve Packet Format

┌─────────────────────────────────────────────────────────────────────────────────┐
│                            GENEVE ENCAPSULATED PACKET                           │
├─────────────────┬──────────────┬──────────────┬──────────────┬──────────────────┤
│ Outer Ethernet  │ Outer IPv4   │ Outer UDP    │ Geneve Base  │ Original Inner   │
│ Header (14 B)   │ Header (20 B)│ Header (8 B) │ Header (8 B) │ Ethernet Frame   │
│ Src/Dst MAC of  │ Src/Dst IP of│ Dst Port:    │ + Variable   │ (VM MACs, IP,    │
│ Next-Hop Switch │ Host TEPs    │ 6081         │ TLV Options  │ TCP/UDP Payload) │
└─────────────────┴──────────────┴──────────────┴──────────────┴──────────────────┘

1. Outer Ethernet Header (14 Bytes): Contains the source MAC address of the transmitting host's physical network adapter (or next-hop router) and the destination MAC address of the adjacent physical switch port.
2. Outer IPv4 Header (20 Bytes): Contains the source IP address of the transmitting host's Tunnel Endpoint (TEP) and the destination IP address of the receiving host's TEP. Physical underlay routers inspect only this header to route the packet across the physical IP fabric.
3. Outer UDP Header (8 Bytes):
   • Destination Port: Fixed at UDP port 6081 (the official IETF assigned port for Geneve).
   • Source Port (Entropy Hash): The source port is dynamically calculated as an ephemeral hash (range 49152–65535) of the inner tenant packet's 5-tuple (source IP, destination IP, protocol, source port, destination port). This is an ingenious architectural feature: because physical leaf-spine switches use the Layer 4 UDP source port in their Equal-Cost Multi-Path (ECMP) hashing algorithms, different virtual machine traffic flows automatically spread evenly across all physical links in the leaf-spine fabric without the physical switches needing to inspect the inner packet!
4. Geneve Header (8 Bytes Base + Variable TLVs): Contains protocol control flags, the Protocol Type field (0x6558 for Transparent Ethernet Bridging), the 24-bit Virtual Network Identifier (VNI), and modular TLV options.
5. Inner Ethernet Frame: The complete, untouched original frame transmitted by the virtual machine, including the guest OS MAC addresses, IP headers, and application payload.

---

Tunnel Endpoints (TEPs) & Multi-TEP Teaming Mechanics

A Tunnel Endpoint (TEP) is a specialized VMkernel network interface (vmk10, vmk11 on ESXi) responsible for originating and terminating Geneve overlay encapsulation.

The Step-by-Step TEP-to-TEP Encapsulation Walkthrough

VM-A (192.168.10.11) on Host 1 ───────────────> VM-B (192.168.10.22) on Host 2
  1. VM-A transmits standard Ethernet frame to local vSwitch port.
  2. In-kernel DFW inspects packet; packet is permitted.
  3. Local Control Plane (netcpa) looks up VM-B MAC in local forwarding table.
  4. Table maps VM-B MAC to Remote TEP IP (172.16.20.52) on Host 2 with VNI 50010.
  5. Host 1 TEP (vmk10: 172.16.10.51) encapsulates frame in Geneve (UDP 6081).
  6. Physical leaf-spine underlay routes outer IP packet via standard L3 ECMP.
  7. Host 2 TEP receives packet, strips Geneve headers, validates VNI 50010.
  8. In-kernel DFW applies ingress inspection; frame delivered to VM-B vNIC.

Multi-TEP Architecture on ESXi Host Transport Nodes

To eliminate single points of failure and maximize network bandwidth, modern ESXi Host Transport Nodes deploy with a Multi-TEP configuration:

• Two dedicated VMkernel TEP interfaces are created (vmk10 and vmk11).
• Each TEP is assigned an independent IP address from a centralized NSX TEP IP Pool.
• Each TEP binds to an independent physical uplink (e.g., vmk10 to physical uplink 1, vmk11 to physical uplink 2).

Uplink Teaming Policies for Multi-TEP

In an NSX Uplink Profile, administrators configure how TEP traffic is balanced across physical uplinks:

Named Teaming Policies for Multi-Uplink Pinning

By default, all segments inherit the cluster's default teaming policy. However, complex VCF architectures frequently require pinning specific traffic types to dedicated physical uplinks. For example, Edge Node VM uplinks connecting to upstream physical routers often require deterministic separation: Edge Uplink 1 must traverse physical vmnic0 (to Leaf-A), while Edge Uplink 2 must traverse physical vmnic1 (to Leaf-B).

To achieve this, administrators define Named Teaming Policies within the Uplink Profile. A named teaming policy defines an explicit failover or active uplink mapping (e.g., Uplink1-Only and Uplink2-Only). When creating VLAN-backed Edge uplink segments, the administrator binds each segment to its respective Named Teaming Policy, guaranteeing deterministic physical path selection.

---

Transport Node Profiles (TNP): Automated Cluster-Wide Preparation

Manually preparing individual ESXi hosts as transport nodes—assigning switches, binding uplink profiles, attaching transport zones, and allocating IP subnets—is tedious and susceptible to human error. In VMware Cloud Foundation 9.0, host preparation is automated using Transport Node Profiles (TNPs).

┌─────────────────────────────────────────────────────────────────────────────────┐
│                     TRANSPORT NODE PROFILE (TNP) SPECIFICATION                  │
├─────────────────────────────────────────────────────────────────────────────────┤
│  1. vSphere Distributed Switch (VDS): Converged-VDS-01                          │
│  2. Transport Zones: Default-Overlay-TZ, Enterprise-VLAN-TZ                     │
│  3. Uplink Profile: Multi-TEP Load-Balance-Source-MAC (MTU 9000)                │
│  4. TEP IP Allocation: Centralized NSX IP Pool (Subnet: 172.16.10.0/24)         │
└────────────────────────────────────────┬────────────────────────────────────────┘
                                         │ Applied at vSphere Cluster Object
                                         ▼
┌─────────────────────────────────────────────────────────────────────────────────┐
│                          AUTOMATED CLUSTER REMEDIATION                          │
│  - Host 01: Installs NSX VIBs, creates vmk10/vmk11, joins TZs -> Healthy Ready   │
│  - Host 02: Installs NSX VIBs, creates vmk10/vmk11, joins TZs -> Healthy Ready   │
│  - Host N:  Newly commissioned hosts automatically inherit TNP during scale-out │
└─────────────────────────────────────────────────────────────────────────────────┘

The Declarative Power of TNPs

A Transport Node Profile is a declarative configuration template that encapsulates:

1. The target vSphere Distributed Switch (VDS) name and switch UUID.
2. The Transport Zones (Overlay and VLAN) the hosts must join.
3. The Uplink Profile defining teaming policy, physical uplink mappings (vmnic0 -> uplink-1, vmnic1 -> uplink-2), and MTU sizing.
4. The TEP IP assignment method (a designated static IP Pool managed by NSX).

When a TNP is attached to a vSphere Cluster object in SDDC Manager or NSX Manager, the system automatically remediates all hosts in the cluster in parallel. Furthermore, when SDDC Manager commissions and adds a new ESXi host to expand cluster capacity (Day-1 / Day-2 operations), the new host automatically inherits the cluster's TNP. The host is installed with NSX kernel modules, configured with VMkernel TEP interfaces, and added to the transport zones without administrative intervention.

---

Exam Watch: Key Scenarios and Candidate Traps

[!IMPORTANT] The Geneve Port Number on the Exam: Memorize that Geneve utilizes UDP destination port 6081. Exam questions frequently list legacy VXLAN (UDP 4789) or L2TP (UDP 1701) as distractors. If an access control list on physical switches blocks UDP 6081, all Geneve overlay communication will fail immediately.

[!WARNING] VLAN Transport Zones Do NOT Encapsulate: A common exam trap asks what encapsulation protocol is used when a virtual machine connects to a segment attached to a VLAN Transport Zone. The answer is NONE. Packets on VLAN segments retain standard IEEE 802.1Q tags and are forwarded directly to the physical switch fabric without Geneve outer headers.

[!NOTE] Multi-TEP Active-Active Without LACP: Candidates often assume that achieving active-active multi-uplink bandwidth aggregation requires configuring LACP on physical switches. In NSX, configuring the Load Balance Source MAC teaming policy on a multi-TEP host provides active-active hashing across physical uplinks natively, requiring zero LACP or port-channel configuration on physical switches.