4.3 Edge Clusters, Tier-0/Tier-1 Gateway Routing, & Uplink Connectivity

4.3 Edge Clusters, Tier-0/Tier-1 Gateway Routing, & Uplink Connectivity

Executive Overview: Scalable routing in VMware Cloud Foundation 9.0 is delivered through a sophisticated two-tier hierarchical gateway architecture that enforces strict operational boundaries between cloud infrastructure providers and multi-tenant consumers. By partitioning routing into Distributed Routers (DR) that execute in the hypervisor kernel and centralized Services Routers (SR) hosted on Edge Clusters, NSX eliminates traditional network hair-pinning while delivering enterprise-grade stateful services. For the VCP-VCF (2V0-17.25) exam, candidates must master Tier-0 and Tier-1 gateway relationships, Edge node sizing profiles, High Availability modes (Active-Active ECMP vs. Active-Standby), dynamic BGP routing, Bidirectional Forwarding Detection (BFD), and route redistribution controls.

---

Two-Tier Routing Architecture (Tier-0 and Tier-1)

Traditional enterprise datacenters rely on monolithic physical core routers or centralized virtual router appliances to route traffic between subnets. When virtual machines on different VLANs communicate, their traffic must leave the physical server, traverse top-of-rack switches, enter a physical router interface, be routed, and re-enter the server. This phenomenon—known as traffic hair-pinning or tromboning—consumes massive physical bandwidth and introduces unpredictable latency.

NSX eliminates hair-pinning through a modular, two-tier logical routing architecture:

┌─────────────────────────────────────────────────────────────────────────────────┐
│                     TIER-0 GATEWAY (PROVIDER / INFRASTRUCTURE)                  │
│  - Managed by Cloud Administrators / Enterprise Network Engineering             │
│  - Northbound Connectivity: eBGP Peering to Physical ToR Switches via Uplinks   │
│  - Centralized Services: NAT, Stateful Gateway Firewall, ECMP (up to 8 paths)   │
└────────────────────────────────────────┬────────────────────────────────────────┘
                                         │ Router Link Port (/31 on 100.64.0.0/16)
┌────────────────────────────────────────▼────────────────────────────────────────┐
│                     TIER-1 GATEWAY (TENANT / APPLICATION WORKLOAD)              │
│  - Managed by Tenant Administrators, DevOps Teams, or Line-of-Business Owners   │
│  - Connects Downstream to Workload Overlay Segments (Web, App, DB)              │
│  - East-West Routing between Tenant Segments; Advertises Subnets to Tier-0      │
└─────────────────────────────────────────────────────────────────────────────────┘

1. The Tier-0 Gateway (Provider Boundary)

The Tier-0 Gateway acts as the top-tier border router connecting the virtual software-defined network to the physical corporate datacenter network:

• Ownership: Governed by cloud infrastructure administrators and enterprise network teams.
• External Peering: Establishes dynamic routing adjacencies (External BGP or OSPF) or static routes with physical Top-of-Rack (ToR) core switches over VLAN-backed uplink segments.
• Enterprise Capabilities: Supports active-active Equal-Cost Multi-Path (ECMP) routing across up to 8 paths, stateful gateway firewalling, enterprise NAT, and route redistribution maps.

2. The Tier-1 Gateway (Tenant Boundary)

The Tier-1 Gateway provides tenant-level isolation, acting as the local default gateway for workload virtual machines:

• Ownership: Delegated to application teams, project owners, or business units.
• Downstream Connectivity: Connects directly to tenant overlay segments (e.g., Web-Segment, App-Segment, DB-Segment), handling east-west routing between application tiers.
• Northbound Uplink: Connects northbound to a parent Tier-0 Gateway. A single Tier-0 gateway can host dozens of independent Tier-1 gateways, providing complete multi-tenant segmentation.

3. The Automated Internal Transit Link

When a Tier-1 gateway is linked to a Tier-0 gateway, NSX automatically creates an internal point-to-point transit connection:

• Router Link Port: NSX automatically carves a /31 subnet out of the reserved carrier-grade NAT / link-local address space (100.64.0.0/16 per RFC 6598).
• Zero-Touch Configuration: Administrators do not configure IP addresses or dynamic routing protocols (such as OSPF or BGP) between Tier-0 and Tier-1 gateways. The NSX management plane automatically programs static routes and manages transit reachability through declarative Route Advertisement.

---

Distributed Router (DR) vs. Services Router (SR) Components

One of the most heavily tested architectural concepts on the VCP-VCF exam is the internal composition of NSX logical gateways into Distributed Routers (DR) and Services Routers (SR).

┌─────────────────────────────────────────────────────────────────────────────────┐
│            DISTRIBUTED ROUTER (DR)              │     SERVICES ROUTER (SR)      │
├─────────────────────────────────────────────────┼───────────────────────────────┤
│ • Runs in the ESXi VMkernel on EVERY transport  │ • Instantiated ONLY on Edge   │
│   node (Hosts and Edges).                       │   Transport Nodes.            │
│ • Executes East-West routing directly at the    │ • Executes centralized        │
│   source hypervisor at line rate.               │   stateful services.          │
│ • Zero hops, zero hair-pinning, zero latency.   │ • Required for: BGP peering,  │
│ • Exists on EVERY Tier-0 and Tier-1 gateway.    │   NAT, Stateful GFW, VPN, LB. │
└─────────────────────────────────────────────────┴───────────────────────────────┘

The Distributed Router (DR) Mechanics

A Distributed Router is an in-kernel routing instance that is instantiated across every single Host Transport Node and Edge Transport Node participating in the transport zone:

• Local Line-Rate Routing: When Virtual Machine A (192.168.10.5) sends a packet to Virtual Machine B (192.168.20.10) residing on the same physical ESXi host, the packet is routed directly in the ESXi VMkernel by the DR. The packet never touches a physical network interface, never crosses a top-of-rack switch, and never traverses an Edge node.
• Distributed Inter-Host Routing: If VM-A and VM-B reside on different ESXi hosts, the DR on the source host routes the packet from Subnet 10 to Subnet 20 in-kernel, encapsulates the routed packet into Geneve, and transmits it directly across the physical underlay to the destination host's TEP. Routing is always performed at the source hypervisor.

The Services Router (SR) Mechanics

A Services Router is a centralized routing instance that is instantiated exclusively on Edge Transport Nodes within an assigned Edge Cluster:

• Why is an SR Needed? Certain enterprise network services cannot be distributed across hundreds of hypervisors because they require centralized state tracking, session tables, or single-point physical peering. An SR is automatically instantiated when any of the following services are enabled on a gateway:
  1. Dynamic routing peering (BGP / OSPF) to physical switches.
  2. Source NAT (SNAT) or Destination NAT (DNAT).
  3. Stateful Gateway Firewalling and URL Filtering.
  4. Edge-based DHCP Server.
  5. Site-to-Site IPSec VPN and Layer 2 VPN.
  6. Centralized Load Balancing.

The Distributed-Only Tier-1 Architecture: If a Tier-1 gateway is deployed solely to provide East-West routing between tenant segments and does NOT have stateful services (such as NAT or stateful firewall) enabled, no SR is instantiated on the Edge cluster. The Tier-1 gateway exists purely as a Distributed Router (DR) inside the ESXi hypervisors, consuming zero Edge node CPU and memory resources!

---

Edge Node Form Factors, Deployment Models, and Sizing Profiles

Edge Transport Nodes provide the physical interfaces and compute capacity required to host gateway Services Routers (SRs).

Form Factors: Virtual Appliance (VM) vs. Bare-Metal

Edge Virtual Machine Sizing Profiles

When deploying Edge VMs via SDDC Manager or NSX Manager, administrators select an appliance sizing profile based on workload throughput and service concurrency requirements:

---

Edge Clusters & High Availability (HA) Modes

An Edge Cluster is a logical management grouping of up to 16 Edge Transport Nodes that pool compute, memory, and networking capacity to host Tier-0 and Tier-1 Services Routers.

ACTIVE-STANDBY HA MODE:                          ACTIVE-ACTIVE ECMP HA MODE (UP TO 8 NODES):
┌──────────────────────────────────────────┐     ┌──────────────────────────────────────────┐
│         Physical Network Core            │     │         Physical Network Core            │
│         ┌──────────────┐                 │     │         ┌──────────────┐                 │
│         │ Physical ToR │                 │     │         │ Physical ToR │                 │
│         └──────┬───────┘                 │     │         └──┬────┬────┬─┘                 │
│       Active   │   Standby (Idle)        │     │  ECMP Hash │    │    │ (Up to 8 Paths)   │
│         ┌──────┴───────┐                 │     │    ┌───────┘    │    └───────┐           │
│         ▼              ▼                 │     │    ▼            ▼            ▼           │
│  ┌─────────────┐ ┌─────────────┐         │     │ ┌─────────┐ ┌─────────┐ ┌─────────┐      │
│  │ Edge Node 1 │ │ Edge Node 2 │         │     │ │ Edge 1  │ │ Edge 2  │ │ Edge N  │      │
│  │ (Primary SR)│ │(Standby SR) │         │     │ │(Active) │ │(Active) │ │(Active) │      │
│  └──────┬──────┘ └──────┬──────┘         │     │ └─────────┘ └─────────┘ └─────────┘      │
│         └──State Sync───┘                │     │ *STATELESS ROUTING & MASSIVE SCALE*      │
│ *MANDATORY FOR STATEFUL SERVICES*        │     │ *NO STATEFUL NAT OR STATEFUL FIREWALL*   │
└──────────────────────────────────────────┘     └──────────────────────────────────────────┘

1. Active-Standby HA Mode

• Mechanics: One Edge node is elected as the Primary Active forwarder for the gateway SR, while a secondary Edge node is designated as the Standby forwarder.
• State Synchronization: The active Edge node continuously synchronizes connection tracking tables, firewall state, and NAT sessions to the standby node across an internal management link.
• Failover: If the primary Edge node fails, the standby node immediately assumes packet forwarding without dropping active TCP connections.
• Mandatory Use Case: Active-Standby mode is strictly mandatory when deploying stateful services that require symmetric packet flows, such as Stateful Gateway Firewalls, Stateful NAT, and IPSec VPN.

2. Active-Active HA Mode with Equal-Cost Multi-Path (ECMP)

• Mechanics: Up to 8 Edge Transport Nodes actively forward north-south traffic simultaneously.
• ECMP Hashing: Upstream physical Top-of-Rack switches use Layer 3/4 ECMP hashing to distribute incoming network traffic across all active Edge nodes in parallel.
• Linear Throughput Scaling: Provides massive aggregate bandwidth (e.g., eight Large Edge VMs each forwarding 25 Gbps deliver up to 200 Gbps of north-south throughput).
• Architectural Constraint: Because different packets in the same TCP session may traverse different Edge nodes (asymmetric routing), stateful services like stateful NAT and stateful firewall are unsupported on Active-Active Tier-0 gateways unless reflexive session routing or distributed state offloading is implemented. Active-Active Tier-0 gateways are typically paired with stateless firewall rules, offloading stateful services to downstream Active-Standby Tier-1 gateways.

---

Dynamic Routing (BGP), Route Redistribution, & Route Leaking

North-South routing between the NSX software-defined fabric and the physical datacenter relies on external Border Gateway Protocol (eBGP).

External BGP (eBGP) Peering Architecture

• Uplink Segments: The Tier-0 gateway establishes eBGP peering sessions with physical ToR switches over dedicated VLAN-backed segments. Redundant uplinks (e.g., Uplink 1 on VLAN 101 to ToR-A, and Uplink 2 on VLAN 102 to ToR-B) ensure physical link and switch redundancy.
• Autonomous System Numbers (ASNs): Administrators assign 2-byte (e.g., 65001) or 4-byte private ASNs to the Tier-0 gateway and physical ToR switches.
• Equal-Cost Multi-Pathing: When peering with multiple physical routers, Tier-0 enables BGP Multi-Path to utilize up to 8 parallel ECMP paths for egress traffic.

Bidirectional Forwarding Detection (BFD)

Standard BGP hold-down timers are typically 90 to 180 seconds—an unacceptable failover delay in a high-availability private cloud. If a physical link or switch silently fails, BGP will black-hole traffic for several minutes before tearing down the neighbor adjacency.

To achieve sub-second failover, NSX mandates enabling Bidirectional Forwarding Detection (BFD) on all BGP peering sessions:

• BFD transmits high-frequency micro-heartbeats (e.g., 300 ms transmit/receive interval with a detect multiplier of 3).
• If three consecutive heartbeats are missed (900 ms), BFD instantly declares the path dead and notifies the BGP routing engine.
• BGP immediately removes the failed path from its forwarding table and redirects traffic to alternate ECMP paths in under one second.

Route Leaking & Redistribution Mechanics

• Tier-1 to Tier-0 Route Advertisement: Tier-1 gateways do not run dynamic routing protocols to Tier-0. Instead, administrators toggle declarative check-boxes under Route Advertisement:
  • Tier-1 Connected Segments: Advertises tenant workload subnets.
  • Tier-1 NAT IPs: Advertises public Source NAT IP addresses.
  • Tier-1 LB VIPs: Advertises virtual load balancer endpoints. The NSX Management Plane translates these selections into static routes automatically programmed into the Tier-0 routing table.
• Tier-0 Route Redistribution to Physical Underlay: Tier-0 must advertise tenant subnets to physical datacenter routers via BGP. Administrators configure Route Redistribution Rules specifying which route sources (Tier-1 Connected Subnets, Tier-0 NAT Pools) are injected into BGP.
• Route Maps and Prefix Lists: Crucial exam best practice: administrators must attach Route Maps to BGP neighbors to filter route advertisements. Prefix lists ensure that only authorized tenant subnets are announced, strictly preventing internal transit networks (100.64.0.0/16) or non-routable management subnets from leaking into the corporate enterprise routing table.

---

Exam Watch: Key Scenarios and Candidate Traps

[!IMPORTANT] The East-West Routing Path Trap: An exam scenario describes two virtual machines connected to different segments on the same Tier-1 gateway, running on the same ESXi host. The question asks which Edge node the traffic traverses to be routed. The traffic NEVER traverses an Edge node. East-west routing between logical segments is performed directly within the ESXi VMkernel by the Distributed Router (DR) in memory.

[!WARNING] Active-Active ECMP with Stateful Services: If an exam question specifies that a Tier-0 gateway must provide stateful Layer 4-7 firewall inspection, stateful NAT, or IPSec VPN, selecting Active-Active ECMP mode is incorrect. Active-Standby High Availability is strictly required to guarantee stateful session synchronization.

[!NOTE] No BGP Between Tier-0 and Tier-1: Watch out for distractors proposing to configure an eBGP or iBGP peering session between a Tier-1 gateway and a Tier-0 gateway. Tier-1 gateways do not support BGP peering to Tier-0; communication between them is established via the internal transit link and declarative Route Advertisement.