4.1 VCF Network Architecture & NSX Core Components

4.1 VCF Network Architecture & NSX Core Components

Executive Overview: Software-defined networking is the critical architectural engine that enables VMware Cloud Foundation (VCF) 9.0 to deliver instantaneous workload provisioning, dynamic multi-tenancy, and microsegmented zero-trust security. Powered by VMware NSX, network virtualization completely decouples virtual machines and container workloads from the physical cabling, VLAN limitations, and static hardware switches of the underlying datacenter. For the VCP-VCF (2V0-17.25) exam, candidates must master the distinct responsibilities of the four architectural planes, the clustering and quorum mechanics of NSX Manager appliances, the operational roles of Host versus Edge Transport Nodes, and the uncompromising physical underlay requirements—most notably leaf-spine IP fabrics and jumbo frame MTU engineering.

---

Full-Stack VCF 9.0 Networking Foundations

In traditional datacenter networking, infrastructure agility is severely throttled by physical network constraints. Provisioning a new multi-tier application requires manual configuration tickets routed to network engineers to create VLANs across access, aggregation, and core switches, update Spanning Tree Protocol (STP) topologies, reconfigure physical router interfaces, and insert hardware firewall rules. This manual workflow creates weeks of lead time and introduces human configuration errors and configuration drift across environments.

VMware Cloud Foundation 9.0 eliminates these physical dependencies by abstracting the entire network and security fabric into software through VMware NSX. In a VCF private cloud:

• Automated Lifecycle Deployment: SDDC Manager automates the deployment, clustering, and initial configuration of the NSX management domain cluster during Day-0 bring-up via the VCF Installer, and provisions dedicated or shared NSX instances when creating Virtual Infrastructure (VI) Workload Domains during Day-1 operations.
• Universal Abstraction: Layer 2 switching, Layer 3 routing, Layer 4-7 firewalling, load balancing, and network address translation (NAT) are synthesized purely in software. Workloads can be provisioned, re-addressed, migrated across server racks, or deleted programmatically in seconds via declarative REST APIs, VMware Cloud Templates, or Terraform without requiring a single physical switch configuration change.
• Unified Compute and Network Convergence: NSX integrates natively with the ESXi hypervisor kernel and the vSphere Distributed Switch (VDS), ensuring that networking and security policies automatically accompany virtual machines during live vMotion migrations across physical clusters.

---

The Four Architectural Planes of VMware NSX

A central design principle of VMware NSX is the strict mathematical and operational decoupling of its functional planes. Understanding the boundaries, daemons, and communication channels between these four planes is essential for both enterprise troubleshooting and scoring highly on the VCP-VCF exam.

┌─────────────────────────────────────────────────────────────────────────────────┐
│                            MANAGEMENT PLANE (MP)                                │
│  - NSX Manager UI, REST API (/policy/api/v1/), SDDC Manager Orchestration       │
│  - Corfu Distributed Log: User Configuration & Policy Desired State             │
└────────────────────────────────────────┬────────────────────────────────────────┘
                                         │ Internal RPC
┌────────────────────────────────────────▼────────────────────────────────────────┐
│                         CENTRAL CONTROL PLANE (CCP)                             │
│  - Distributed Service (cloudnet) on NSX Manager Nodes                          │
│  - Calculates Ephemeral Runtime State: MAC/IP/VNI Tables, Distributed Routing   │
│  - NEVER touches, inspects, or forwards user data packets                       │
└────────────────────────────────────────┬────────────────────────────────────────┘
                                         │ TLS (TCP 1234)
┌────────────────────────────────────────▼────────────────────────────────────────┐
│                          LOCAL CONTROL PLANE (LCP)                              │
│  - Local Daemon (netcpa) running on every Host and Edge Transport Node          │
│  - Translates CCP State into Local Data Plane Forwarding Entries                │
│  - Monitors Local VDS Ports, Link Status, and VM Lifecycle Events               │
└────────────────────────────────────────┬────────────────────────────────────────┘
                                         │ Kernel IPC
┌────────────────────────────────────────▼────────────────────────────────────────┐
│                              DATA PLANE (DP)                                    │
│  - Host Transport Nodes: ESXi Kernel Modules (nsx-vdl2, DFW, Distributed Router)│
│  - Edge Transport Nodes: DPDK High-Performance Fast-Path Forwarding Engine     │
│  - Encapsulates Geneve Overlay (UDP 6081) & Enforces Line-Rate Microsegmentation│
└─────────────────────────────────────────────────────────────────────────────────┘

1. The Management Plane (MP)

The Management Plane represents the single point of entry for configuring, managing, and monitoring the NSX environment.

• Access Methods: Administrators and automation engines interact with the MP via the HTML5 NSX Manager user interface, comprehensive RESTful APIs, and SDDC Manager lifecycle workflows.
• Declarative Policy Architecture: Modern NSX standardizes on the Policy API (/policy/api/v1/), which operates declaratively. Administrators declare the desired end state (e.g., "Tier-1 Gateway with Segment App-Tier on subnet 192.168.10.0/24"), and the policy engine automatically calculates and translates the underlying imperative objects.
• Corfu Distributed Datastore: The MP relies on Corfu, a cloud-scale, distributed transaction engine and replicated append-only log running in-memory across the NSX Manager cluster nodes. Corfu maintains the persistent source of truth for all user-defined configurations, guaranteeing strict ACID (Atomicity, Consistency, Isolation, Durability) transactional integrity.

2. The Central Control Plane (CCP)

The Central Control Plane runs as a distributed daemon (cloudnet) on the NSX Manager cluster appliances alongside the Management Plane, but maintains strict logical separation.

• Runtime State Calculation: The CCP receives desired state configuration from the MP and computes the global, dynamic runtime state of the network. It tracks virtual machine MAC addresses, IP addresses, Virtual Network Identifiers (VNIs), and tunnel endpoint (TEP) locations across the entire datacenter.
• ARP Suppression: By maintaining a complete directory of IP-to-MAC bindings across all overlay segments, the CCP allows local hypervisors to answer guest OS ARP requests locally, eliminating broadcast storms across the physical datacenter fabric.
• Isolation from Data Traffic: The Central Control Plane never inspects, handles, or forwards actual tenant payload packets. Packet forwarding occurs exclusively in the Data Plane.
• Communication: The CCP communicates with the Local Control Plane on each transport node over a secure, mutual-TLS encrypted control channel utilizing TCP port 1234.

3. The Local Control Plane (LCP)

The Local Control Plane resides locally on every participating hypervisor and Edge appliance, operating as the Network Control Plane Agent (netcpa) daemon.

• State Programming: The LCP receives runtime forwarding tables and security policies from the CCP and compiles them directly into low-level forwarding tables within the local hypervisor kernel or Edge forwarding engine.
• Telemetry and Topology Feedback: The LCP continuously monitors local virtual switch ports, physical link states, and VM lifecycle events (such as power-on, power-off, and vMotion), dynamically reporting local state changes back up to the Central Control Plane.
• Autonomous Operation: If connection to the CCP is temporarily lost, the LCP retains all existing forwarding entries, allowing the local data plane to continue routing and switching existing flows without interruption.

4. The Data Plane (DP)

The Data Plane is the engine that processes, encapsulates, inspects, and forwards actual network packets.

• Host Transport Nodes (ESXi Hypervisors): The data plane is implemented directly within the ESXi VMkernel through specialized kernel modules (nsx-vdl2, distributed virtual switch modules, and distributed firewall filters). Packet processing, Geneve encapsulation/decapsulation, distributed routing, and microsegmentation occur entirely in kernel memory at native physical line rate without user-space context switching.
• Edge Transport Nodes (VM or Bare-Metal): The data plane runs as a high-performance user-space fast-path process leveraging the Intel Data Plane Development Kit (DPDK). DPDK polls physical NICs directly, bypassing operating system kernel interrupt overhead to deliver millions of packets per second for centralized network services such as external BGP routing, NAT, and stateful gateway firewalls.

---

NSX Manager Appliance Cluster & Virtual IP (VIP) Architecture

High availability and resiliency of the management and control planes are paramount in enterprise cloud deployments.

┌─────────────────────────────────────────────────────────────────────────────────┐
│                     NSX MANAGER THREE-NODE CLUSTER ARCHITECTURE                 │
├─────────────────────────────────────────────────────────────────────────────────┤
│                 Virtual IP (VIP) - e.g., 172.16.10.100 (Single L2 Subnet)       │
│                         │ (Gratuitous ARP to Active Leader)                     │
│         ┌───────────────┼───────────────┐                                       │
│         ▼               ▼               ▼                                       │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐                                │
│  │ NSX Node 1  │ │ NSX Node 2  │ │ NSX Node 3  │                                │
│  │ 172.16.10.11│ │ 172.16.10.12│ │ 172.16.10.13│                                │
│  │  (Leader)   │ │  (Follower) │ │  (Follower) │                                │
│  └──────┬──────┘ └──────┬──────┘ └──────┬──────┘                                │
│         └───────────────┼───────────────┘                                       │
│                         ▼                                                       │
│         Corfu Distributed Datastore (2-of-3 Quorum Majority)                    │
└─────────────────────────────────────────────────────────────────────────────────┘

The Three-Node Quorum Mandate

In VMware Cloud Foundation 9.0, the NSX management cluster must be deployed with exactly three (3) NSX Manager appliances.

• Consensus and Split-Brain Prevention: Corfu relies on a Paxos-derived consensus algorithm to guarantee data consistency. To achieve quorum, a strict majority of nodes must be active and communicating: Quorum = floor(N / 2) + 1. For a 3-node cluster, quorum requires at least two (2) operational nodes.
• Why Two Nodes Are Unsupported: A 2-node cluster provides zero additional fault tolerance over a single node because losing one node leaves only one surviving node (1 < floor(2/2) + 1 = 2), causing the datastore to enter read-only mode to prevent split-brain data corruption. A 3-node cluster can tolerate the complete failure of any single manager appliance while maintaining full read-write operational capabilities.

Appliance Sizing Specifications

When SDDC Manager deploys NSX Manager appliances, it provisions virtual machines based on standardized sizing profiles:

Virtual IP (VIP) vs. External Load Balancing

To provide a single, resilient management endpoint for vCenter Server, SDDC Manager, and administrative users, NSX includes a built-in Virtual IP (VIP) mechanism:

• Layer 2 Requirement: The built-in VIP operates by binding an additional IP address to the management network interface of the active cluster leader and announcing it via Gratuitous ARP (GARP). Consequently, all three NSX Manager appliances must reside on the exact same Layer 2 broadcast domain and IP subnet.
• Multi-Subnet Exception: If an organization requires distributing NSX Manager appliances across distinct routed Layer 3 subnets (for example, across different server racks or failure domains), the built-in VIP cannot be utilized. In this topology, an external Layer 4/Layer 7 Load Balancer (such as VMware NSX Advanced Load Balancer) or DNS round-robin with FQDN redirection must be deployed in front of the cluster nodes.

---

Transport Nodes: Host Transport Nodes vs. Edge Transport Nodes

A Transport Node is any physical server or virtual appliance that participates in the NSX data plane, prepared with the software modules required to originate, terminate, and forward encapsulated overlay frames.

Host Transport Nodes (HTNs)

Host Transport Nodes are the workhorses of the private cloud compute fabric. When an ESXi host is prepared as a transport node, SDDC Manager or NSX installs kernel VIBs (vSphere Installation Bundles). These VIBs inject distributed routing, distributed switching, and distributed firewalling directly into the VMkernel IOChain. Because packet processing occurs within the hypervisor on the physical host where the workload VM is executing, East-West traffic between virtual machines is routed and secured at physical wire speed without traversing external networking hardware.

Edge Transport Nodes (ETNs)

Edge Transport Nodes are dedicated service appliances designed to handle network functions that cannot be distributed across hundreds of hypervisor kernels. For example, establishing a dynamic BGP peering session with physical datacenter core routers requires a deterministic, centralized IP endpoint. Similarly, stateful services such as Source NAT (SNAT) and stateful Layer 7 gateway inspection require centralized connection tracking tables. Edge nodes host the Services Router (SR) components of Tier-0 and Tier-1 gateways, acting as the secure gateway between the software-defined overlay fabric and the physical network underlay.

---

Physical Network Underlay Fabric Requirements

While NSX virtualizes network topologies in software, the virtual overlay is entirely dependent upon the health, stability, and throughput of the physical network underlay. A poorly architected physical network introduces packet drops, high jitter, and degraded overlay throughput.

1. Leaf-Spine (Clos) IP Fabric Topology

Modern VMware Cloud Foundation deployments mandate a two-tier Leaf-Spine (Clos) network architecture:

• Deterministic Latency: Every Leaf (Top-of-Rack) switch connects to every Spine switch. Traffic between any two ESXi hosts across different racks requires exactly three physical hops (Leaf -> Spine -> Leaf), guaranteeing consistent, low-latency East-West performance.
• Elimination of Spanning Tree (STP): Layer 2 Spanning Tree is eliminated from the physical core. The leaf-spine fabric is built as a pure routed Layer 3 IP underlay. Uplinks between leaf and spine switches utilize routed /31 point-to-point subnets.
• Equal-Cost Multi-Path (ECMP): Traffic across the physical spine switches is load-balanced dynamically using ECMP at Layer 3, ensuring 100% active-active link utilization across all available physical cables.

2. Physical MTU Sizing: The 1700+ / 9000 Jumbo Frame Mandate

Virtual overlay networking encapsulates standard tenant Ethernet frames inside outer UDP packets using the Geneve protocol. This encapsulation adds packet header overhead:

• Outer Ethernet Header: 14 bytes
• Outer IP Header: 20 bytes
• Outer UDP Header: 8 bytes
• Geneve Base Header & TLV Options: 8 bytes (minimum) up to 24+ bytes

The total overhead added by NSX Geneve encapsulation is at least 50 bytes.

Standard Tenant VM Frame: 1500 Bytes MTU
  + Geneve Encapsulation Overhead: 50+ Bytes
  = Total Physical Encapsulated Frame: 1550+ Bytes

If the physical network switches are left at the standard Ethernet default MTU of 1500 bytes, physical switches will immediately fragment or drop the encapsulated Geneve packets. Packet fragmentation causes severe CPU exhaustion on physical switches and host hypervisors, resulting in catastrophic application timeouts and broken tunnel connectivity.

The MTU Rule for VCF: The physical underlay network must support an MTU of at least 1700 bytes end-to-end between all host and edge Tunnel Endpoints (TEPs). To ensure maximum throughput, eliminate fragmentation risk, and optimize storage protocols (such as vSAN and vMotion), Broadcom strongly standardizes on MTU 9000 (Jumbo Frames) across all physical switch ports, leaf-spine links, and host network adapters.

3. Underlay Dynamic Routing: Leaf-Spine BGP Peering

To establish routable connectivity between host Tunnel Endpoints residing on different Layer 2 subnets across different racks, the physical underlay runs a dynamic routing protocol—typically External BGP (eBGP):

• Each Leaf switch pair is assigned a unique private Autonomous System Number (ASN).
• The Spine switches share a common Spine ASN.
• Host TEP subnets configured on Top-of-Rack switches are advertised into the underlay BGP routing table, allowing any ESXi host TEP to establish direct UDP 6081 encapsulation tunnels to any other host or Edge TEP across the datacenter.

---

Exam Watch: Key Scenarios and Candidate Traps

[!IMPORTANT] The Plane Independence Principle: A favorite scenario on the VCP-VCF exam describes a complete outage of all three NSX Manager appliances (for example, caused by an administrative storage array failure hosting the management VMs). The question asks what happens to running production virtual machine traffic. The data plane continues to forward packets without interruption. Because the Local Control Plane (LCP) and VMkernel data plane operate autonomously with cached forwarding tables, existing routing, switching, Geneve encapsulation, and firewall rules continue operating at line rate. Only management operations (creating new segments, updating firewall rules, or deploying new VMs) and dynamic control-plane MAC learning updates are temporarily paused.

[!WARNING] The Two-Node Cluster Trap: Do not select two appliances as a valid high-availability configuration for NSX Manager. Two nodes cannot achieve quorum during a network partition, leading to an operational deadlock where neither node can safely accept writes. A minimum of three nodes is strictly required.

[!NOTE] Virtual IP (VIP) Subnet Boundary: Remember that the built-in NSX VIP relies on Gratuitous ARP and cannot span across routed Layer 3 subnets. If NSX Manager nodes are placed in different subnets across racks, an external load balancer must be used instead of the native VIP.