7.1 AHV Virtual Networking Architecture, Open vSwitch (OVS), and Bridges

AHV Virtual Networking Architecture, Open vSwitch (OVS), and Bridges

Quick Summary: Nutanix AHV delivers enterprise virtualization without relying on proprietary third-party switching fabrics or external management servers. Virtual networking in AHV is powered natively by Open vSwitch (OVS), an open-source, production-grade distributed virtual multilayer switch. OVS operates directly within the Linux kernel and user-space layers of the AHV hypervisor. Network traffic is partitioned across specialized virtual bridges—most notably br0 (which aggregates physical host NIC uplinks, Controller VM management traffic, and guest VM virtual NICs) and virbr0 (an internal, isolated Linux bridge dedicated to private host-to-CVM communication on the 192.168.5.0/24 subnet). Understanding the split-plane OVS architecture, virtual bridge topologies, and OpenFlow flow processing is fundamental to administering Nutanix enterprise clouds.

---

1. Open vSwitch (OVS) Architectural Foundations

In traditional enterprise virtualization platforms, hypervisors employ proprietary virtual switches (such as the VMware vSphere Standard Switch or vSphere Distributed Switch) that require dedicated licensing and centralized management appliances. Nutanix AHV adopts an open, modular approach by standardizing on Open vSwitch (OVS).

OVS is designed to operate in high-density multi-tenant environments, supporting standard network management protocols, telemetry (sFlow, NetFlow, IPFIX), and automated flow programmability via OpenFlow.

Kernel Space vs. User Space Split-Plane Design

OVS in AHV divides networking responsibilities between two operational domains to maximize packet throughput while preserving dynamic programmability:

+-------------------------------------------------------------------------+
|                         AHV Host User Space                             |
|                                                                         |
|   +--------------------------+          +---------------------------+   |
|   |       ovsdb-server       |          |        ovs-vswitchd       |   |
|   | (Configuration Database) |<-------->| (OpenFlow Switching & Flow|   |
|   |      [/etc/openvswitch]  |  IPC/RPC |      Pipeline Engine)     |   |
|   +--------------------------+          +-------------+-------------+   |
+-------------------------------------------------------|-----------------+
                                                        | Upcall / Netlink
+-------------------------------------------------------|-----------------+
|                         AHV Host Kernel Space         v                 |
|   +-----------------------------------------------------------------+   |
|   |                      openvswitch.ko (Datapath)                  |   |
|   |      - Exact Match Cache (EMC) & Megaflow Fast Path             |   |
|   |      - Hardware NIC Interfacing & Ring Buffer Processing        |   |
|   +-----------------------------------------------------------------+   |
+-------------------------------------------------------------------------+

1. User Space Components:

   • ovsdb-server (Configuration Database Server): A lightweight, transactional database server that stores the virtual switch schema, bridge definitions, port configurations, interface mappings, and bond states in /etc/openvswitch/conf.db. It ensures persistence across host reboots.
   • ovs-vswitchd (OVS Daemon): The switching core that implements the OpenFlow control engine. It communicates with ovsdb-server via local IPC, processes complex OpenFlow classification rules, interfaces with the Nutanix management plane (Acropolis), and computes forwarding decisions for unrecognized packet streams.

2. Kernel Space Components:

   • openvswitch.ko (Fast-Path Datapath Module): A lean, hyper-optimized kernel module executing in the Linux kernel space. It inspects packet headers against cached flow rules. When an incoming packet matches an existing cached flow, openvswitch.ko immediately modifies, forwards, or drops the packet directly in kernel memory without incurring an expensive user-space context switch.

[!NOTE] By processing the vast majority of ongoing network traffic inside the kernel datapath, AHV achieves near-line-rate switching performance with negligible hypervisor CPU overhead.

---

2. Virtual Bridge Topology: br0 vs. virbr0

AHV uses logical virtual bridges to direct traffic between physical adapters, virtual machines, and the internal storage infrastructure. Every Nutanix AHV host deploys two default bridges upon installation: br0 and virbr0.

+---------------------------------------------------------------------------------+
|                                AHV Physical Host                                |
|                                                                                 |
|  +--------------------+   +--------------------+     +-----------------------+  |
|  |    User VM 1       |   |    User VM 2       |     |     Controller VM     |
|  |   [vnet0 / tap]    |   |   [vnet1 / tap]    |     |         (CVM)         |
|  +---------+----------+   +---------+----------+     +---+---------------+---+  |
|            |                        |                    |               |      |
|            |                        |               eth0 |          eth1 |      |
|            v                        v                    v               |      |
|  +-------------------------------------------------------+--+            |      |
|  |                     Virtual Bridge: br0                  |            |      |
|  |                      (Open vSwitch)                      |            |      |
|  |  Aggregates VM vNICs, CVM eth0, and Physical Bond        |            |      |
|  +---------------------------+------------------------------+            |      |
|                              |                                           |      |
|                              v                                           |      |
|                 +--------------------------+                             |      |
|                 |    Bond Port: br0-up     |                             |      |
|                 |  (Active-Backup / LACP)  |                             |      |
|                 +-------------+------------+                             |      |
|                               |                                          |      |
|                     +---------+---------+                                |      |
|                     |                   |                                |      |
|                     v                   v                                |      |
|                 [ eth2 ]             [ eth3 ]                            |      |
|                 10/25GbE             10/25GbE                            |      |
|                     |                   |                                |      |
|  +------------------+-------------------+--------------------------------+---+  |
|  |                                      |                                |   |  |
|  | Physical Network                     v                                v   |  |
|  | ToR Switches                    +----+--------------------------------+   |  |
|  |                                 |       Virtual Bridge: virbr0        |   |  |
|  |                                 |       (Standard Linux Bridge)       |   |  |
|  |                                 |   Subnet: 192.168.5.0/24 (Private)  |   |  |
|  |                                 |   AHV Host: .1  <---->  CVM eth1: .2|   |  |
|  |                                 +-------------------------------------+   |  |
+--+---------------------------------------------------------------------------+--+

The External Bridge: br0 (Open vSwitch)

br0 is the primary data and management bridge of the AHV host. It is instantiated as an Open vSwitch bridge and manages all network ingress and egress between the host, the virtual workloads, and the physical datacenter switching fabric.

Key interfaces attached to br0 include:

• br0-up (Uplink Bond): The logical bonded port that aggregates physical server network adapters (e.g., eth2, eth3). All external network traffic leaving or entering the node traverses br0-up.
• br0 (Host Interface): The virtual host management interface. It carries the primary management IP address of the AHV host, enabling communication with other AHV nodes, Prism Central, and external network services (DNS, NTP, Syslog).
• CVM eth0: The primary network interface of the local Controller Virtual Machine. It carries external storage I/O, Distributed Storage Fabric (DSF) metadata synchronization, remote replication, and Prism web management traffic.
• Guest VM Virtual Interfaces (vnet* / tap*): As virtual machines power on, AHV generates a dedicated virtual network tap interface for each vNIC. These tap ports attach directly to br0 with appropriate IEEE 802.1Q VLAN membership tags.

The Internal Bridge: virbr0 (Standard Linux Bridge)

In stark contrast to br0, virbr0 is NOT an Open vSwitch bridge. It is a traditional Linux bridge managed by the Linux kernel bridge module (brctl).

Critical characteristics of virbr0:

• Isolated Private Network: It is strictly assigned to the link-local, non-routable private subnet 192.168.5.0/24 with an MTU of 1500.
• Zero Physical Uplinks: virbr0 has no physical network adapters attached and contains no external routes. Network packets traversing virbr0 can never leave the physical server chassis.
• Fixed IP Addressing:
  • The AHV hypervisor assigns itself 192.168.5.1 on interface virbr0.
  • The local Controller VM assigns itself 192.168.5.2 on interface eth1.
• Operational Purpose:
  1. Cluster Bootstrapping and Genesis Startup: During initial node boot, the CVM and hypervisor establish cluster communication across this deterministic internal path before physical network links or external IP routing are fully online.
  2. Localized Storage Redirection: When an AHV host requires local storage access (such as mounting local CVM storage endpoints, dispatching iSCSI/SCSI control primitives, or dumping hypervisor crash core dumps), traffic flows across 192.168.5.0/24 without traversing physical ToR switches.
  3. Out-of-Band Host Management: Administrators connected to the CVM via SSH can jump directly into the underlying AHV hypervisor using ssh root@192.168.5.1 without requiring external host routing.

[!WARNING] Never delete, modify, or add physical network interfaces to virbr0. Altering virbr0 breaks local CVM-to-hypervisor communication, causing the host to enter an unmanageable state and terminating local storage heartbeat processing.

---

3. Comparison: AHV Open vSwitch vs. VMware ESXi Virtual Switching

Understanding how AHV networking maps to legacy virtualization platforms helps clarify Nutanix design advantages:

Decentralized Control Plane Resilience

In a VMware vSphere environment utilizing a Distributed Switch (vDS), the control plane resides within the vCenter Server appliance. If the vCenter Server VM crashes, suffers database corruption, or loses network connectivity, administrators cannot create port groups, assign VLANs, or migrate VM interfaces until vCenter is restored.

In Nutanix AHV, each node runs its own autonomous instance of ovsdb-server and ovs-vswitchd. The Acropolis service running across the CVM cluster coordinates network state changes. Configuration updates dispatched through Prism Element or Prism Central are replicated across the cluster using the distributed ZooKeeper consensus engine. Even if one or more CVMs are rebooted or Prism Central is temporarily offline, each AHV host continues switching packets and enforcing flow rules without disruption.

---

4. Flow Table Mechanics and the Packet Forwarding Pipeline

To understand how AHV forwards packets at microsecond latencies, one must analyze the interaction between the Kernel Datapath and the User-Space OpenFlow Pipeline.

Incoming Packet from VM Tap / Physical Uplink
                     |
                     v
     +-------------------------------+   MISS (Upcall)
     |  Kernel Datapath Cache (EMC)  |-------------------+
     +---------------+---------------+                   |
                     |                                   |
          HIT        |                                   v
                     |                 +-----------------------------------+
                     |                 |  ovs-vswitchd (User Space Engine) |
                     |                 |  1. Parse L2/L3/L4 Packet Headers |
                     |                 |  2. Traverse OpenFlow Tables      |
                     |                 |  3. Check Flow Security Rules     |
                     |                 |  4. Generate Datapath Flow Action |
                     |                 +-----------------+-----------------+
                     |                                   |
                     |                                   v
                     |                 +-----------------------------------+
                     |                 | Install Flow into Kernel Datapath |
                     |                 +-----------------+-----------------+
                     |                                   |
                     +-----------------+-----------------+
                                       |
                                       v
                       Execute Actions (Push/Pop VLAN,
                       Forward to Port, or Drop Packet)

The Step-by-Step Packet Forwarding Process

1. Packet Ingress: A packet arrives on an AHV bridge port—either from a virtual machine's vnet tap interface or from a physical switch port via br0-up.
2. Kernel Fast-Path Lookup (Exact Match Cache - EMC):
   • The kernel datapath calculates a hash of the packet's 5-tuple header fields (Source IP, Destination IP, Source Port, Destination Port, Protocol) plus input port and VLAN.
   • It checks the Exact Match Cache (EMC). If an identical flow has recently passed through the switch, a cache hit occurs.
   • The kernel module immediately executes the stored action (e.g., strip 802.1Q tag and transmit to vnet2) in a few microseconds.
3. Megaflow Cache Evaluation:
   • If the EMC misses, the datapath checks the Megaflow Cache, which supports wildcarded flow entries (e.g., matching an entire subnet rather than a single IP).
   • A hit here executes the cached action and updates the EMC.
4. The Upcall (Slow Path to User Space):
   • If the packet does not match any entry in the kernel cache, the kernel datapath generates an upcall via a Linux Netlink socket, passing the packet header to ovs-vswitchd in user space.
5. OpenFlow Multi-Table Pipeline Traversal:
   • ovs-vswitchd receives the packet and passes it through an ordered series of OpenFlow tables:
     • Table 0 (Ingress Classification): Validates source port security and assigns internal metadata.
     • Table 1 (VLAN Tagging / Untagging): Determines whether 802.1Q tags must be added (access to trunk) or stripped.
     • Table 2 (Flow Network Security / Conntrack): Evaluates stateful firewall rules via Linux connection tracking (conntrack).
     • Table 3 (Forwarding Decision): Looks up the destination MAC address in the OVS MAC learning table (FDB) to identify the destination bridge port.
6. Datapath Flow Installation and Egress:
   • ovs-vswitchd transmits the forwarding instruction back to the kernel datapath and installs a new exact-match rule in the kernel cache.
   • The datapath executes the action and forwards the packet out the target port.
   • All subsequent packets belonging to that same session hit the kernel fast path directly, bypassing user space entirely.