7.2 Host Network Uplinks, Virtual Switch Configuration, and NIC Bonding Modes

Host Network Uplinks, Virtual Switch Configuration, and NIC Bonding Modes

Quick Summary: In a Nutanix enterprise cloud, physical network adapters (NICs) on each AHV host are aggregated into logical bonds attached to virtual switches. The architectural selection of physical adapters, switch configurations, and bonding modes directly dictates cluster throughput, failover behavior, and operational resilience. AHV provides three distinct bonding modes: active-backup (the default, zero-configuration mode providing instant failover), balance-slb (source-MAC load balancing across multiple active uplinks without upstream switch complexity), and balance-tcp (enterprise LACP link aggregation with Layer 2–4 per-flow distribution). Managing these configurations is streamlined through the Prism Virtual Switch UI and the host-level manage_ovs CLI utility.

---

1. Physical NIC Selection and Hardware Topology

Enterprise hyperconverged infrastructure demands reliable, high-bandwidth interconnects because physical network uplinks carry both traditional client-to-VM traffic and east-west Distributed Storage Fabric (DSF) storage replication data.

Network Adapter Speed and Homogeneity

Nutanix nodes typically ship with dual-port or quad-port Network Interface Cards supporting 10GbE (SFP+), 25GbE (SFP28), 40GbE (QSFP+), or 100GbE (QSFP28) interfaces.

+-------------------------------------------------------------------------+
|                        Uplink Speed Homogeneity                         |
+------------------------------------+------------------------------------+
|   RECOMMENDED: Homogeneous Pair    |   PROHIBITED: Asymmetric Mixing    |
|                                    |                                    |
|   +------------+  +------------+   |   +------------+  +------------+   |
|   | eth2:10GbE |  | eth3:10GbE |   |   | eth0: 1GbE |  | eth2:10GbE |   |
|   +------------+  +------------+   |   +------------+  +------------+   |
|         \                /         |         \                /         |
|          v              v          |          v              v          |
|        [ Bond: br0-up (20G) ]      |     [ Bond: br0-up (Unstable) ]    |
|   Predictable latency & failover   |     Severe throughput choke-point  |
+------------------------------------+------------------------------------+

[!CRITICAL] Never mix 1GbE and 10GbE+ interfaces in the same bond or bridge. While standard Linux bonding technically permits grouping adapters of disparate speeds, doing so in an AHV storage cluster introduces severe hazards. If an active-backup bond fails over from a 10GbE adapter to a 1GbE adapter, the sudden 90% reduction in bandwidth chokes storage write replication, triggering I/O latency spikes and cluster health degradation. 1GbE interfaces (typically onboard motherboard ports) should either be left disconnected or assigned to a dedicated out-of-band management bridge (br1), leaving high-speed 10GbE/25GbE adapters for br0.

Redundant Top-of-Rack (ToR) Cabling Architecture

To eliminate single points of physical failure:

• Each AHV node must connect its bonded uplinks across two separate physical Top-of-Rack (ToR) switches (e.g., Switch A and Switch B).
• For active-backup and balance-slb, the physical switches do not need to be clustered or stacked; standard independent Layer 2/3 switches operate seamlessly.
• For balance-tcp, the two physical switches must support a multi-chassis link aggregation technology—such as Cisco Virtual PortChannel (vPC), Arista Multi-Chassis Link Aggregation (MLAG), or Dell Virtual Link Trunking (VLT)—so that the host views the physical uplinks as terminating on a single logical switch.

---

2. AHV NIC Bonding Modes: Technical Deep Dive

In AHV, physical network interfaces attached to a virtual bridge are pooled into a single bonded interface port named br0-up (or vs0-up). AHV natively supports three bonding modes, each balancing configuration complexity against traffic distribution efficiency.

+---------------------------------------------------------------------------------+
|                            AHV NIC Bonding Modes                                |
+---------------------+-------------------------+---------------------------------+
| Mode                | Upstream Switch Config  | Traffic Distribution Algorithm  |
+---------------------+-------------------------+---------------------------------+
| active-backup       | None (Any switch)       | 1 Active Uplink; Standby on fail|
| balance-slb         | None (Any switch)       | Source-MAC hash across uplinks  |
| balance-tcp (LACP)  | Dynamic LACP (MLAG/vPC) | L2-L4 5-tuple hash per flow     |
+---------------------+-------------------------+---------------------------------+

Mode 1: active-backup (Default Out-of-the-Box Mode)

active-backup is the default bonding mode configured during cluster foundation. It is engineered for maximum operational simplicity and universal hardware compatibility.

• Operational Mechanics: Exactly one physical adapter in the bond is elected as the active interface; all other adapters remain in standby. All network traffic generated by the Controller VM (eth0), the AHV host (br0), and all running guest VMs traverses this single active adapter.
• Upstream Switch Requirements: Zero configuration. Upstream physical switch ports are configured as standard independent trunk ports carrying the required VLANs. No port channels, LACP, or switch stacking are required.
• Failover Characteristics: If the active physical adapter detects a carrier loss (cable pull, transceiver failure, or switch port shutdown), AHV initiates failover in under 100 milliseconds. OVS immediately transitions an inactive standby adapter to active and broadcasts gratuitous ARP (GARP) updates to notify upstream switches of the MAC address relocation.
• Failback Behavior: When the failed primary link restores, AHV by default continues operating on the currently active adapter to avoid unnecessary link flapping, unless explicit failback policies are enforced.

Mode 2: balance-slb (Source-Load Balancing)

balance-slb enhances uplink utilization by distributing traffic across all active physical adapters without introducing complex switch configurations.

• Operational Mechanics: OVS assigns each virtual machine's virtual network interface (and CVM interfaces) to an active physical uplink based on the virtual machine's source MAC address. If Host Node 1 has two 10GbE uplinks (eth2, eth3) and runs four VMs:
  • VM 1 and VM 2 (hashed to eth2) transmit via eth2.
  • VM 3, VM 4, and CVM (hashed to eth3) transmit via eth3.
• Upstream Switch Requirements: Zero configuration. Because each specific virtual MAC address is strictly pinned to a single physical port at any given instant, upstream switches never observe the same MAC address appearing across multiple ports simultaneously. This avoids MAC flapping and switch port security violations without requiring physical switch clustering.
• Dynamic Load Rebalancing: OVS periodically evaluates the traffic load across bonded physical interfaces (default rebalance timer is 10 seconds). If the bandwidth utilization between two physical adapters differs by more than 25%, OVS dynamically re-hashes one or more virtual machine MAC addresses to the less utilized uplink.
• Performance Ceiling: A single virtual machine (or single virtual NIC) cannot transmit faster than the line rate of a single physical adapter. A VM on a dual-10GbE balance-slb bond has a maximum throughput ceiling of 10 Gbps.

[!CAUTION] balance-slb is not compatible with upstream switch Link Aggregation Control Protocol (LACP) configurations. Furthermore, environments utilizing IGMP snooping or complex multicast topologies should avoid balance-slb due to potential multicast packet duplication or drops during dynamic rebalancing.

Mode 3: balance-tcp (LACP / IEEE 802.3ad Link Aggregation)

balance-tcp is the premier enterprise bonding mode for environments requiring true multi-flow load distribution and maximum bandwidth utilization.

• Operational Mechanics: balance-tcp utilizes the standard IEEE 802.3ad Link Aggregation Control Protocol (LACP). Rather than hashing based solely on source MAC addresses, OVS parses the packet headers and calculates a 5-tuple hash:

$\text{Hash} = f(\text{Source IP}, \text{Destination IP}, \text{Source Port}, \text{Destination Port}, \text{Protocol})$

• Multi-Flow Acceleration: Because hash calculations occur per TCP/UDP flow rather than per VM MAC, a single virtual machine with multiple concurrent network connections can utilize multiple physical uplinks simultaneously.
  • A high-transaction Microsoft SQL Server establishing dozens of database sessions can transmit across both eth2 and eth3 concurrently, achieving aggregate throughput approaching 20 Gbps on a dual-10GbE bond.
  • The Controller VM (CVM) distributing storage replication extents across multiple remote nodes utilizes all bonded uplinks simultaneously, significantly accelerating cluster write performance.
• Upstream Switch Requirements: Upstream physical switches MUST be explicitly configured for dynamic LACP. In multi-switch topologies, switches must run a cross-chassis protocol (Cisco vPC, Arista MLAG, Juniper MC-LAG).
• LACP Timers (Fast vs. Slow):
  • Slow Rate (30-second heartbeat): LACP Control Protocol Data Units (LPDUs) exchange every 30 seconds; link failure detection can take up to 90 seconds.
  • Fast Rate (1-second heartbeat - Recommended): LPDUs exchange every 1 second; link failure is detected within 3 seconds, ensuring rapid convergence.
• lacp-fallback Protection: If an AHV host boots while upstream physical switches are undergoing maintenance or unconfigured for LACP, standard LACP bonds drop all traffic. AHV implements lacp-fallback, which automatically falls back to standard active-backup operation if LACP negotiations fail, preventing host isolation.

---

3. Configuring Virtual Switches via Prism UI

Introduced in AOS 5.11 and expanded in modern AOS releases, the Prism Virtual Switch management engine abstracts underlying OVS complexity into an intuitive graphical interface within Prism Element and Prism Central.

+-------------------------------------------------------------------------+
|                    Prism Virtual Switch Architecture                    |
|                                                                         |
|   Virtual Switch: vs0 (Default Virtual Switch)                          |
|   ├── Underlying OVS Bridge: br0                                        |
|   ├── MTU Configuration: 1500 (Standard) or 9000 (Jumbo Frames)         |
|   ├── Bond Configuration: vs0-up (Interfaces: eth2, eth3)               |
|   └── Bonding Mode: Active-Backup | Balance-SLB | Balance-TCP           |
+-------------------------------------------------------------------------+

Prism Virtual Switch Workflows

1. Navigating to Network Configuration: In Prism Element, navigate to Settings (Gear Icon) -> Network Configuration -> Virtual Switch.
2. Editing the Default Virtual Switch (vs0):
   • By default, all AHV nodes belong to vs0 (mapped to br0).
   • Administrators can review member nodes, assigned physical uplinks, and current bond modes.
3. Switching Bond Modes:
   • To change from active-backup to balance-tcp, select vs0, click Edit, choose Connected with LACP (balance-tcp), and set the LACP rate to Fast.
   • Prism automatically orchestrates rolling updates across all hosts in the cluster, validating link state before applying changes to prevent cluster-wide disconnects.
4. Jumbo Frames (MTU 9000):
   • Sizing MTU to 9000 reduces CPU packet header processing overhead for high-throughput storage networks.
   • MTU must be enabled end-to-end: on the Prism Virtual Switch, on the physical ToR switch ports, and on any intervening routers. A mismatched MTU causes packet fragmentation and severe I/O timeouts.

---

4. Host CLI Administration: manage_ovs and ovs-vsctl

While the Prism UI handles standard operations, system administrators require CLI tools for deployment automation, troubleshooting, and advanced diagnostics.

The Nutanix manage_ovs Command-Line Tool

manage_ovs is the official, Nutanix-supported cluster-aware Python utility for inspecting and configuring network uplinks and bridges on AHV nodes.

# Inspect all physical network adapters and current uplink bond states
manage_ovs show_uplinks

# Detailed interface status and link carrier state (speed, duplex, link)
manage_ovs show_interfaces

# Configure a 10GbE uplink bond in active-backup mode
manage_ovs --bridge_name br0 --interfaces eth2,eth3 --bond_mode active-backup update_uplinks

# Configure uplinks with balance-tcp (LACP) and fast transmission rate
manage_ovs --bridge_name br0 --interfaces eth2,eth3 --bond_mode balance-tcp --lacp_mode fast update_uplinks

Cluster-Wide Execution via allssh

To query or modify network configurations across all nodes in the cluster simultaneously, administrators execute manage_ovs from any Controller VM using the allssh wrapper:

# Check uplink status across every AHV node in the entire cluster
allssh "ssh root@192.168.5.1 manage_ovs show_uplinks"

Low-Level Diagnostics with ovs-vsctl

ovs-vsctl is the raw Open vSwitch management command. It directly queries the local ovsdb-server database:

# Display full OVS bridge, port, interface, and bond hierarchy
ovs-vsctl show

# Inspect specific port bonding options and LACP status
ovs-vsctl list port br0-up

# Inspect LACP partner negotiation status on physical links
ovs-appctl bond/show br0-up
ovs-appctl lacp/show br0-up

[!IMPORTANT] Always use manage_ovs or the Prism UI for configuration changes. Manual changes executed via ovs-vsctl may be overwritten during AHV upgrades or Acropolis cluster synchronization tasks.