7.3 NIC Teaming, Load Balancing, VLANs & NIOC

7.3 NIC Teaming, Load Balancing, VLANs & NIOC

Virtual switches connect virtual infrastructure to the physical world through physical network adapters (vmnics). Connecting multiple physical adapters to a single virtual switch forms a NIC team. A properly designed NIC teaming, load balancing, and traffic governance architecture prevents network outages, optimizes bandwidth utilization, enforces tenant isolation, and guarantees quality of service (QoS) across competing hypervisor workloads.

---

NIC Teaming Architecture & Failover Configurations

NIC Teaming allows multiple physical network adapters to act as a redundant, cooperative group. Within a port group or distributed port group, physical uplinks are partitioned into three operational tiers:

1. Adapter Operational Tiers

• Active Adapters: Forward traffic during normal conditions. If multiple adapters are active, the configured load balancing algorithm determines how outbound traffic is distributed among them.
• Standby Adapters: Remain online and physically linked to the switch but pass zero traffic as long as at least one Active adapter is functional. A standby uplink can replace a failed active member according to the configured failover policy.
• Unused Adapters: Completely excluded from the port group. Traffic on this port group will never utilize an Unused adapter, even if every Active and Standby adapter fails. Unused assignments are used to enforce physical traffic separation (e.g., reserving vmnic2 exclusively for iSCSI).

2. Teaming Failover Settings

• Failback Policy: Controls how a restored physical adapter is handled:
  • Yes (Default): When a failed physical adapter recovers, it immediately reclaims its active status, displacing any Standby adapter that stepped in.
  • No (Enterprise Best Practice for Flapping Links): Prevents the restored adapter from reclaiming active status until the currently active adapter fails. This prevents link thrashing—a condition where an intermittent, flapping physical switch port repeatedly disrupts network state.
• Notify Switches: When set to Yes (Default), the ESXi host generates gratuitous Reverse ARP (RARP) frames whenever a virtual machine fails over to a new physical uplink. This immediately updates upstream physical switch Content-Addressable Memory (CAM) tables with the VM's new physical port location, preventing dropped packets.

---

Failover Detection Mechanisms: Link Status Only vs. Beacon Probing

When a network failure occurs, the hypervisor must detect the disruption and initiate failover to an alternate uplink. vSphere provides two detection methods:

1. Link Status Only

• Mechanism: Relies entirely on the physical network card's link-state carrier signal. It detects when the physical link between the ESXi host and the immediately connected access switch is broken (e.g., unplugged cable, physical switch power outage).
• Limitations: Cannot detect upstream transit failures. If the physical switch port remains powered on, but an upstream core switch link or inter-switch trunk fails—or if the physical switch experiences an internal software lockup—Link Status Only continues to report the link as healthy. Traffic sent out that uplink is dropped into a "black hole."

2. Beacon Probing

• Mechanism: The VMkernel broadcasts periodic Layer 2 beacon packets across all physical network adapters in the team. Sibling physical adapters listening on the same broadcast domain expect to receive these beacon frames.
• Detection: If an adapter stops receiving beacons from its peers, the hypervisor recognizes an upstream transit disruption, marks the path down, and triggers failover, even if the physical link light remains solid green.

Beacon-Probing Constraints

Use at least three appropriately connected uplinks for meaningful failure isolation. With only two, missing beacons cannot reliably identify the failed path. Validate the physical switch topology and shared broadcast reachability; three NICs connected to one failed switch do not provide three independent paths. Beacon probing is not supported with IP-hash teaming.

Load Balancing Algorithms: Mechanics, Hashing & Switch Requirements

vSphere supports four distinct load balancing algorithms. Each operates under specific hashing formulas, behavioral characteristics, and physical switch prerequisites.

1. Route Based on Originating Virtual Port ID (Default)

• Algorithm: The virtual switch examines the virtual port ID assigned to the virtual machine vNIC on the switch and hashes it to select an active physical uplink from the team.
• Behavior: Deterministic 1-to-1 pinning. A virtual machine's vNIC is bound to a single physical adapter and will never use any other adapter unless the assigned adapter suffers a physical failure.
• Switch Configuration: None. Works seamlessly with standard access and trunk ports on any vendor's physical switch.
• Advantage: Low CPU overhead, deterministic behavior, fast failover.

2. Route Based on Source MAC Hash

• Algorithm: Calculates an uplink assignment based on the virtual machine's virtual MAC address.
• Behavior: Similar to Port ID, providing persistent 1-to-1 binding between a vNIC and a physical uplink. Useful in environments where virtual port IDs change frequently but MAC addresses remain static.
• Switch Configuration: None.

3. Route Based on IP Hash

• Algorithm: Selects an uplink from the source/destination IP pair; treat this as a flow-distribution mechanism, not a documented universal XOR formula.

• Behavior: Allows a single virtual machine communicating with multiple distinct IP destinations to distribute its sessions across multiple physical uplinks simultaneously.

• Physical Switch Configuration: Mandatory. The upstream physical switch ports must be configured into a static Link Aggregation Group (EtherChannel or static IEEE 802.3ad LAG).

• Strict Teaming Constraints:

  • All physical adapters in the team must be set to Active. Do not use standby members in that IP-hash team; adapters excluded from a port group are not members of its active aggregation.
  • Beacon Probing is strictly unsupported with IP Hash.

4. Route Based on Physical NIC Load (Load-Based Teaming - LBT)

• Availability: Exclusive to the vSphere Distributed Switch (vDS). Not available on standard switches.
• Initial Placement: VMs are initially assigned to an uplink using their virtual port ID.
• Dynamic Rebalancing Mechanics: The vDS continuously monitors the transmit and receive bandwidth utilization of every physical uplink in the team at 30-second sampling intervals.
• The 75% Threshold: If sustained bandwidth utilization on any physical adapter exceeds 75% of link capacity over the 30-second window, the vDS automatically and transparently migrates the vNIC of one or more virtual machines to an alternate, less utilized physical uplink in the team.
• Physical Switch Configuration: No LAG is required; normal VLAN/MTU/redundancy configuration is still necessary. LBT requires no physical switch configuration, no EtherChannel, and no LACP.
• Architectural Value: LBT provides the dynamic multi-uplink bandwidth optimization of link aggregation without the operational fragility and switch-vendor lock-in of physical switch port-channels.

---

Comparison of Load Balancing Algorithms

---

VLAN Tagging Modes: EST, VST & VGT

VLAN tagging in vSphere determines where IEEE 802.1Q VLAN headers are inserted and stripped in the Ethernet frame.

1. External Switch Tagging (EST)

• Configuration: Set VLAN ID to 0 (or None) on the virtual switch port group.
• Mechanics: The virtual switch performs no tagging. Physical switch ports connecting to the ESXi host are configured as Access ports mapped to a specific VLAN.
• Use Case: Legacy environments where each physical NIC pair is dedicated to a single VLAN. Highly inefficient because running multiple VLANs requires multiple pairs of physical NICs.

2. Virtual Switch Tagging (VST - Standard Enterprise Architecture)

• Configuration: Set VLAN ID to a specific number (1 to 4094) on the virtual switch port group.
• Mechanics: Upstream physical switch ports are configured as 802.1Q Trunk ports. Virtual machines send standard untagged frames. When an egress frame enters the port group, the virtual switch adds an 802.1Q VLAN tag before passing it to the physical uplink. On ingress, the virtual switch strips the 802.1Q tag before delivering the frame to the target virtual machine.
• Use Case: The standard deployment model for enterprise datacenters, allowing hundreds of VLANs to traverse a single pair of physical uplinks.

3. Virtual Guest Tagging (VGT)

• Configuration: On a standard-switch port group, use VLAN 4095 for VGT. On a distributed port group, configure VLAN trunking with the permitted VLAN range.
• Mechanics: VLAN ID 4095 is a special reserved configuration in vSphere that instructs the virtual switch to preserve all 802.1Q VLAN tags. The virtual switch acts as a transparent trunk, passing tagged frames directly into the virtual machine's virtual network card without stripping or modifying the headers.
• Use Case: Appropriate for virtualized network appliances, such as virtual firewalls, routers (e.g., VMware NSX Edges), intrusion detection sensors, and container host operating systems that manage multi-VLAN sub-interfaces directly within the guest OS.

---

Network I/O Control (NIOC v3): Bandwidth Allocation & QoS

On modern enterprise hosts equipped with high-speed 25 GbE or 100 GbE converged interfaces, multiple traffic streams (vSAN, vMotion, Management, and virtual machines) share the same physical network pipes. During peak periods (such as a massive vSAN rebuild occurring simultaneously with multi-host vMotion migrations), unmanaged traffic can saturate physical links and starve critical management or VM traffic. Network I/O Control (NIOC v3) provides software-defined Quality of Service (QoS) on vSphere Distributed Switches.

1. System Traffic Resource Pools

NIOC v3 allocates bandwidth across standard vSphere system traffic categories:

• vSAN Traffic
• vMotion Traffic
• Management Traffic
• iSCSI / NFS Storage Traffic
• vSphere Replication / NFC Traffic
• Fault Tolerance (FT) Traffic
• Virtual Machine Traffic

2. Bandwidth Allocation Controls

For each traffic type, administrators can configure three allocation parameters:

• Shares (Contention Control): Dictates relative priority during periods of link saturation. Expressed as Low (25), Normal (50), High (100), or Custom numeric values. Shares determine relative entitlement during contention; they do not cap unused bandwidth when capacity is available.
• Reservations (Guaranteed Minimum): Reserves a minimum entitlement on the host physical-adapter resource under supported admission rules. Total reservations are limited to 75% of the physical adapter capacity. This does not reserve capacity across an upstream fabric or guarantee application throughput.
• Limits (Enforced Ceiling): Defines a strict maximum ceiling (in Mbps or Gbps) that the traffic class can consume, preventing bursty traffic (like vMotion) from consuming spare capacity if an administrator wishes to cap it.

3. Admission Control & User-Defined Resource Pools

NIOC v3 introduces Bandwidth Allocation for Virtual Machines via user-defined Network Resource Pools:

• Administrators can associate a distributed port group with a custom network resource pool and assign guaranteed bandwidth reservations to mission-critical virtual machines.
• Admission Control: When a virtual machine configured with a network reservation powers on, vSphere evaluates cluster uplink capacity. If the required reservation cannot be satisfied within the supported host/pool allocation, admission can prevent VM power-on. Inspect the actual reservation, available allocation and eligible uplinks rather than treating every link's full line rate as reservable.

Official Reading

Use the official 2V0-16.25 objective guide for scope and Broadcom technical documentation for the installed 9.0 component. This independent lesson was reviewed October 18, 2026; actual deployment support depends on the applicable version and configuration.

Review Takeaways

• Active, standby and unused assignments govern the port group's eligible failover uplinks.
• Reliable beacon failure isolation needs at least three appropriately connected uplinks and is unsupported with IP-hash teaming.
• LBT on a vDS responds to physical uplink load without requiring link aggregation.
• Standard-switch VGT uses VLAN 4095; a distributed port group uses a VLAN trunking range.
• NIOC shares, reservations and limits govern bandwidth under their supported admission and contention rules.

See the official vSphere 9.0 Performance Best Practices for NIOC reservation limits and performance guidance.