5.2 Port Group Policies & NIC Teaming

5.2 Port Group Policies & NIC Teaming

NIC Teaming allows an ESXi host to combine multiple physical network adapters (vmnics) into a redundant, load-balanced logical trunk. Configuring appropriate teaming, load balancing, and failover policies at the virtual switch and port group levels is critical for high availability, deterministic performance, and physical switch compatibility.

---

Uplink Port Groups & Logical Mapping

On a vSphere Distributed Switch, physical NICs are not mapped directly to individual distributed port groups. Instead, they attach through an intermediate abstraction layer called the Uplink Port Group (dpuplink):

• Logical Uplinks: The vDS defines a set of logical uplinks (e.g., Uplink 1, Uplink 2, Uplink 3, Uplink 4).
• Host Physical Mapping: When an ESXi host is added to the vDS, its physical network interfaces are assigned to these logical uplinks (e.g., Host A assigns vmnic0 to Uplink 1 and vmnic1 to Uplink 2).
• Policy Inheritance & Overrides: Distributed port groups inherit their NIC teaming, failover, VLAN, and security policies from the vDS root by default. However, administrators can explicitly override these settings on any distributed port group or even on a per-port basis.

---

NIC Teaming Load Balancing Algorithms

vSphere provides four distinct load balancing algorithms. Each algorithm handles traffic distribution differently and imposes specific requirements on the upstream physical network switches.

+---------------------------------------------------------------------------------------------------+
|                                 vSphere NIC Teaming Algorithms                                    |
+------------------------------------+--------------------------+-----------------+-----------------+
| Algorithm                          | Upstream Switch Config   | vDS Exclusive?  | Balances Load?  |
+------------------------------------+--------------------------+-----------------+-----------------+
| Route based on originating port    | None (Independent ports) | No (vSS & vDS)  | No (Static pin) |
| Route based on source MAC hash     | None (Independent ports) | No (vSS & vDS)  | No (Static pin) |
| Route based on IP hash             | Static EtherChannel/LAG  | No (vSS & vDS)  | Multi-IP flows  |
| Route based on physical NIC load   | None (Independent ports) | Yes (vDS only)  | Yes (>75% load) |
+------------------------------------+--------------------------+-----------------+-----------------+

1. Route Based on Originating Virtual Port (Default)

This is the default load balancing algorithm for both vSS and vDS.

• Mechanism: When a virtual machine powers on or its vNIC connects to a port group, ESXi assigns it an internal virtual port ID. The VMkernel hashes this virtual port ID against the number of active physical uplinks in the team: Uplink = (Virtual Port ID) modulo (Number of Active Uplinks).
• Traffic Flow: All traffic from that specific vNIC will consistently traverse the assigned physical uplink. The binding remains static until the VM disconnects, migrates via vMotion, or the assigned physical uplink experiences a hardware link failure.
• Pros: Extremely low CPU overhead; requires zero physical switch configuration (the physical switch ports must be configured as standard, independent access or trunk ports).
• Cons: Does not balance based on bandwidth consumption. If three high-traffic database VMs happen to be hashed to Uplink 1 and three idle test VMs are hashed to Uplink 2, Uplink 1 can become heavily congested while Uplink 2 sits idle. The maximum bandwidth of any single vNIC is strictly limited to the speed of one physical uplink.

2. Route Based on Source MAC Hash

• Mechanism: The VMkernel computes an uplink assignment based on a hash of the virtual machine's virtual network adapter MAC address: Uplink = (Source MAC Hash) modulo (Number of Active Uplinks).
• Behavior: Very similar to originating virtual port. Because a VM's MAC address rarely changes, traffic from that VM is statically pinned to a single physical uplink.
• Use Case: Primarily beneficial in nested virtualization or specialized virtual appliances where a single virtual machine port hosts multiple distinct MAC addresses, distributing traffic from different nested MACs across different physical uplinks.
• Switch Requirement: None (independent physical switch ports).

3. Route Based on IP Hash

Route based on IP hash provides multi-uplink bandwidth distribution for individual virtual machines, but introduces strict architectural prerequisites.

• Mechanism: For every outgoing IP packet, the VMkernel inspects both the source IP address and the destination IP address, computes an XOR hash, and selects the physical uplink: Uplink = (Source IP XOR Destination IP) modulo (Number of Active Uplinks).
• Multi-Stream Aggregation: If a single virtual machine initiates multiple TCP/UDP sessions to different destination IP addresses (e.g., a web server responding to hundreds of distinct client IPs, or an NFS client communicating with multiple storage array IPs), each individual conversation can traverse a different physical uplink. This allows a single VM to consume the aggregated bandwidth of multiple physical NICs.
• Mandatory Configuration Requirements (Exam Traps):
  1. Physical Switch Configuration Required: The physical switch ports connected to the ESXi uplinks must be bundled into a static Link Aggregation Group (LAG / EtherChannel, IEEE 802.3ad static). Dynamic LACP is supported only on vDS, not on vSS.
  2. Active Uplink Requirement: All physical uplinks participating in the team must be placed in the Active failover list. Having Standby or Unused uplinks is strictly forbidden and invalidates the EtherChannel.
  3. Beacon Probing Incompatibility: Beacon probing cannot be used with Route based on IP hash. Beacon frames broadcast across an EtherChannel bundle are treated as loopback traffic or misdirected, causing false failovers.
  4. Multi-Chassis Restrictions: Physical switch ports must terminate on the same physical switch or on switches supporting multi-chassis link aggregation (such as Cisco vPC, Arista MLAG, or physical switch stacking). Connecting IP hash uplinks to independent, unstacked physical switches results in MAC table flapping and network black-holing.

4. Route Based on Physical NIC Load (Load-Based Teaming - LBT)

Route based on physical NIC load is an enterprise load balancing algorithm available exclusively on vSphere Distributed Switches.

• Mechanism: LBT initially assigns virtual machines to uplinks using the originating virtual port ID algorithm. However, the Host Proxy Switch continuously monitors the transmit and receive bandwidth utilization on all active physical uplinks over a 30-second sliding window.
• Dynamic Rebalancing: If the utilization of any physical uplink exceeds 75% of its maximum capacity for 30 consecutive seconds, ESXi identifies virtual machines consuming substantial bandwidth on that saturated uplink and migrates their virtual port bindings to an alternate, under-utilized physical uplink.
• Advantages: Provides true dynamic traffic distribution without requiring any proprietary multi-switch link aggregation, EtherChannel, or LACP configuration on the physical switches. Physical switch ports remain standard independent trunk ports.

---

Network Failover Detection Mechanisms

When a physical network failure occurs, the virtual switch must detect the outage immediately to divert traffic to surviving active or standby uplinks. vSphere supports two detection methods:

1. Link Status Only

• Operation: Relies strictly on the physical network adapter's hardware state (carrier signal / link pulse). It detects immediate hardware events such as an unplugged network cable, a failed SFP+/QSFP transceiver, or an unpowered physical switch.
• Limitation: Blind to upstream failures. If the physical access switch's uplink to the distribution/core layer fails, but the connection between the ESXi host and the access switch remains electrically active, "Link Status Only" still considers the link healthy. Outgoing frames sent onto that uplink are discarded by the disconnected access switch (black-holing).

2. Beacon Probing

• Operation: The ESXi VMkernel sends periodic broadcast beacon frames out of every physical uplink in the team, about once per second. All other physical adapters in the team listen for these beacon frames.
• Upstream Failure Detection: If an upstream switch fails or a VLAN is severed in the physical switching tier, the surviving uplinks will stop receiving beacon frames from the isolated uplink. ESXi identifies the failure even though the local physical link remains electrically "up".
• The Three-NIC Minimum Requirement (Exam Trap):
  • Beacon probing requires a minimum of three physical network adapters in the team.
  • Why? If a team has only two uplinks (vmnic0 and vmnic1) and a failure occurs, vmnic0 stops receiving beacons from vmnic1, and vmnic1 stops receiving beacons from vmnic0. The host knows a failure has occurred, but it cannot determine which of the two links is broken! With three or more physical uplinks, majority voting resolves the ambiguity: if vmnic0 and vmnic1 can see each other's beacons, but neither sees beacons from vmnic2, ESXi declares vmnic2 failed with a 2-to-1 consensus.

---

Uplink Failover Order & Status

Every port group maintains an ordered list of physical uplinks categorized into three operational states:

1. Active Uplinks: Physical adapters currently forwarding traffic according to the configured load balancing policy.
2. Standby Uplinks: Physical adapters kept in a dormant state. They carry no virtual machine or VMkernel traffic as long as at least one Active uplink remains functional. If all Active uplinks fail, the primary Standby uplink is immediately promoted to active status.
3. Unused Uplinks: Physical adapters permanently excluded from forwarding traffic for this specific port group. If an active uplink fails, traffic will never fail over to an Unused uplink.
   • Use Case: Segregating traffic over shared physical adapters. For example, a host with two physical NICs can set vmnic0 as Active and vmnic1 as Unused for Management traffic, while setting vmnic1 as Active and vmnic0 as Unused for vMotion traffic, guaranteeing complete physical bandwidth isolation.

---

Switch Notification & Failback Policies

Notify Switches (Reverse ARP)

• Configuration: Boolean setting (Yes or No), enabled (Yes) by default.
• Operation: When a failover occurs, or when a virtual machine is moved to a new host via vSphere vMotion, the ESXi host immediately sends out a series of Reverse ARP (RARP) broadcast frames on behalf of the affected virtual machines.
• Purpose: Informs the physical network switches to update their MAC address tables (CAM tables) instantly. Without switch notification, the physical switch would continue directing incoming packets to the old, failed physical switch port until the CAM table entry timed out (typically 300 seconds), causing prolonged application disruption.

Failback Policy (Network Flapping Prevention)

• Configuration: Boolean setting (Yes or No), enabled (Yes) by default.
• Failback = Yes: When a previously failed active physical adapter recovers, the virtual switch immediately shifts traffic from the standby adapter back to the restored active adapter.
• Failback = No: The virtual switch keeps traffic flowing across the standby adapter even after the primary active adapter recovers. The active adapter remains idle until the standby adapter fails or an administrator manually triggers a rebalance.

Exam Trap & Best Practice: In environments where physical switch ports or fiber connections experience intermittent link flapping (rapidly toggling between up and down states), setting Failback to No is recommended. With Failback set to Yes, a flapping link causes continuous failover and failback cycles, triggering endless RARP storms, MAC table thrashing, and dropped TCP connections.