5.4 Network I/O Control (NIOC) & TCP/IP Stacks

5.4 Network I/O Control (NIOC) & TCP/IP Stacks

Modern data center architectures converge multiple disparate network traffic types—such as Management, vMotion, vSAN, iSCSI, and virtual machine workloads—onto high-bandwidth physical network interfaces (10GbE, 25GbE, and 100GbE). To prevent high-volume burst traffic from starving latency-sensitive services, vSphere provides Network I/O Control (NIOC). Complementing NIOC, vSphere provides Dedicated TCP/IP Stacks to separate routing domains and default gateways across host services.

---

Network I/O Control version 3 (NIOC v3) Architecture

Network I/O Control version 3 is an enterprise QoS mechanism available exclusively on vSphere Distributed Switches. (A vDS created with Network Offloads compatibility for DPUs has NIOC disabled; see Section 1.3.) In converged networking designs where distinct traffic flows share common physical uplinks, an unconstrained operation (such as a massive parallel vMotion migration or storage array backup) can saturate physical link buffers and induce severe packet drops for vSAN storage or management heartbeats.

NIOC v3 guarantees predictable quality of service by enforcing bandwidth scheduling directly at the vDS transmission queue layer.

System Traffic Types Managed by NIOC

NIOC v3 recognizes and independently schedules nine predefined system traffic types:

1. vSphere vMotion: Live virtual machine state and memory transfer.
2. Management Traffic: Host-to-vCenter communication, HA heartbeats, and ESXCLI/API calls.
3. vSAN Traffic: Distributed storage synchronization, read/write I/O, and witness heartbeats.
4. Fault Tolerance (FT) Logging: Lockstep execution synchronization between FT VM pairs.
5. iSCSI Traffic: Software and hardware iSCSI block storage protocols.
6. NFS Traffic: Network File System datastore storage I/O.
7. vSphere Replication (VR): Asynchronous replication change streams.
8. vSphere Backup (NFC): Network File Copy traffic utilized by backup appliances.
9. Virtual Machine Traffic: Production and non-production guest OS traffic flows.

---

Bandwidth Allocation Primitives: Shares, Reservations, and Limits

NIOC v3 controls traffic through three parameters:

+---------------------------------------------------------------------------------------------------+
|                                 NIOC v3 Bandwidth Primitives                                      |
+-----------------+--------------------------+-----------------------+------------------------------+
| Primitive       | Unit of Measurement      | When Enforced?        | Operational Impact           |
+-----------------+--------------------------+-----------------------+------------------------------+
| Shares          | Low (25) / Normal (50)   | Only during physical  | Relative ratio of excess     |
|                 | High (100) / Custom      | link congestion       | bandwidth distribution       |
+-----------------+--------------------------+-----------------------+------------------------------+
| Reservations    | Megabits per second      | Constantly guaranteed | Minimum guaranteed bandwidth;|
|                 | (Mbps) or % of link      | even during failure   | subject to 75% max cap       |
+-----------------+--------------------------+-----------------------+------------------------------+
| Limits          | Megabits per second      | Constantly enforced   | Hard throughput ceiling;     |
|                 | (Mbps) or Gbps           | at all times          | cannot burst past limit      |
+-----------------+--------------------------+-----------------------+------------------------------+

1. Shares

• Mechanism: Shares define the relative priority of a traffic type only when a physical uplink experiences congestion (100% saturation). Under uncongested conditions, any traffic type can consume up to 100% of the available uplink bandwidth.
• Predefined Ratios: Shares follow a standard 1:2:4 ratio:
  • Low: 25 shares
  • Normal: 50 shares
  • High: 100 shares
  • Custom: Explicit numerical value
• Contention Math: If vSAN traffic is set to High (100 shares) and vMotion traffic is set to Normal (50 shares), and both contend for a saturated 10GbE uplink, vSAN is guaranteed 100 / (100 + 50) = 66.7% of the link capacity (6.67 Gbps), while vMotion receives 50 / (100 + 50) = 33.3% (3.33 Gbps).

2. Reservations

• Mechanism: A reservation specifies a guaranteed minimum bandwidth allocation in Megabits per second (Mbps) or as a percentage of physical link capacity. Unlike shares, a reservation is statically guaranteed, ensuring that latency-sensitive traffic (such as vSAN or Management) always receives sufficient throughput even during catastrophic link saturation.
• Cluster-Wide Enforcement & Admission Control: In NIOC v3, reservations can be applied cluster-wide. When a virtual machine associated with a network resource pool powers on, vSphere evaluates whether the underlying host has sufficient physical bandwidth reservation capacity. If the reservation cannot be satisfied, the VM power-on operation is blocked by admission control.
• The 75% Maximum Reservation Rule (Exam Trap):
  • NIOC v3 enforces a strict maximum reservation ceiling: The sum of all reservations for system traffic types and VM resource pools cannot exceed 75% of the capacity of the lowest-speed physical uplink in the team.
  • Example: On an ESXi host with two 10 Gbps uplinks, the maximum allocatable reservation pool is 7.5 Gbps (75% of 10 Gbps). The remaining 25% (2.5 Gbps) is permanently unreserved to accommodate system overhead, burst traffic, and unreserved traffic.

3. Limits

• Mechanism: A limit imposes a strict maximum throughput ceiling on a traffic type or VM network resource pool. Even if the physical network adapters are 90% idle, a traffic type capped by a limit cannot exceed its configured threshold.
• Use Case: Restricting aggressive non-critical traffic—such as developmental VM replication or backup copying—from consuming excess bandwidth.

---

User-Defined Virtual Machine Network Resource Pools

In addition to system traffic types, NIOC v3 allows administrators to create custom Virtual Machine Network Resource Pools:

• Custom Tiers: Administrators create resource pools (e.g., Production-Tier, Dev-Test-Tier) and assign shares, reservations, and limits to each pool.
• Distributed Port Group Association: Individual distributed port groups are mapped to these network resource pools. All virtual machines connected to that distributed port group automatically draw bandwidth from that pool's quota.
• Single Root I/O Virtualization (SR-IOV) Exception: Virtual machines configured with SR-IOV bypass the VMkernel virtual switch entirely, connecting directly to the physical NIC's virtual functions (VF). Consequently, SR-IOV workloads cannot be shaped, monitored, or throttled by NIOC v3.

---

Dedicated TCP/IP Stacks in vSphere

In early vSphere versions, an ESXi host operated with a single unified TCP/IP stack. All VMkernel adapters (vmk0, vmk1, vmk2) shared a single routing table, a single DNS configuration, and a single system default gateway. This created major architectural limitations when vMotion or IP storage traffic needed to be routed across Layer-3 network boundaries without traversing the management gateway.

Modern vSphere implements Dedicated TCP/IP Stacks to provide complete routing isolation.

+-------------------------------------------------------------------------+
|                         ESXi Host VMkernel Stacks                       |
+--------------------+---------------------+------------------------------+
| Default Stack      | vMotion Stack       | Provisioning Stack           |
| - Routing Table A  | - Routing Table B   | - Routing Table C            |
| - Gateway: vmk0    | - Gateway: vmk1     | - Gateway: vmk2              |
| - DNS: Corporate   | - No DNS needed     | - Optional DNS               |
| - Mgmt, HA, Syslog | - vMotion Only      | - Cold VM clone, snapshot    |
+--------------------+---------------------+------------------------------+

Built-in System TCP/IP Stacks

Every ESXi host includes three pre-configured TCP/IP stacks:

1. Default TCP/IP Stack:
   • Handles host management traffic (vmk0), DNS lookups, Active Directory authentication, syslog forwarding, SNMP, NTP, and any VMkernel interface not explicitly bound to a dedicated stack.
   • Uses the host's system default gateway.
2. vMotion TCP/IP Stack:
   • Exclusively handles vSphere vMotion traffic.
   • Features its own isolated routing table, default gateway, and ARP cache.
   • Primary Benefit: Enables Layer-3 routed vMotion across data centers or routed campus clusters without requiring stretched Layer-2 VLANs, complex static routes, or management gateway traversal.
3. Provisioning TCP/IP Stack:
   • Carries provisioning traffic, which uses Network File Copy (NFC): cold migration of powered-off VMs, cloning, and snapshot migration. It can also carry the NFC portion of long-distance vMotion.
   • Isolates bulk image transfer traffic from production vMotion and management networks.

Custom TCP/IP Stacks

For specialized networking requirements—such as multi-tenant isolation, dedicated backup networks, or routed iSCSI/NFS storage arrays—administrators can define custom TCP/IP stacks:

• Creation via CLI: Custom stacks are created via the ESXi command line:

  esxcli network ip netstack add -N BackupStack
  

• Configuration: Once created, administrators can assign a dedicated default gateway, custom DNS servers, and specific MTU settings to the custom stack.
• VMkernel Binding: A new VMkernel adapter is created and bound directly to the custom TCP/IP stack, providing complete routing table isolation from the default management network.