3.1 Storage Access Protocols & Multipathing

3.1 Storage Access Protocols & Multipathing

VMware vSphere supports a versatile array of storage access protocols categorized broadly into block-based and file-based storage architectures. Block storage presents raw SCSI LUNs directly to the ESXi host, which then formats them with the Virtual Machine File System (VMFS). File-based storage mounts remote filesystems directly over a local area network (LAN) via Network File System (NFS), where file locking and metadata allocation are coordinated between ESXi and the NAS storage controller.

Selecting, configuring, and troubleshooting these protocols requires understanding their physical topologies, addressing schemes, network constraints, and how the VMkernel manages redundant paths using the Pluggable Storage Architecture (PSA).

---

Block Storage Protocols: FC, FCoE, and iSCSI

Block-level storage protocols encapsulate SCSI commands across specialized SAN fabrics or standard Ethernet networks. In all block storage models, the ESXi host acts as the SCSI Initiator, while the storage processor represents the SCSI Target.

1. Fibre Channel (FC)

Fibre Channel is a high-speed, lossless, dedicated serial network designed specifically for storage communication. In enterprise vSphere environments, FC remains widely deployed due to its deterministic latency, hardware offloading, and isolation from standard IP traffic.

• Addressing & Identity:
  • WWNN (World Wide Node Name): A 64-bit unique identifier (16 hexadecimal digits) assigned to an entire device or multi-port Host Bus Adapter (HBA).
  • WWPN (World Wide Port Name): A 64-bit unique identifier assigned to each individual physical port on the HBA. In vSphere administration, SAN zoning and LUN masking are configured using WWPNs, never WWNNs.
• SAN Fabric Architecture & Zoning:
  • Fibre Channel topologies implement redundant, physically isolated fabrics (traditionally referred to as Fabric A and Fabric B) to eliminate any single point of failure.
  • Zoning: Enforced on Fibre Channel switches to restrict communication between initiators and targets. VMware strongly recommends single-initiator single-target zoning (or single-initiator multi-target zoning if switch scalability requires it). Single-initiator zoning ensures that Registered State Change Notifications (RSCNs)—broadcast when a port goes offline or reboots—are confined only to that specific initiator, preventing fabric-wide disruption or link-state storms.

2. Fibre Channel over Ethernet (FCoE)

FCoE encapsulates native Fibre Channel frames inside standard IEEE 802.3 Ethernet frames (EtherType 0x8906), allowing local area network and storage area network traffic to converge across shared 10GbE, 25GbE, or 100GbE physical cabling.

• Lossless Ethernet Prerequisites (Converged Enhanced Ethernet - CEE):
  • Standard Ethernet permits packet drops during congestion; Fibre Channel protocol cannot tolerate dropped frames. FCoE mandates a lossless fabric utilizing Priority-based Flow Control (PFC, IEEE 802.1Qbb), which pauses traffic on dedicated class-of-service queues (typically CoS 3) without affecting ordinary IP network traffic.
  • Enhanced Transmission Selection (ETS, IEEE 802.1Qaz) guarantees strict bandwidth allocation among traffic types on the converged link.
  • Data Center Bridging Exchange (DCBX) automatically negotiates PFC and ETS parameters between ESXi Converged Network Adapters (CNAs) and top-of-rack switches.
• FCoE Initialization Protocol (FIP, EtherType 0x8914): Discovers FCoE VLANs, establishes virtual point-to-point links, and handles SAN fabric logins (FLOGI).
• Adapter Architectures: Hardware FCoE converged network adapters (CNAs) offload FIP and frame processing. vSphere 8.0 removed the software FCoE adapter, so FCoE on ESXi 8 requires hardware FCoE adapters.

3. Internet Small Computer Systems Interface (iSCSI)

iSCSI transports SCSI commands over TCP/IP networks via port 3260, enabling storage area networks to operate over commodity switches, routers, and Ethernet adapters.

• Naming Formats:
  • IQN (iSCSI Qualified Name): The standard naming format, following the schema iqn.yyyy-mm.naming-authority:unique-name (for example, iqn.1998-01.com.vmware:esxi-host01-2a4b6c8d).
  • EUI (Extended Unique Identifier): An IEEE-derived format consisting of eui. followed by a 16-character hexadecimal string representing a 64-bit unique identifier.
• Target Discovery Mechanisms:
  • Static Discovery: The administrator manually enters the target's IP address and exact IQN.
  • Dynamic Discovery (SendTargets): The administrator supplies the IP address and port of the target portal. The ESXi host issues a SendTargets request, and the storage array dynamically responds with a list of all available target IQNs and associated portal IPs.
  • Internet Storage Name Service (iSNS): An automated directory service where initiators and targets register and discover each other dynamically.
• Adapter Types:
  • Software iSCSI Adapter: Uses the standard ESXi VMkernel TCP/IP networking stack and standard physical network adapters (vmnics). Initiator processing is handled by host CPU cycles.
  • Dependent Hardware iSCSI Adapter: A specialized NIC (e.g., Broadcom or Marvell/QLogic) that offloads iSCSI processing to the hardware ASIC, but depends on the ESXi VMkernel network configuration for IP address assignment and network configuration.
  • Independent Hardware iSCSI Adapter: A dedicated HBA that handles the entire iSCSI and TCP/IP stack on the card itself, configuring its own IP addressing and gateway outside of the ESXi VMkernel virtual switch configuration.

+-----------------------------------------------------------------------------+
|                   iSCSI Network Port Binding Architecture                   |
+-----------------------------------------------------------------------------+
|                                                                             |
|   [ Software iSCSI Initiator ]                                              |
|             |                                                               |
|             +-------------------------+                                     |
|             |                         |                                     |
|      [ vmk1 (192.168.10.11) ]   [ vmk2 (192.168.10.12) ]                    |
|             |                         |                                     |
|       Active: vmnic0            Active: vmnic1                              |
|       Unused: vmnic1            Unused: vmnic0                              |
|             |                         |                                     |
|      +--------------+          +--------------+                             |
|      | Physical SW1 |          | Physical SW2 |  <-- Same L2 Subnet         |
|      +--------------+          +--------------+      (192.168.10.0/24)      |
|             \                         /                                     |
|              \                       /                                      |
|               +----------+----------+                                       |
|                          |                                                  |
|                 [ Storage Array ]                                           |
|             Portal IP: 192.168.10.100                                       |
+-----------------------------------------------------------------------------+

Critical Architecture Rule: iSCSI Network Port Binding

When using the Software iSCSI Adapter, multipathing requires connecting the software initiator to multiple VMkernel ports bound to individual physical adapters. This process is called Network Port Binding.

---

Network File System (NFS): v3 vs. v4.1

NFS provides file-based shared storage by mounting remote file exports over standard TCP/IP. ESXi supports both NFS version 3 and NFS version 4.1, but the protocols exhibit radically different architectural characteristics and locking semantics.

1. NFS Version 3

• Transport & Architecture: A largely stateless protocol whose data traffic uses TCP port 2049, with helper RPC services (such as MOUNT and the portmapper) used when the export is mounted.
• Locking Mechanism: ESXi does not use the Network Lock Manager (NLM) with NFS 3. It uses VMware's own client-side locking: hidden .lck-<ID> files that each host creates on the export and refreshes while it holds the lock.
• Multipathing Limitations: NFS v3 does not support multipathing; each datastore is mounted from one server address, and redundancy comes from NIC teaming on the virtual switch. vSphere 8.0 Update 1 added nConnect, which opens several TCP connections to the same server address for more throughput, but it is not multipathing across server addresses.
• Security: Relies on AUTH_SYS (UNIX style UID/GID authentication). Security is enforced primarily by export rules on the NAS server (restricting access to specific ESXi host IP addresses) and root squashing configurations.

2. NFS Version 4.1

• Transport & Architecture: A stateful protocol running exclusively over a single well-known port (TCP 2049), greatly simplifying firewall rule sets.
• In-Band Locking: File locking is integrated directly into the NFS v4.1 protocol stream via stateful leases, completely eliminating the need for NLM or proprietary .lck sideband files.
• Multipathing & Session Trunking: NFS v4.1 natively supports Session Trunking. An ESXi host can establish multiple parallel TCP connections across multiple VMkernel ports to one or more IP addresses presented by the NAS storage cluster. This delivers true active/active I/O load balancing and rapid path failover.
• Kerberos Security: ESXi supports two Kerberos modes for NFS 4.1:
  • krb5: Kerberos authentication.
  • krb5i: Kerberos authentication plus data integrity checking. ESXi does not support krb5p (full payload encryption).

Operational Caveats & Migration Rules

• No In-Place Upgrades: An existing NFS v3 datastore cannot be upgraded in-place to NFS v4.1. The volume must be unmounted and remounted as NFS v4.1, or a new NFS v4.1 datastore must be created and virtual machines moved using Storage vMotion.
• Locking Conflict Hazard: Never mount the same underlying storage export to some ESXi hosts using NFS v3 and to other hosts using NFS v4.1. Because their locking engines are completely incompatible, simultaneous access will result in silent data corruption.
• Feature Compatibility: While NFS 4.1 supports Kerberos and session trunking, certain vSphere features (such as Storage DRS and Storage I/O Control) have specific constraints or limited support depending on the exact vSphere release.

---

Pluggable Storage Architecture (PSA) & Multipathing

The Pluggable Storage Architecture (PSA) is an open, modular framework within the VMkernel that manages storage multipathing and device discovery. The PSA coordinates communication between the physical HBAs, the operating system kernel, and storage controllers.

+-----------------------------------------------------------------------------+
|                   Pluggable Storage Architecture (PSA)                      |
+-----------------------------------------------------------------------------+
|                                                                             |
|                        [ Virtual Machine (VMDK) ]                           |
|                                    |                                        |
|                           [ SCSI Virtual Layer ]                            |
|                                    |                                        |
|   +--------------------------------+------------------------------------+   |
|   | VMkernel PSA                                                        |   |
|   |                                                                     |   |
|   |   +-------------------------------------------------------------+   |   |
|   |   | Native Multipathing Plugin (NMP)                            |   |   |
|   |   |                                                             |   |   |
|   |   |    [ Storage Array Type Plugin ]    [ Path Selection Plugin ] |   |   |
|   |   |               (SATP)                         (PSP)          |   |   |
|   |   |     - Monitors path state           - Selects I/O path      |   |   |
|   |   |     - Handles ALUA state changes    - Fixed / MRU / RR      |   |   |
|   |   +-------------------------------------------------------------+   |   |
|   +--------------------------------+------------------------------------+   |
|                                    |                                        |
|                    +---------------+---------------+                        |
|                    |                               |                        |
|            [ HBA Path 1 ]                  [ HBA Path 2 ]                   |
|                    |                               |                        |
|       +------------+------------+     +------------+------------+           |
|       | SAN Fabric Switch A     |     | SAN Fabric Switch B     |           |
|       +------------+------------+     +------------+------------+           |
|                    \                               /                        |
|                     +--------------+--------------+                         |
|                                    |                                        |
|                        [ Storage Controller ]                               |
+-----------------------------------------------------------------------------+

Native Multipathing Plugin (NMP)

By default, the VMkernel provides the Native Multipathing Plugin (NMP). Third-party vendors can write custom Multipathing Plugins (MPPs), but NMP remains the standard for the vast majority of enterprise arrays. NMP delegates storage management tasks to two sub-plugins:

1. Storage Array Type Plugins (SATP)

SATPs manage array-specific hardware characteristics. They are responsible for:

• Monitoring physical path health and detecting link failures.
• Executing failover operations to activate standby paths.
• Responding to Asymmetric Logical Unit Access (ALUA) state changes reported by the array controllers.

Common SATPs and ALUA States:

• VMW_SATP_ALUA: Used for all modern storage arrays supporting the SCSI ALUA standard (T10 SPC-3). ALUA defines four path states:
  1. Active/Optimized (AO): Primary path providing lowest latency and highest throughput.
  2. Active/Non-Optimized (ANO): Secondary path routed through a passive controller; traversing this path incurs internal inter-controller bus latency.
  3. Standby: Path cannot process I/O until explicitly transitioned to active.
  4. Unavailable: Path cannot be accessed.
• VMW_SATP_DEFAULT_AA: Used for symmetric Active-Active arrays where any controller processes I/O with identical performance.
• VMW_SATP_DEFAULT_AP: Used for legacy Active-Passive arrays where passive paths drop I/O unless an explicit trespass command activates them.
• VMW_SATP_LOCAL: Used for local SAS/SATA devices claimed by NMP. Local NVMe devices and NVMe over Fabrics devices are normally claimed by the VMware High-Performance Plug-in (HPP) instead of NMP.

2. Path Selection Plugins (PSP)

PSPs determine which specific physical path is used to dispatch each I/O request:

• Most Recently Used (VMW_PSP_MRU):
  • The host selects the path used during boot or discovery. It continues routing all I/O through this path until the path fails.
  • Upon failure, it switches to an alternate healthy path. Critical Rule: When the original path recovers, MRU does not fail back. This prevents "path thrashing" on Active-Passive arrays where rapid failbacks would degrade controller performance.
  • Default PSP for: Legacy Active-Passive arrays.
• Fixed (VMW_PSP_FIXED):
  • The host uses a designated Preferred Path configured by the administrator.
  • If the preferred path fails, the host fails over to an alternate path.
  • Critical Rule: As soon as the preferred path recovers, Fixed immediately fails back to the preferred path.
  • Default PSP for: Non-ALUA Active-Active arrays.
• Round Robin (VMW_PSP_RR):
  • The host distributes I/O requests across all currently available Active/Optimized paths using a rotation algorithm.
  • Default Switching Limit: By default, VMW_PSP_RR switches to the next path after 1,000 I/O operations (the iops limit type). A bytes limit type (default 10 MB) also exists, and vSphere 6.7 Update 1 and later offer a latency-based Round Robin mode that prefers the path with the lowest measured latency.
  • Round Robin Performance Optimization: On high-performance all-flash arrays, sending 1,000 consecutive I/Os down one path before switching can leave other paths underused. Many array vendors therefore recommend an I/O limit of 1, which alternates paths on every I/O. Apply it only when your array vendor recommends it:

    esxcli storage nmp psp roundrobin deviceconfig set --type=iops --iops=1 --device=naa.6000eb31b37d425b000000000000001a
    

---

Storage Protocol Architecture Comparison