3.1 vSphere Core Storage Architecture: VMFS, NFS, & vVols

3.1 vSphere Core Storage Architecture: VMFS, NFS, & vVols

Executive Overview: Shared storage represents a bedrock architectural pillar within VMware Cloud Foundation (VCF) 9.0. While VMware vSAN serves as the default hyperconverged storage fabric for the Management Domain and principal Workload Domains, enterprise cloud deployments routinely integrate external shared storage arrays. Whether provisioning high-density Supplemental Storage or supporting specialized database workloads on dedicated arrays, administrators must master the three foundational vSphere storage architectures: the Virtual Machine File System (VMFS), the Network File System (NFS), and vSphere Virtual Volumes (vVols), alongside the Pluggable Storage Architecture (PSA) that governs fabric multipathing.

---

Shared Storage in VMware Cloud Foundation 9.0

In VMware Cloud Foundation 9.0, storage architectures are classified by their operational role in the software-defined datacenter:

• Principal Storage: The primary storage cluster designated during domain bring-up and workload domain provisioning. In the standardized VCF Management Domain, vSAN is the mandatory principal storage. However, in Virtual Infrastructure (VI) Workload Domains, SDDC Manager supports external storage options—including Fibre Channel (FC) VMFS, Network File System (NFS), and Virtual Volumes (vVols)—as principal storage.
• Supplemental Storage: Additional storage capacity attached to existing clusters post-deployment. Supplemental storage enables administrators to scale storage capacity independently of compute nodes, attach legacy arrays for workload migration, or dedicate isolated flash tiers to transactional databases.

To decouple virtual workloads from physical hardware idiosyncrasies, the ESXi hypervisor abstracts physical disks and file exports into logical Datastores. Virtual machine virtual disks (.vmdk files) reside within these datastores, serviced through standardized SCSI, NVMe, or file commands regardless of the backend storage fabric.

---

Enterprise Storage Fabrics & Transport Protocols

Modern enterprise storage relies on four primary fabric categories, each presenting distinct architectural characteristics and configuration rules:

1. Fibre Channel (FC) Fabrics

Fibre Channel remains the benchmark for deterministic, low-latency, lossless block storage. Modern deployments utilize 16 Gbps, 32 Gbps, or 64 Gbps Host Bus Adapters (HBAs) and SAN switches:

• Addressing: Devices are identified by 64-bit World Wide Port Names (WWPNs) and World Wide Node Names (WWNNs).
• Zoning: Storage fabrics must be partitioned into logical security zones. VMware best practice dictates Single-Initiator, Multi-Target Zoning. Each zone contains exactly one ESXi HBA port (initiator) paired with one or more array target ports. This isolates fabric events, preventing registered state change notifications (RSCNs) from disrupting other hypervisors.
• Fabric Login: HBAs undergo Fabric Login (FLOGI) to receive a 24-bit Fibre Channel ID (FCID), followed by Port Login (PLOGI) and Process Login (PRLI) to establish SCSI communication sessions with array target ports.

2. Fibre Channel over Ethernet (FCoE)

FCoE encapsulates standard Fibre Channel frames into Ethernet packets, enabling a unified 10 GbE or 25 GbE network infrastructure using Converged Network Adapters (CNAs). FCoE mandates a Lossless Ethernet fabric implementing Data Center Bridging (DCB) extensions:

• Priority Flow Control (PFC, IEEE 802.1Qbb): Pauses specific traffic classes (typically CoS 3 for FCoE) to prevent buffer overflows without dropping frames.
• Enhanced Transmission Selection (ETS, IEEE 802.1Qaz): Guarantees minimum bandwidth allocation to storage traffic during network congestion.

3. iSCSI Fabrics & Port Binding Rules

iSCSI encapsulates SCSI command descriptor blocks (CDBs) inside TCP/IP packets over standard TCP port 3260. ESXi supports Software iSCSI (utilizing the hypervisor TCP/IP stack) and Hardware iSCSI HBAs:

• Addressing: Targets and initiators are designated using iSCSI Qualified Names (IQNs) or Enterprise Unique Identifiers (EUIs).
• Discovery: Configured via Static Discovery (manual IP/IQN entry) or Dynamic Discovery (SendTargets), where querying a single array portal returns all available target IQNs.
• Network Port Binding Rules: Port binding attaches ESXi VMkernel adapters directly to the software iSCSI stack to enable multipathing. However, misconfigurations lead to severe storage outages. Administrators must memorize these strict architectural constraints:
  1. Port binding requires that all VMkernel ports and storage target portals reside in the same Layer 2 broadcast domain and IP subnet.
  2. Each VMkernel port bound to iSCSI must be mapped to a dedicated physical uplink (1:1 mapping with no standby or failover NICs).
  3. Crucial Exam Rule: Port binding must NEVER be used when storage target portals reside on routed, different IP subnets. Using port binding across routed subnets overrides standard TCP routing, causing path flapping, erratic failover, and disconnects.
• Jumbo Frames (MTU 9000): Maximizes throughput and lowers host CPU utilization by reducing packet processing overhead. The MTU must be set to 9000 end-to-end: on the VMkernel port, the vSphere Distributed Switch (vDS), the physical switch fabric, and the storage array network interfaces.

4. NVMe-over-Fabrics (NVMe-oF)

NVMe-oF represents the latest generation of high-throughput, ultra-low-latency block storage, replacing legacy SCSI architectures. ESXi supports NVMe-over-FC, NVMe-over-RDMA (RoCEv2), and NVMe-over-TCP. By eliminating SCSI command conversion and leveraging up to 64,000 queues with 64,000 commands per queue, NVMe-oF matches direct-attached NVMe latency over shared enterprise fabrics.

---

VMFS-6 Architectural Deep-Dive

The Virtual Machine File System version 6 (VMFS-6) is VMware's enterprise clustered file system, engineered specifically for parallel virtual machine execution across ESXi clusters.

Architectural Enhancements of VMFS-6

┌────────────────────────────────────────────────────────────────────────┐
│                     VMFS-6 CLUSTERED FILE SYSTEM                       │
├──────────────────────────────────┬─────────────────────────────────────┤
│ Standardized 1 MB File Blocks    │ 64 KB Small Sub-Blocks              │
│ (Supports VMDKs up to 64 TB)     │ (Metadata & Tiny Files Efficiency)  │
├──────────────────────────────────┼─────────────────────────────────────┤
│ Native 4Kn Drive Support         │ Automatic Background UNMAP          │
│ (4096-byte Native Sector Media)  │ (Continuous Stranded Space Recovery)│
├──────────────────────────────────┴─────────────────────────────────────┤
│ ATS-Only Heartbeating (Atomic Test and Set Hardware Locking)           │
└────────────────────────────────────────────────────────────────────────┘

1. Standardized 1 MB Block Size: Unlike legacy VMFS-3 which required variable block sizes (1 MB to 8 MB) to support larger files, VMFS-6 standardizes exclusively on a 1 MB file block size. VMFS-6 natively supports virtual disks up to 64 TB in size without block size adjustments.
2. 64 KB Sub-Block Allocation: For small files (such as .vmx configuration files, logs, and small directories), VMFS-6 introduces 64 KB sub-blocks. This eliminates the storage wastage that previously occurred when small files consumed full 1 MB blocks.
3. Native 4K (4Kn) Drive Support: VMFS-6 provides native support for 4K Native (4Kn) physical storage drives (4096-byte physical sectors) as well as 512-byte emulation (512e) drives. VMFS-5 only supported 512e and 512-byte native (512n) drives; formatting a true 4Kn LUN requires VMFS-6.
4. Automatic Space Reclamation (UNMAP): In VMFS-5, reclaiming dead storage blocks left behind by deleted, moved, or shrunk virtual disks was a manual, disruptive process requiring the CLI command esxcli storage vmfs unmap. VMFS-6 introduces automatic, asynchronous background space reclamation. ESXi continuously tracks deallocated blocks and sends SCSI UNMAP primitives to thin-provisioned storage arrays in the background. Administrators can tune reclamation priority via vSphere Client or CLI (none, low, medium, or high), throttling reclamation bandwidth between 25 MB/s and 2000 MB/s to prevent SAN saturation.
5. Atomic Test and Set (ATS) Heartbeat: Legacy VMFS used SCSI-2 reservation locks, which locked an entire physical LUN during metadata changes (creating lock contention when dozens of VMs resided on one LUN). VMFS-6 relies exclusively on ATS (VAAI primitive). ATS enables fine-grained sector-level hardware locking. VMFS-6 utilizes ATS-only heartbeating to verify host liveliness and renew on-disk file leases without disrupting active I/O on the LUN.

[!WARNING] Non-Disruptive Upgrade Limitation: There is no in-place software upgrade from VMFS-5 to VMFS-6. Because VMFS-6 features a completely redesigned on-disk metadata layout, 64 KB sub-blocks, and new alignment structures, upgrading requires creating a new VMFS-6 datastore and performing non-disruptive Storage vMotion migrations for all virtual machines residing on the VMFS-5 datastore.

vSphere Storage APIs for Array Integration (VAAI)

VMFS-6 heavily leverages VAAI block hardware acceleration primitives to offload intensive storage tasks to the array:

• Atomic Test and Set (ATS): Hardware-assisted locking for metadata updates and cluster heartbeats.
• Clone Blocks (XCOPY / Full Copy): Offloads VM cloning and Storage vMotion data copies to the array storage processors, consuming zero ESXi host CPU and zero SAN fabric bandwidth.
• Write Same (Zero Blocks): Offloads block zeroing when initializing EagerZeroedThick virtual disks, accelerating disk creation from minutes to seconds.
• Block Deletion (SCSI UNMAP): Informs thin-provisioned arrays that deleted VMDK blocks can be reclaimed into the global free storage pool.

---

NFS File Storage: NFS v3 vs. NFS v4.1

Network File System (NFS) provides file-level shared storage over standard IP networks. vSphere 9.0 supports both NFS v3 and NFS v4.1, but their operational architectures diverge significantly:

Detailed Protocol Comparison

1. Locking Mechanics: NFS v3 is inherently stateless; the server does not track client states. To prevent concurrent writes, ESXi implements client-side locking coordinated through the external Network Lock Manager (NLM) protocol over sideband ports. In contrast, NFS v4.1 is fully stateful; the storage server tracks all active file opens, byte-range locks, and leases within the main protocol connection, eliminating split-brain lock corruption.
2. Multipathing & Session Trunking: In NFS v3, an ESXi host establishes a single TCP connection to a single IP address on the NAS server. Multipathing is impossible; if multiple uplinks exist on the vSwitch, one uplink remains idle in an active/passive failover relationship. NFS v4.1 introduces Session Trunking. An ESXi host can open multiple parallel TCP sessions across different VMkernel interfaces to multiple Logical Interfaces (LIFs) or IP addresses on the NAS server, enabling true active/active load balancing and transparent sub-second path failover.
3. Kerberos Security Levels: NFS v4.1 supports enterprise-grade Kerberos authentication via three progressive modes:
   • krb5: DES/AES user authentication via Kerberos ticket validation.
   • krb5i: User authentication plus cryptographic data integrity checksums to prevent packet tampering.
   • krb5p: User authentication, data integrity, and full packet payload privacy (wire encryption).

[!IMPORTANT] NFS Interoperability Rule: You cannot mount an NFS export as NFS v3 on some ESXi hosts and as NFS v4.1 on other ESXi hosts in the same cluster. Mixing protocol versions on the same export breaks distributed file locking and will cause catastrophic virtual disk corruption.

---

Virtual Volumes (vVols) Architecture

vSphere Virtual Volumes (vVols) represents a fundamental paradigm shift from traditional LUN-centric or volume-centric storage to VM-centric software-defined storage.

┌────────────────────────────────────────────────────────────────────────┐
│                     VASA PROVIDER (CONTROL PLANE)                      │
│        (Out-of-band HTTPS: Exports Array Capabilities into SPBM)       │
├────────────────────────────────────────────────────────────────────────┤
│                 PROTOCOL ENDPOINT (DATA PLANE ROUTER)                  │
│    (In-band SCSI/NFS Sub-LUN I/O Demultiplexer: Replaces 256 LUN Limit) │
├────────────────────────────────────────────────────────────────────────┤
│             STORAGE CONTAINER (RAW ARRAY PHYSICAL POOL)                │
│     (Logical Pool presented as vVol Datastore - No Local Filesystem)   │
├──────────────────┬──────────────────┬─────────────────┬────────────────┤
│   Config-VVol    │    Data-VVol     │    Swap-VVol    │ Snapshot-VVol  │
│  (.vmx, logs)    │ (.vmdk 1:1 Disk) │  (Memory Swap)  │ (Point-in-Time)│
└──────────────────┴──────────────────┴─────────────────┴────────────────┘

The Core Components of vVols

1. Storage Container: A logical allocation of physical storage capacity created by the storage administrator on the physical SAN/NAS array. When presented to vCenter Server, it appears as a vVol Datastore. Crucially, no local filesystem (such as VMFS) is formatted onto a Storage Container. It remains raw array capacity managed dynamically by array firmware.
2. Virtual Volumes (vVols): Individual virtual machine files are encapsulated as native storage objects created directly on the physical array. Each component is an independent storage volume:
   • Config-VVol: A 4 GB storage volume formatted with a mini-filesystem that stores .vmx files, descriptor files, and VM logs.
   • Data-VVol: Corresponds 1:1 with each virtual machine disk (.vmdk). Read and write I/O is issued directly to this array volume.
   • Swap-VVol: Created when a VM powers on to hold virtual memory swap space, sized to match VM configured memory minus reservation.
   • Snapshot-VVol: Point-in-time branch volumes created natively by the array when taking snapshots, eliminating software redo-log overhead.
3. VASA Provider (Storage Provider): An out-of-band control-plane software service developed by the storage vendor (running embedded on storage controllers or as an external virtual appliance). It communicates with vCenter Server via HTTPS over VASA (vSphere APIs for Storage Awareness) 3.0 or 4.0 specifications. The VASA provider exports the physical array's hardware capabilities (e.g., RAID levels, encryption, compression, snapshot schedules, replication targets) into vCenter. These capabilities populate Storage Policy-Based Management (SPBM).
4. Protocol Endpoints (PE): The in-band data-path access point. ESXi hosts establish standard Fibre Channel, iSCSI, or NFS connections to the Protocol Endpoint. The PE acts as a high-performance I/O demultiplexer (sub-LUN addressing proxy), routing I/O to individual virtual volumes. Because hundreds of VMs communicate through a single PE, vVols completely bypasses the legacy vSphere limit of 256 or 512 physical SCSI LUNs per host.

[!NOTE] VASA Provider Resiliency: If the VASA Provider fails or loses network connectivity, all running virtual machines continue active I/O without interruption. Because active I/O traverses the Protocol Endpoint (data plane), data path traffic is insulated from control plane outages. However, control plane management operations (powering on VMs, creating new disks, cloning, taking snapshots, or changing SPBM policies) will fail until the VASA Provider recovers.

---

Pluggable Storage Architecture (PSA) & Multipathing Tuning

The ESXi Pluggable Storage Architecture (PSA) is a modular framework in the VMkernel that manages storage multipathing, failover, and load balancing.

                        ┌─────────────────────────┐
                        │       NATIVE NMP        │
                        └────────────┬────────────┘
                                     │
                ┌────────────────────┴────────────────────┐
                ▼                                         ▼
     ┌─────────────────────┐                   ┌─────────────────────┐
     │     SATP PLUGIN     │                   │     PSP PLUGIN      │
     │ (Array Failover)    │                   │ (Path Load Balance) │
     ├─────────────────────┤                   ├─────────────────────┤
     │ • VMW_SATP_ALUA     │                   │ • VMW_PSP_RR (Round)│
     │ • VMW_SATP_DEFAULT_AA│                  │ • VMW_PSP_FIXED     │
     │ • VMW_SATP_DEFAULT_AP│                  │ • VMW_PSP_MRU       │
     └─────────────────────┘                   └─────────────────────┘

Native Multipathing Plugin (NMP)

By default, ESXi utilizes the VMware Native Multipathing Plugin (NMP), which coordinates two sub-plugins for every storage device:

1. Storage Array Type Plugin (SATP): Array-specific plugin responsible for monitoring path health, detecting controller failures, executing hardware-specific failover instructions, and tracking target port group states (e.g., Active/Optimized, Active/Non-Optimized, Standby). Examples include VMW_SATP_ALUA (for Asymmetric Logical Unit Access arrays) and VMW_SATP_DEFAULT_AA (for true Active/Active arrays).
2. Path Selection Plugin (PSP): Responsible for selecting which physical storage path is utilized to transmit an individual I/O request:
   • Round Robin (VMW_PSP_RR): Automatically distributes I/O requests across all active, optimized paths. By default, ESXi sends 1,000 I/O operations (or 10 MB of data) down one path before switching to the next. On modern all-flash arrays, this default creates micro-burst queue bottlenecks.
   • Fixed (VMW_PSP_FIXED): Designates one specific path as the Preferred Path. All I/O flows through this path. If the preferred path fails, traffic fails over to an alternate path; when the preferred path recovers, ESXi automatically fails back to it.
   • Most Recently Used (VMW_PSP_MRU): Uses the current active path until it fails. When failure occurs, it selects an alternate path and does NOT fail back when the original path recovers. MRU is mandatory on legacy Active/Passive arrays to prevent "path thrashing" (ping-ponging LUN ownership between controllers).

Performance Tuning: The Round Robin iops=1 Best Practice

For high-performance all-flash Fibre Channel and iSCSI arrays running under VMW_PSP_RR, VMware and storage vendor best practices mandate tuning the path switching frequency from 1,000 IOPS down to 1 IOPS:

# Query current PSP configuration for a storage device
esxcli storage nmp device list -d naa.6000eb3a123456780000000000000001

# Configure Round Robin to switch paths every single I/O operation
esxcli storage nmp psp roundrobin deviceconfig set --type=iops --iops=1 --device=naa.6000eb3a123456780000000000000001

Tuning to iops=1 forces the VMkernel to interleave each I/O command across all available host HBAs and array controller ports. This maximizes queue depth parallelism, eliminates path saturation, and reduces average read/write latency by up to 30-40%.

---

VCF Core Storage Exam Pitfalls & Best Practices

[!IMPORTANT] Summary of Candidate Traps on 2V0-17.25:

1. In-Place VMFS Upgrades: Do not fall for questions asking how to upgrade VMFS-5 in-place. It is structurally impossible; you must create a new VMFS-6 datastore and Storage vMotion the workloads.
2. iSCSI Port Binding vs. Routing: Never configure iSCSI port binding when the initiator and target reside in different routed IP subnets. Port binding is strictly for single-subnet Layer 2 topologies.
3. VASA Provider Failure vs. PE Failure: Losing a VASA provider disrupts the control plane (power-on, snapshots, cloning fail), but ongoing VM disk I/O through Protocol Endpoints continues unaffected.
4. NFS Version Mixing: Never mount the same physical NFS export as NFS v3 on Host A and NFS v4.1 on Host B. This breaks distributed locking and corrupts data.
5. Round Robin Switch Default: The default Round Robin PSP switch frequency is 1,000 IOPS, but the production tuning best practice for all-flash arrays is 1 IOPS (--type=iops --iops=1).