4.1 Storage Pools, Storage Containers, and vDisk Virtual Architecture

Storage Pools, Storage Containers, and vDisk Virtual Architecture

Quick Summary: The Nutanix Acropolis Operating System (AOS) Distributed Storage Fabric (DSF) replaces legacy SAN and NAS architectures with a distributed, software-defined storage platform. Physical NVMe, SSD, and HDD media across all nodes aggregate into a cluster-wide Storage Pool. Administrators carve this capacity into Storage Containers—logical policy boundaries governing Redundancy Factor, compression, deduplication, and hypervisor datastore presentation. Virtual machines interact with storage through vDisks, which are logically divided into 1 MB vBlocks, organized as 1 MB extents, and stored physically in 1 MB or 4 MB extent groups, with thin provisioning enforced natively across the entire fabric.

---

1. Architectural Overview of the AOS Distributed Storage Fabric (DSF)

In conventional three-tier enterprise architectures, storage management is fragmented. Server virtualization hosts connect through redundant Fibre Channel (FC) or iSCSI host bus adapters (HBAs), traverse complex SAN switches, and map to dedicated storage arrays containing dual active-passive or active-active storage controllers. Scaling capacity or performance requires purchasing expensive controller shelves or expanding complex RAID groups, creating storage silos, administrative overhead, and predictable I/O bottlenecks.

The Nutanix Distributed Storage Fabric (DSF)—historically termed the Nutanix Distributed File System (NDFS)—eliminates the external storage array entirely. DSF pools local direct-attached storage (NVMe PCIe SSDs, SATA/SAS SSDs, and spinning HDDs) across all physical cluster nodes into a unified, shared storage fabric that scales linearly in lockstep with compute.

+-------------------------------------------------------------------------+
|                           Nutanix Cluster                               |
|                                                                         |
|   Node 1 (AHV/ESXi)          Node 2 (AHV/ESXi)          Node 3 (AHV/ESXi)|
|   +---------------+          +---------------+          +---------------+|
|   | VM1   |  CVM  |          | VM2   |  CVM  |          | VM3   |  CVM  ||
|   +-------+-------+          +-------+-------+          +-------+-------+|
|   | NVMe | SSD | HDD |       | NVMe | SSD | HDD |       | NVMe | SSD | HDD ||
+---+--------------------------+--------------------------+---------------+-+
    |                          |                          |                 
    +--------------------------+--------------------------+                 
                               v                                            
+-------------------------------------------------------------------------+
|                   Unified Distributed Storage Fabric                    |
|                  Cluster-Wide Storage Pool (All Tiers)                  |
+-------------------------------------------------------------------------+
|  Storage Container A (RF2, Inline Comp) | Storage Container B (RF3, Dedup) |
+-------------------------------------------------------------------------+
|  vDisk 1 (VM1-OS)                       | vDisk 2 (VM2-Data)               |
|  - 1 MB vBlocks                         | - 1 MB vBlocks                   |
|  - 1 MB Extents in 1/4 MB groups        | - 1 MB Extents in 1/4 MB groups  |
+-------------------------------------------------------------------------+

The Core Engine: The Stargate Process

At the foundation of DSF is the Stargate service. Stargate is the core data-path daemon that runs inside every Controller VM (CVM):

• I/O Interception: Stargate serves all read and write storage requests issued by virtual machines residing on its local hypervisor host.
• Data Transformation: It executes real-time storage optimizations including inline compression, checksum generation, fingerprinting for deduplication, and replication management.
• Physical Disk Management: Stargate communicates directly with physical media pass-through controllers (via LSI/Broadcom SAS controllers or direct PCIe NVMe bus access) to manage the underlying block devices.
• Distributed Coordination: Stargates communicate across the internal cluster network (eth0 or dedicated backplane eth1 interfaces) using high-efficiency Remote Procedure Calls (RPC) to orchestrate data placement, remote replication, and background data tiering.

---

2. Storage Pools: Physical Device Aggregation

A Storage Pool is the physical foundation of Nutanix DSF. It represents a cluster-wide aggregation of physical storage devices across all participating Controller VMs.

Characteristics of Storage Pools

• Cluster-Wide Span: Unlike traditional storage arrays where a pool or RAID group is tied to specific drive enclosures, a Nutanix Storage Pool spans across every physical drive in every node in the cluster.
• Multi-Tier Media Coexistence: A Storage Pool aggregates diverse physical drive types simultaneously:
  • NVMe Tier: Ultra-low latency, high-throughput non-volatile memory express solid-state drives connected directly to the PCIe bus.
  • SSD Tier: High-performance SATA or SAS solid-state drives.
  • HDD Tier: High-capacity spinning hard disk drives utilized for cold archival data.
• Single Pool Best Practice: In modern Nutanix AOS releases (version 5.x, 6.x, and beyond), Nutanix strongly recommends and defaults to a single Storage Pool per cluster. Creating multiple storage pools fragments IOPS aggregation, introduces administrative complexity, and reduces the efficiency of distributed self-healing and data tiering.
• Dynamic Expansion: When a new node is added to the cluster via Prism, its physical drives are automatically recognized, formatted with the Nutanix storage layout, and ingested into the existing Storage Pool without downtime or volume re-striping.

---

3. Storage Containers: Software-Defined Policy Boundaries

A Storage Container is a logical software-defined abstraction carved directly out of a Storage Pool. Storage containers do not carve out dedicated slices of physical media; rather, they serve as the management and policy boundary for virtual disks.

Software-Defined Policies Configured at the Container Level

When administrators create a Storage Container in Prism Element or Prism Central, they configure critical operational and data efficiency policies:

1. Redundancy Factor (RF):
   • Determines how many copies of data extents are maintained across independent nodes.
   • Containers can be configured for RF2 (two copies of data, surviving 1 failure) or RF3 (three copies of data, surviving 2 simultaneous failures).
2. Compression:
   • Inline Compression: Data is compressed in memory before being committed to persistent media. Recommended for sequential workloads and batch writes.
   • Post-Process Compression: Data is initially written uncompressed to the high-performance write buffer (OpLog) and subsequently compressed in the background by the Curator service after a configurable delay timer (default: 0 minutes for instant background processing).
   • Typical compression algorithms include LZ4 and Snappy, offering 1.5x to 3x space savings on compressible enterprise workloads.
3. Deduplication:
   • In-Cache Deduplication: Fingerprints read-heavy blocks and eliminates duplicates from CVM RAM and SSD read caches (ideal for Virtual Desktop Infrastructure / VDI boot storms).
   • Capacity Deduplication (Fingerprinting): SHA-1 fingerprints are generated for extents, and Curator merges identical physical extents in the Extent Store to conserve disk space.
4. Erasure Coding (EC-X):
   • A software-defined data protection mechanism based on Reed-Solomon encoding (e.g., RS 4+1, 4+2).
   • Merges cold, stable data extents across nodes into parity stripes, reducing storage overhead from 50% (RF2) to approximately 25%, while retaining full fault tolerance.
5. Capacity Reservations and Quotas:
   • Advertised Capacity: A soft or hard quota limit exposed to the hypervisor, preventing a single container from consuming the entire physical storage pool.
   • Reserved Capacity: Guarantees that a specified amount of physical storage is strictly held in reserve for that container, preventing over-commitment from exhausting critical production capacity.

Hypervisor Presentation Protocols

DSF provides multi-hypervisor and bare-metal flexibility by presenting storage containers across standard enterprise storage protocols:

• VMware ESXi: The storage container is mounted as an NFS v3 datastore across all ESXi hosts. The CVMs act as distributed NFS servers, with each ESXi host communicating with its local CVM over the standard VMkernel network.
• Microsoft Hyper-V: The storage container is presented as a high-availability Server Message Block (SMB 3.0) share.
• Nutanix AHV: AHV integrates natively with DSF. Storage containers are accessed directly through internal iSCSI loopback redirects on 127.0.0.1:3260, eliminating third-party filesystem encapsulation.
• Nutanix Volume Groups (VGs): Storage containers can present direct block-level iSCSI LUNs to virtual machines or bare-metal operating systems. This enables shared-disk clustering solutions such as Microsoft Windows Server Failover Clustering (WSFC) and Oracle Real Application Clusters (RAC).

[!NOTE] A single Storage Pool can host dozens of distinct Storage Containers simultaneously. For example, an administrator can maintain an Exchange-Container configured with inline compression and RF2 alongside a MissionCritical-DB-Container configured with RF3 and reserved capacity—both drawing from the same underlying physical drives.

---

4. vDisk Virtual Architecture: Blocks, Extent Slices, and Extents

Virtual machines do not write directly to raw flash sectors. Instead, AOS exposes virtual disks through a multi-tier abstraction hierarchy known as the vDisk architecture.

What is a vDisk?

In Nutanix terminology, a vDisk is any virtual storage entity larger than 512 KB created on the Distributed Storage Fabric. Examples include:

• Virtual machine hard disk files (.vmdk on ESXi, .vhdx on Hyper-V, or raw disk images on AHV).
• Virtual machine snapshot delta files.
• Nutanix Volume Group block disks exposed via iSCSI.

The DSF Geometric Hierarchy: vBlocks, Extents, and Extent Groups

To manage petabytes of data with sub-millisecond response times, DSF decomposes each vDisk into a granular hierarchy. Learn the four terms in order, because they are frequently quoted incorrectly:

+-------------------------------------------------------------------------+
|                        vDisk (e.g., 500 GB disk)                        |
+-------------------------------------------------------------------------+
|  1 MB vBlock 0  |  1 MB vBlock 1  |  1 MB vBlock 2  |  1 MB vBlock 3 |..|
+-----------------+-----------------+-----------------+----------------+--+
          |                 |                 |
          v                 v                 v
+-------------------------------------------------------------------------+
|                          Extents  (1 MB each)                           |
|   logically contiguous data made up of contiguous vBlocks; read and      |
|   modified on a sub-extent basis known as a SLICE                        |
+-------------------------------------------------------------------------+
          |                 |                 |
          v                 v                 v
+-------------------------------------------------------------------------+
|              Extent Groups  (1 MB or 4 MB)  -> Extent Store             |
|   physically contiguous stored chunks, held as files on a device         |
|   owned by a CVM; extents from different vDisks may share a group        |
+-------------------------------------------------------------------------+

1. vBlock (vDisk Block) — 1 MB:
   • The vDisk address space is divided into uniform 1 MB units called vBlocks. A 100 MB vDisk therefore consists of 100 vBlocks.
   • When the guest OS issues I/O at a logical block address, DSF resolves which vBlock covers that offset.
2. Extent — 1 MB:
   • An extent is a 1 MB logically contiguous piece of data, made up of a number of contiguous vBlocks. How many depends on the block size configured by the guest operating system.
   • Extents are the unit that gets compressed, deduplicated, and replicated.
   • Extents are read, modified, and written on a sub-extent basis known as a slice, which gives DSF finer granularity than a whole 1 MB operation and lets a slice be trimmed when it moves into cache.
3. Extent Group — 1 MB or 4 MB:
   • An extent group is a physically contiguous stored chunk of 1 MB or 4 MB, held as a file on a storage device owned by a CVM.
   • Extents belonging to different vDisks may be stored together in the same extent group, which is what allows DSF to pack data efficiently on the underlying media.
4. Extent Store:
   • The Extent Store is the persistent storage layer spanning the NVMe, SSD, and HDD devices, where extent groups live.
   • Stargate writes to it, calculates checksums, and registers the resulting metadata in Cassandra.

[!WARNING] Two traps here. First, the sizes: vBlocks and extents are 1 MB, and it is the extent group that is 1 MB or 4 MB. Second, the direction: an extent is logical data, an extent group is the physical chunk it is stored in — not the other way round. A slice is smaller than an extent, not a layer above it.

---

5. Thin Provisioning and Capacity Reclamation

Traditional enterprise storage systems often require administrators to choose between "thick-provisioned eager zeroed" (pre-allocating all blocks on disk) and "thin-provisioned" disks. Pre-allocating blocks wastes immense amounts of expensive storage capacity on unwritten space, while legacy thin provisioning often suffered severe write penalties during on-the-fly allocation.

Native Thin Provisioning by Default

In the Nutanix Distributed Storage Fabric, thin provisioning is native, mandatory, and enforced by default:

• Zero-Allocation on Creation: When an administrator provisions a 2 TB vDisk and assigns it to a virtual machine, DSF creates only the top-level vDisk metadata entry. Zero physical bytes of storage pool capacity are allocated on the NVMe or SSD media.
• Write-on-Demand: Physical capacity is allocated in the Extent Store strictly when the guest operating system writes actual, non-zero data.
• Zero-Block Detection: If an operating system or backup application writes streams of zeroes (e.g., during full disk formatting or image initialization), Stargate intercepts the zero pattern in memory. It discards the write payload and updates metadata pointers to point to a shared global zero-extent, consuming zero additional physical storage.

Dynamic Space Reclamation: SCSI UNMAP and TRIM

When a guest operating system deletes a file, traditional filesystems mark the filesystem allocation table as free, but the underlying storage array has no way of knowing those sectors are dead. Over time, thin-provisioned storage pools steadily bloat to 100% capacity.

Nutanix DSF resolves this through native space reclamation:

• SCSI UNMAP / ATA TRIM Pass-Through: Modern guest operating systems (Windows Server, Linux) issue SCSI UNMAP or TRIM commands upon file deletion.
• Metadata De-allocation: When the hypervisor receives an UNMAP request, Stargate intercepts the command and removes the metadata pointers corresponding to the freed extent slices.
• Curator Space Reclaim: During background maintenance passes, the Curator service identifies extents with zero active metadata pointers, de-allocates the underlying extent group capacity from the Extent Store, and returns the free capacity directly back to the Storage Pool.

[!TIP] Because DSF is inherently thin-provisioned, administrators should monitor Storage Runway in Prism Central. Capacity runway uses predictive machine learning to project when the physical storage pool will be exhausted based on historical write trends, allowing teams to plan node expansions well before thresholds are breached.