3.2 vSAN Architecture: Express Storage Architecture (ESA) vs. Original Storage Architecture (OSA)

3.2 vSAN Architecture: Express Storage Architecture (ESA) vs. Original Storage Architecture (OSA)

Executive Overview: VMware vSAN represents the software-defined, hyperconverged storage foundation of VMware Cloud Foundation. Historically, vSAN relied on the Original Storage Architecture (OSA), a two-tier storage model architected during the era of spinning magnetic hard drives and SAS/SATA solid-state disks. In VMware Cloud Foundation 9.0 (powered by vSphere 9.0), the Express Storage Architecture (ESA) serves as the premier storage platform. Re-architected from the ground up for modern Non-Volatile Memory Express (NVMe) solid-state drives, high-core CPUs, and 25/100 GbE networking, ESA delivers near-bare-metal performance, hardware-efficient erasure coding, and simplified operations.

---

The Architectural Evolution: From OSA to ESA

To appreciate the design decisions of vSAN ESA, administrators must understand the hardware landscape that dictated OSA's original design:

• When vSAN was introduced, fast enterprise flash memory was exceptionally expensive and constrained in capacity. Commodity storage relied heavily on slow, high-capacity magnetic spinning disks (HDDs).
• To balance cost and performance, OSA adopted a two-tier caching model: a small, ultra-fast cache drive absorbed writes and buffered reads, shielding the slower, high-capacity backend drives.
• Over the subsequent decade, hardware underwent a generational paradigm shift: high-capacity TLC and QLC NVMe solid-state drives achieved price parity with legacy media while delivering millions of IOPS over PCIe Gen 4 and Gen 5 buses. High-speed 25 GbE and 100 GbE networking became standard. High-core-count processors introduced vectorized SIMD instructions (AVX-512).
• In this modern hardware environment, OSA's two-tier design became an architectural bottleneck: CPU cycles were wasted managing cache-to-capacity destaging queues, and write performance was artificially throttled by the endurance and capacity of a single cache SSD.

---

vSAN OSA: The Two-Tier Disk Group Architecture

vSAN Original Storage Architecture organizes physical drives within each ESXi host into discrete Disk Groups:

┌────────────────────────────────────────────────────────────────────────┐
│                     vSAN OSA ESXi HOST ARCHITECTURE                    │
├──────────────────────────────────┬─────────────────────────────────────┤
│           DISK GROUP 1           │            DISK GROUP 2             │
│ ┌──────────────────────────────┐ │  ┌──────────────────────────────┐   │
│ │ 1 Dedicated Cache SSD        │ │  │ 1 Dedicated Cache SSD        │   │
│ │ (Write Buffer max 1.6 TB)    │ │  │ (Write Buffer max 1.6 TB)    │   │
│ ├──────────────────────────────┤ │  ├──────────────────────────────┤   │
│ │ Capacity Tier (1-7 Drives)   │ │  │ Capacity Tier (1-7 Drives)   │   │
│ │ Flash SSDs or Magnetic HDDs  │ │  │ Flash SSDs or Magnetic HDDs  │   │
│ └──────────────────────────────┘ │  └──────────────────────────────┘   │
└──────────────────────────────────┴─────────────────────────────────────┘

Rules of OSA Disk Groups

• Each ESXi host in an OSA cluster can host between 1 and 5 Disk Groups.
• Each disk group MUST contain exactly 1 Cache drive and between 1 and 7 Capacity drives.
• Cache Tier Roles:
  • In All-Flash OSA, 100% of the cache device is dedicated to write buffering (up to an architectural maximum buffer cap of 600 GB or 1.6 TB depending on release); read requests are serviced directly from the capacity flash tier.
  • In Hybrid OSA, 70% of the cache device is dedicated to read caching, and 30% is allocated to write buffering.

The Inherent Flaws of the OSA Model

1. Single Point of Failure (SPOF) per Group: The single cache SSD represents a catastrophic single point of failure for the entire disk group. If the cache drive fails, the entire disk group—including all 7 capacity drives and every component stored upon them—instantly goes offline.
2. The Destaging Bottleneck: Incoming writes hit the cache drive's write buffer. The Log-Structured Object Manager (LSOM) must asynchronously "destage" this buffered data to the slower capacity tier. Under prolonged, write-heavy enterprise workloads, the write buffer saturates faster than LSOM can destage to capacity disks. When saturation occurs, vSAN aggressively throttles incoming guest VM writes to match the slow backend media speed, producing severe I/O latency spikes.
3. Stranded Capacity: Cache devices contribute zero bytes of usable storage capacity to the cluster. In a host with four 1.6 TB NVMe cache drives, 6.4 TB of premium flash storage is consumed purely as scratchpad buffer.

---

vSAN ESA: The Single-Tier Storage Pool Architecture

vSAN Express Storage Architecture completely abolishes the concept of disk groups and dedicated cache devices. Instead, ESA introduces a unified, single-tier Storage Pool:

┌────────────────────────────────────────────────────────────────────────┐
│                     vSAN ESA ESXi HOST ARCHITECTURE                    │
├────────────────────────────────────────────────────────────────────────┤
│                       UNIFIED ESA STORAGE POOL                         │
│  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐   │
│  │ NVMe Drive 1 │ │ NVMe Drive 2 │ │ NVMe Drive 3 │ │ NVMe Drive N │   │
│  │ (Log + Data) │ │ (Log + Data) │ │ (Log + Data) │ │ (Log + Data) │   │
│  └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘   │
│  • Single-Tier Architecture: All drives share write logging & data     │
│  • 100% Usable Capacity: Zero stranded cache storage                   │
│  • Isolated Failure Domain: 1 failed NVMe impacts only local component │
└────────────────────────────────────────────────────────────────────────┘

Key Principles of the ESA Storage Pool

• Uniform Drive Participation: All certified NVMe TLC drives claimed by the host belong to a single, flat Storage Pool. Every drive participates equally in absorbing write traffic, storing log metadata, and holding persistent data.
• 100% Usable Capacity: Because there are no dedicated cache drives, 100% of installed NVMe storage capacity is contributed directly to cluster capacity.
• Minimized Failure Blast Radius: If an NVMe drive fails in an ESA Storage Pool, only the specific data components residing on that individual drive are marked degraded. The surviving NVMe drives in the host continue operating at full throughput without interruption. The blast radius of a drive failure is reduced by up to 85% compared to an OSA disk group failure.

---

Log-Structured File System (LFS) vs. LSOM: The Write Path Revolution

The most transformative innovation within vSAN ESA lies in its software write path. ESA introduces a purpose-built Log-Structured File System (LFS) that replaces the legacy Log-Structured Object Manager (LSOM).

The OSA Erasure Coding Dilemma: The Read-Modify-Write Penalty

In vSAN OSA, applying RAID-5 or RAID-6 erasure coding delivered dramatic space savings compared to RAID-1 mirroring, but imposed a crippling write penalty:

• Whenever a virtual machine issued a write to a RAID-5/6 object in OSA, vSAN had to execute an expensive 4-way read-modify-write cycle:
  1. Read the existing data block from the capacity drive.
  2. Read the existing parity block from the parity drive across the network.
  3. Calculate the new parity delta in software.
  4. Write the new data block and new parity block back across the network to disk.
• This cycle generated severe write amplification, consumed heavy CPU overhead, saturated inter-host networks, and multiplied I/O latency. Consequently, architects routinely avoided RAID-5/6 for write-intensive databases in OSA, forcing the use of expensive RAID-1 mirroring.

The ESA LFS Solution: Parity Without Penalty

ESA eliminates the read-modify-write penalty entirely through its Log-Structured File System:

1. In-Memory Write Coalescing: Incoming writes from virtual machines are received directly into host memory and appended to a distributed, fast write log spread across the NVMe drives in the Storage Pool.
2. Full-Stripe Writes: As writes accumulate in memory, LFS coalesces them into full stripe widths (e.g., 4 data blocks + 1 parity block). Parity is computed in a single pass using vectorized CPU SIMD instructions (AVX-512).
3. Direct Single-Pass Commit: LFS writes the entire full stripe—both data and parity—directly to persistent NVMe storage in a single sequential operation. No pre-reading of old data or parity is ever required.

OSA Write Path (RAID-5): Incoming Write ──> Read Old Data + Read Old Parity ──> Compute Delta ──> Write New Data + Write New Parity (4-Way Overhead!)

ESA Write Path (RAID-5): Incoming Writes ──> Coalesce in Memory ──> Compute Parity (AVX-512) ──> Full-Stripe Write to NVMe (Zero Penalty!)

Because ESA writes full stripes directly, RAID-5 and RAID-6 erasure coding in ESA achieve write performance equivalent to RAID-1 mirroring, while retaining 33-50% raw storage capacity savings!

---

Ingestion-Layer Data Services (Top of Stack)

In vSAN OSA, data services such as compression, deduplication, and encryption operated at the bottom of the storage stack inside LSOM, executing during the destaging phase within each local disk group.

vSAN ESA completely relocates data services to the very top of the storage stack at the ingestion layer, immediately below the virtual machine interface:

┌────────────────────────────────────────────────────────────────────────┐
│                     vSAN ESA DATA SERVICES PIPELINE                    │
├────────────────────────────────────────────────────────────────────────┤
│ 1. INGESTION LAYER (Top of Stack):                                     │
│    Incoming VM I/O ──> In-line Compression ──> Encryption              │
├────────────────────────────────────────────────────────────────────────┤
│ 2. NETWORK REPLICATION LAYER:                                          │
│    Compressed & Encrypted Blocks Replicated Across 25/100 GbE Network  │
│    (Up to 4x Network Bandwidth Reduction!)                             │
├────────────────────────────────────────────────────────────────────────┤
│ 3. PERSISTENT STORAGE LAYER:                                           │
│    LFS Full-Stripe Commits directly to NVMe Storage Pool               │
└────────────────────────────────────────────────────────────────────────┘

Architectural Advantages of Top-of-Stack Placement

• Up to 4x Inter-Host Network Bandwidth Reduction: In ESA, data is compressed and encrypted before it is replicated across the network to other ESXi hosts in the cluster. Rather than transmitting 100 GB of uncompressed data across the network and compressing it on each remote host, ESA compresses the data once on the local host and transmits only 25-50 GB of compressed payload across the network fabric. This drastically reduces inter-switch network congestion.
• Elimination of Deduplication: Deduplication is completely omitted in vSAN ESA. Real-world enterprise analysis revealed that deduplication on ultra-fast NVMe storage introduced severe global lock contention, inflated metadata tables, and created massive failure domains. ESA replaces deduplication with a modern, high-ratio compression algorithm (achieving up to 8:1 compression ratios) that runs at line rate with negligible CPU overhead.

---

The Native B-Tree Snapshot Engine

Snapshots are vital for backup software, development branches, and patch testing. However, snapshots in vSAN OSA represented an operational hazard:

• OSA Redo-Log Mechanics: OSA utilized redo-log snapshots (vsan-sparse or seSparse). Each snapshot created a new child delta disk. Over time, chains of delta disks formed. Servicing a single read request required ESXi to traverse up through the parent chain, producing severe read amplification.
• The SCSI Stun Problem: When deleting or consolidating snapshots in OSA, ESXi had to merge gigabytes of delta blocks back into the base disk while freezing guest VM execution. This caused prolonged "SCSI stun" events (pausing VM execution for seconds or minutes), routinely crashing database workloads, aborting transactional logs, and violating SLA targets.

ESA Native Snapshot Mechanics

ESA replaces redo-log chains with a completely redesigned, native B-tree snapshot engine integrated directly into the Log-Structured File System:

• When a snapshot is taken in ESA, no delta disk is created. Instead, the snapshot is merely a lightweight, point-in-time branch metadata pointer within the B-tree directory.
• Virtual machine read and write performance remains identical regardless of whether 0 or 30 snapshots exist; performance degradation is less than 1-2%.
• Snapshot deletion executes as background metadata reclamation. The VM is never stunned, zero child disks are merged, and consolidation completes almost instantaneously without risk to transactional databases.

---

Hardware Prerequisites & Deployment Requirements for ESA

Because vSAN ESA is engineered for extreme throughput, Broadcom enforces strict hardware baselines:

1. Certified vSAN ESA ReadyNodes: Deployments must standardize on officially validated vSAN ESA ReadyNodes (profiles ranging from ESA-AF-0 for small environments to ESA-AF-8 for mission-critical tiers). Assembling custom, non-certified whitebox configurations is strictly unsupported.
2. NVMe TLC Solid-State Drives: All drives claimed in the Storage Pool must be certified NVMe TLC solid-state drives adhering to PCIe Gen 4 or Gen 5 bus standards with enterprise endurance ratings. SAS SSDs, SATA SSDs, and magnetic HDDs are strictly prohibited in ESA.
3. Mandatory 25 GbE+ Networking: Production vSAN ESA clusters require a minimum of 25 GbE networking per host. 10 GbE networking is strictly unsupported for production ESA clusters due to the extreme throughput generated by NVMe full-stripe writes. High-density profiles mandate 100 GbE.
4. Host Compute Sizing: ESXi hosts require high-core CPUs supporting modern SIMD vector extensions (AVX-512) to accelerate compression and parity calculations, alongside a recommended baseline of at least 512 GB of RAM per host.

---

Comprehensive Comparison: vSAN OSA vs. vSAN ESA

---

Exam Watch: Key Scenarios and Candidate Traps

[!IMPORTANT] The Erasure Coding Performance Myth: In vSAN OSA, best practice dictated avoiding RAID-5/6 for write-intensive workloads due to the 4-way read-modify-write penalty. On the VCF 9.0 exam, remember that in vSAN ESA, RAID-5/6 has NO write penalty. ESA coalesces writes into full stripes in memory and writes them in a single pass, matching RAID-1 write performance while saving 33-50% storage capacity.

[!WARNING] The 10 GbE Trap for ESA: If an exam question asks whether an organization can upgrade an existing 10 GbE cluster to vSAN ESA by simply swapping in NVMe drives, the correct answer is NO. vSAN ESA strictly requires minimum 25 GbE network infrastructure. Attempting to deploy ESA on 10 GbE will fail pre-checks in SDDC Manager.

[!TIP] Deduplication in ESA: If a question asks how to enable deduplication in a vSAN ESA Storage Pool, the correct answer is that deduplication does not exist in ESA. ESA deliberately eliminated deduplication to avoid lock contention on NVMe devices, relying exclusively on high-performance compression.