3.3 vSAN Cluster Deployment, Storage Pools, & Disk Configurations

3.3 vSAN Cluster Deployment, Storage Pools, & Disk Configurations

Executive Overview: Deploying vSAN in VMware Cloud Foundation 9.0 requires aligning cluster topologies, host counts, physical failure domains, and cryptographic security with enterprise business continuity mandates. Whether configuring a high-density standard cluster in an enterprise datacenter, deploying an agile 2-Node cluster at a retail edge location, or architecting a zero-RPO Stretched Cluster across metropolitan sites, administrators must master the prerequisites of disk claiming, storage pool lifecycle management, Fault Domain engineering, and the vSAN Skyline Health validation engine.

---

vSAN Cluster Deployment Topologies in VCF 9.0

VMware Cloud Foundation 9.0 supports three distinct vSAN cluster topologies to address diverse availability and geographic requirements:

┌────────────────────────────────────────────────────────────────────────┐
│                     vSAN CLUSTER DEPLOYMENT MODELS                     │
├──────────────────┬───────────────────────┬─────────────────────────────┤
│ Standard Cluster │ 2-Node ROBO Cluster   │ Stretched Cluster           │
│ (3 to 64 Hosts)  │ (2 Hosts + Witness)   │ (Site A + Site B + Witness) │
├──────────────────┼───────────────────────┼─────────────────────────────┤
│ Datacenter Core  │ Remote & Branch Edge  │ Metropolitan Dual-Site HA   │
│ Intra-cluster HA │ Low bandwidth witness │ Zero RPO Synchronous Mirror │
└──────────────────┴───────────────────────┴─────────────────────────────┘

1. Standard vSAN Clusters

Standard clusters represent the core deployment model for both the VCF Management Domain and enterprise Workload Domains. Hosts reside within a single datacenter fabric, connected via low-latency 25 GbE or 100 GbE switches. Standard clusters scale from 3 to 64 hosts, delivering shared storage across all cluster members.

2. 2-Node ROBO (Remote Office / Branch Office) Clusters

Engineered specifically for edge environments, retail stores, and remote offices where deploying 3 or more physical servers is cost-prohibitive:

• Consists of exactly two physical ESXi hosts at the local site hosting all compute and storage data components.
• The two nodes can be connected via a cost-effective direct-connect cable configuration (eliminating expensive 10/25 GbE physical switches) or through a shared local switch.
• An external vSAN Witness Host is deployed as a lightweight virtual appliance at a centralized primary datacenter or in VMware Cloud on AWS. The witness maintains quorum metadata and arbitrates split-brain conditions.
• Network Prerequisites: The inter-node link requires 10 GbE. The link between the data nodes and the remote Witness Host requires modest bandwidth (100 to 200 Mbps) and can tolerate high round-trip latency of up to 500 ms RTT, making it ideal for standard WAN and cellular backup connections.

3. vSAN Stretched Clusters

Stretched clusters extend a single vSAN cluster across two geographically separated active sites (Preferred Site and Secondary Site) within metropolitan distances, backed by an independent Witness Host at a third site:

• Delivers zero Recovery Point Objective (RPO) and automated Recovery Time Objective (RTO) failover across datacenters.
• Inter-Site Network Requirements: The link between the Preferred and Secondary sites carries synchronous storage replication. It strictly mandates a maximum round-trip time (RTT) latency of 5 ms or less (typically representing a geographic distance under 50-100 kilometers) and dedicated 10/25 GbE+ bandwidth.
• Witness Site Network Requirements: The Witness Host must reside in an independent third site (outside the failure domains of both data sites). It requires low bandwidth and can tolerate up to 100-200 ms RTT latency to both data sites.
• Dual-Layer Protection (Nested Policies): Storage policies configure two tiers of resilience:
  • Primary Level of Failures to Tolerate (PFTT): Governs cross-site protection via Dual-Site Mirroring (RAID-1 across Site A and Site B).
  • Secondary Level of Failures to Tolerate (SFTT): Governs local protection within each individual site (e.g., RAID-1 mirroring or RAID-5 erasure coding among local hosts within Site A and Site B independently).

---

Host Sizing & Minimum Requirements: 3-Host vs. 4-Host Reality

A critical topic tested on the VCF Administrator exam is the operational distinction between the functional minimum host count and the production enterprise requirement:

The 3-Host Minimum vs. 4-Host Production Mandate

• Functional Minimum: A standard vSAN cluster can technically be initialized with 3 ESXi hosts. Under an FTT=1 (RAID-1) storage policy, vSAN creates three components: Data Replica 1, Data Replica 2, and a Witness component. These three components are placed on Host 1, Host 2, and Host 3 respectively.
• The Maintenance and Failure Dilemma: In a 3-host cluster, zero spare hosts exist. If Host 1 experiences an unexpected hardware failure or is placed into Maintenance Mode with Full Data Evacuation, vSAN cannot rebuild or restore redundancy for the missing components because no 4th host exists to receive them. The cluster enters a degraded state where a single subsequent drive or host failure will cause permanent data unavailability.
• The 4-Host Production Mandate: Broadcom and VMware Cloud Foundation architecture guidelines strongly mandate a minimum of 4 ESXi hosts for all production clusters. In a 4-host cluster:
  • If one host fails, the surviving 3 hosts provide immediate target capacity for vSAN to self-heal and rebuild the missing components, restoring full FTT=1 compliance automatically.
  • Routine ESXi rolling patches can be executed by SDDC Manager without leaving virtual machine storage in an unprotected, degraded state during host reboots.

Slack Space Architecture

To ensure uninterrupted self-healing, snapshot consolidation, and dynamic policy reconfigurations, administrators must reserve 20% to 30% of total cluster capacity as Slack Space. Operating a cluster above 80% capacity triggers automated Skyline Health warnings and restricts vSAN background rebalancing operations.

---

Disk Claiming and Storage Pool Management in ESA

In legacy vSAN OSA, disk provisioning was a complex manual process requiring administrators to select specific SSDs to form disk groups, ensuring that cache-to-capacity ratios remained within strict vendor guidelines.

vSAN ESA replaces this complexity with automated Storage Pool Claiming:

┌────────────────────────────────────────────────────────────────────────┐
│                     vSAN ESA DISK CLAIMING WORKFLOW                    │
├────────────────────────────────────────────────────────────────────────┤
│ 1. DEVICE DISCOVERY:                                                   │
│    ESXi scans PCIe bus and identifies physical NVMe solid-state drives │
├────────────────────────────────────────────────────────────────────────┤
│ 2. COMPATIBILITY AUDIT:                                                │
│    Firmware & performance characteristics checked against ESA VCG      │
├────────────────────────────────────────────────────────────────────────┤
│ 3. STORAGE POOL INGESTION:                                             │
│    Compatible NVMe drives claimed into host Storage Pool               │
│    (All drives participate equally; zero cache tier configuration)     │
├────────────────────────────────────────────────────────────────────────┤
│ 4. DYNAMIC POOL EXPANSION:                                             │
│    New NVMe drives inserted into server bays automatically claimable   │
└────────────────────────────────────────────────────────────────────────┘

Storage Pool Lifecycle Operations

• Automatic vs. Manual Claiming: During cluster creation in SDDC Manager, administrators can set disk claiming to Automatic (all compatible NVMe drives matching the ESA ReadyNode specification are ingested immediately) or Manual (allowing specific NVMe devices to be excluded for local scratch or alternative uses).
• Dynamic Capacity Expansion: When an enterprise requires additional storage capacity, technicians simply hot-plug certified NVMe drives into available server bays. The administrator claims the new devices via vSphere Client or SDDC Manager, and the Storage Pool immediately assimilates the new capacity without downtime.
• Drive Decommissioning: When replacing a drive exhibiting predictive failure warnings, administrators initiate an evacuation workflow choosing between:
  • Ensure Accessibility: Fast evacuation ensuring all VMs remain accessible, but temporarily tolerating reduced redundancy during drive removal.
  • Full Data Migration: Completely mirrors and evacuates all components to surviving drives before the drive is unclaimed, maintaining 100% policy compliance throughout the maintenance window.

---

Fault Domains (FD) Engineering

By default, vSAN treats each individual physical ESXi host as an isolated failure domain. If a host fails, vSAN assumes other hosts remain unaffected. However, in enterprise datacenters, physical hardware shares upstream failure boundaries:

• An entire server rack shares Top-of-Rack (ToR) network switches.
• A blade enclosure shares redundant power supply modules and midplanes.
• Multiple server racks share a common Power Distribution Unit (PDU).

┌────────────────────────────────────────────────────────────────────────┐
│                     FAULT DOMAINS RACK-AWARE DESIGN                    │
├──────────────────┬──────────────────┬──────────────────┬───────────────┤
│  FAULT DOMAIN 1  │  FAULT DOMAIN 2  │  FAULT DOMAIN 3  │ FAULT DOMAIN 4│
│     (Rack 1)     │     (Rack 2)     │     (Rack 3)     │   (Rack 4)    │
├──────────────────┼──────────────────┼──────────────────┼───────────────┤
│ Host 1, Host 2   │ Host 3, Host 4   │ Host 5, Host 6   │ Host 7, Host 8│
├──────────────────┼──────────────────┼──────────────────┼───────────────┤
│ Data Replica 1   │ Data Replica 2   │ Witness Comp.    │ Spare Capacity│
└──────────────────┴──────────────────┴──────────────────┴───────────────┘

The Mechanics of Fault Domains

A Fault Domain (FD) is a user-defined logical grouping of ESXi hosts that corresponds to a shared physical failure boundary. When Fault Domains are configured, vSAN alters its object component placement algorithms: no two replicas or parity stripes belonging to the same object are ever placed within the same Fault Domain.

Critical Sizing Mathematics for Fault Domains

1. The 2n + 1 Rule: To satisfy a storage policy using RAID-1 mirroring with a Primary Level of Failures to Tolerate of FTT=n, the cluster strictly requires a minimum of 2n + 1 Fault Domains:
   • FTT=1: Requires (2 * 1) + 1 = 3 Fault Domains (e.g., 3 separate racks).
   • FTT=2: Requires (2 * 2) + 1 = 5 Fault Domains.
   • FTT=3: Requires (2 * 3) + 1 = 7 Fault Domains.
2. Erasure Coding Rule: To satisfy RAID-5 erasure coding in OSA (3+1), a minimum of 4 Fault Domains is required. To satisfy RAID-6 (4+2), a minimum of 6 Fault Domains is required.
3. Symmetric Sizing Imperative: All Fault Domains must contain identical host counts and storage capacities. If Rack 1 contains four hosts with 40 TB, but Rack 2 and Rack 3 contain only two hosts with 20 TB, vSAN's placement engine will be prematurely constrained by the smallest domain, causing virtual disks to report out-of-space errors while the largest rack sits half-empty.

---

Security Architecture: vSAN Data-at-Rest Encryption (DARE)

vSAN provides native, hypervisor-level Data-at-Rest Encryption (DARE) using FIPS 140-2 validated AES-XTS 256-bit encryption. DARE protects against physical drive theft from datacenters and ensures secure drive decommissioning without manual degaussing.

┌────────────────────────────────────────────────────────────────────────┐
│                     vSAN ENVELOPE ENCRYPTION MODEL                     │
├────────────────────────────────────────────────────────────────────────┤
│ 1. KEY MANAGEMENT LAYER (External KMS or Native Key Provider NKP):     │
│    Generates & distributes Key Encryption Key (KEK)                    │
├────────────────────────────────────────────────────────────────────────┤
│ 2. HYPERVISOR STORAGE LAYER (ESXi Kernel):                             │
│    • ESXi generates unique Data Encryption Key (DEK) per object        │
│    • DEK encrypts raw storage blocks via AES-256                       │
│    • KEK encrypts (wraps) the DEK and embeds it in component metadata  │
└────────────────────────────────────────────────────────────────────────┘

The Two-Tier Envelope Encryption Model

1. Key Encryption Key (KEK): Managed externally. vCenter Server interfaces with an external KMIP 1.1-compliant Key Management Server (KMS) or leverages the built-in vSphere Native Key Provider (NKP). When an ESXi host joins the cluster, it requests the KEK from vCenter.
2. Data Encryption Key (DEK): Generated locally by the ESXi host for every individual vSAN object component using an internal cryptographic pseudo-random number generator. The DEK encrypts the actual data blocks committing to storage. The ESXi host then uses the KEK to wrap (encrypt) the DEK, storing the wrapped DEK in the object's metadata header.
3. Key Rotation: Administrators can rotate KEKs on-demand with zero workload downtime. Rotating the KEK merely requires re-wrapping the small DEK headers; terabytes of underlying data blocks do not need to be decrypted and re-encrypted.

vSphere Native Key Provider (NKP)

Traditional enterprise encryption mandated deploying and maintaining expensive, third-party external KMS clusters. vSphere introduces the Native Key Provider (NKP), built natively into vCenter Server:

• Eliminates third-party KMS dependencies.
• High availability is achieved by backing up the key provider configuration archive (protected by a passphrase) and caching keys in physical host TPM 2.0 (Trusted Platform Module) chips.
• Seamlessly satisfies PCI-DSS, HIPAA, and government compliance requirements for encrypted private cloud storage.

Data-in-Transit Encryption

Complementing DARE, vSAN supports Data-in-Transit Encryption. Utilizing AES-256-GCM, ESXi encrypts all inter-host vSAN kernel replication and witness traffic traversing physical switches, preventing packet snooping across datacenter network fabrics.

---

vSAN Skyline Health Service & Operational Diagnostics

The vSAN Skyline Health Service is an intelligent, automated diagnostics engine integrated directly into vCenter Server and SDDC Manager. It continuously executes dozens of proactive health checks across critical categories:

1. Hardware Compatibility (HCL): Automatically synchronizes with the VMware Compatibility Guide online. Alerts if physical NVMe drives, SAS controllers, or network interface card (NIC) firmware and driver versions drift from the certified Software Bill of Materials (BOM).
2. Physical Disk Health & Wear-Out Endurance: Continuously polls NVMe SMART telemetry. Generates warnings when solid-state drive write endurance reaches 80% wear, escalating to critical alerts at 90% wear, allowing proactive replacement before media failure.
3. Network Health & MTU Consistency: Transmits automated probe packets across all vSAN VMkernel interfaces. Verifies that MTU sizes match end-to-end (detecting MTU 1500 vs. MTU 9000 mismatches), validates VLAN tagging, and tests inter-node network latency.
4. Cluster Capacity & Balance: Audits free disk capacity across all nodes. If data placement becomes skewed (e.g., one node exceeds 80% utilization while others sit at 40%), Skyline Health automatically initiates proactive background rebalancing.

---

Exam Watch: Key Scenarios and Candidate Traps

[!IMPORTANT] The Fault Domain Math Trap: Questions testing Fault Domain sizing routinely offer 3, 4, or 5 as answer choices for an FTT=2 RAID-1 policy. Remember the formula: 2n + 1. For FTT=2, (2 * 2) + 1 = 5 Fault Domains. Configuring 4 Fault Domains will cause the policy to fail compliance.

[!WARNING] Stretched Cluster Latency vs. 2-Node Latency: Do not confuse witness latency requirements! The inter-site data link in a stretched cluster requires <= 5 ms RTT. However, the link to the Witness Host in a 2-node ROBO deployment can tolerate up to 500 ms RTT (and up to 100-200 ms RTT in a stretched cluster).

[!TIP] Envelope Encryption Mechanics: If an exam question asks what happens to virtual machine data during a KEK rotation, the answer is nothing. The data blocks remain untouched. Only the per-object Data Encryption Keys (DEKs) are decrypted with the old KEK and re-encrypted with the new KEK.