2.2 vSphere Cluster Design, High Availability (HA), and DRS

2.2 vSphere Cluster Design, High Availability (HA), and DRS

Exam Focus: Resiliency, capacity planning, and dynamic workload placement in VMware Cloud Foundation (VCF) 9.0 depend directly on the design of vSphere clusters. For the VCP-VCF (2V0-17.25) exam, candidates must master cluster sizing boundaries, the internal operation of vSphere High Availability (HA) including Fault Domain Manager (FDM) agent election and heartbeat paths, admission control calculations, host isolation responses, Distributed Resource Scheduler (DRS) automation levels and migration thresholds, affinity and anti-affinity rules, and Proactive HA hardware degradation workflows.

---

vSphere Cluster Architecture & Sizing Limits in VCF

A vSphere cluster represents an aggregated boundary of compute (CPU), memory, storage, and networking resources managed collectively by vCenter Server. Within VMware Cloud Foundation, clusters serve two distinct functional roles:

• Management Domain Clusters: Dedicated entirely to hosting the VCF control plane and management infrastructure—including the vCenter Server Appliance, SDDC Manager, the NSX Manager cluster (three unified management nodes), and VCF Operations analytics nodes. To satisfy high availability, support vSAN storage policies (FTT=1 with RAID-1 mirroring or RAID-5 erasure coding), and maintain N+1 operational redundancy during rolling updates, VCF architecture mandates a strict minimum of 4 ESXi hosts in the Management Domain cluster.
• Virtual Infrastructure (VI) Workload Domain Clusters: Dedicated to running tenant applications, Kubernetes workloads, and business services. When backed by principal vSAN storage, a VI Workload Domain cluster requires at least 3 hosts (for standard RAID-1) or 4 hosts (for RAID-5 erasure coding and robust lifecycle maintenance). Non-vSAN clusters (e.g., using external FC or NFS storage) can be provisioned with a minimum of 2 or 3 hosts depending on architectural requirements.

Under vSphere 8.0 and 9.0 architectural specifications, a single vSphere cluster supports up to 96 ESXi hosts and up to 10,000 virtual machines (or up to 1,024 VMs per host). When vSAN is enabled, recommended operational limits align cluster sizes between 32 and 64 nodes per cluster to optimize fault domain blast radiuses and resynchronization traffic.

---

vSphere HA Architecture: Fault Domain Manager (FDM) & Heartbeat Mechanics

vSphere High Availability provides rapid, automated virtual machine restart capabilities in the event of underlying physical server hardware failures, operating system crashes, or network isolation events. vSphere HA operates independently of vCenter Server once configured; even if vCenter is completely powered off or undergoing maintenance, HA will successfully detect host failures and restart impacted virtual machines.

The Fault Domain Manager (FDM) Agent

When HA is enabled on a cluster, vCenter pushes and configures the Fault Domain Manager (FDM) agent (vmware-fdm) on every ESXi host in the cluster:

• Master (Primary) Host: During cluster initialization or upon master failure, the FDM agents hold an election. The host that possesses access to the greatest number of shared datastores is elected master; if multiple hosts have access to an identical number of datastores, the host with the highest lexicographical Managed Object ID (MOID, e.g., host-45 over host-12) is selected. Exactly one master host is active per cluster.
  • Master Responsibilities: Tracks the inventory of protected virtual machines and their power states; continuously monitors the health of all secondary hosts; coordinates and executes VM restart workflows upon host failure; maintains cluster health telemetry and reports state changes to vCenter Server.
• Secondary (Slave) Hosts: All remaining hosts in the cluster operate as secondary nodes.
  • Secondary Responsibilities: Execute running virtual machines; dispatch regular network heartbeats to the master host; monitor local VM state changes and report them to the master; initiate a new master election if communication with the active master is severed.

Heartbeat Mechanisms: Network vs. Datastore

vSphere HA implements a dual-path heartbeat architecture to distinguish accurately between an actual physical server crash, a network partition, and a host network isolation condition:

1. Network Heartbeating: FDM agents exchange point-to-point UDP unicast packets across the ESXi Management VMkernel network on port 8182 once every second (1,000 ms). If a secondary host fails to transmit heartbeats for 14 consecutive seconds, the master marks the network link dead and begins secondary verification.
2. Datastore Heartbeating: To prevent catastrophic split-brain scenarios (where two hosts attempt to run the same virtual machine simultaneously), vSphere HA inspects shared datastores. A minimum of two shared datastores must be accessible to hosts in the cluster. FDM agents write periodic heartbeat timestamps to hidden files (.vSphere-HA/<cluster-id>/host-<id>-hb).
   • If network heartbeats stop but datastore heartbeats continue, the master knows the secondary host is alive but disconnected from the management network (network partition or isolation).
   • If both network heartbeats and datastore heartbeats stop, the master concludes the host has suffered a fatal hardware crash or power loss, and immediately initiates failover restarts on surviving nodes.

---

Admission Control Policies & Failover Capacity

vSphere HA Admission Control reserves a specific portion of cluster compute capacity to guarantee that sufficient physical CPU and memory resources are available to restart virtual machines if one or more ESXi hosts fail.

[!TIP] Admission Control Best Practice: VMware Cloud Foundation best practice strongly mandates the Cluster Resource Percentage policy. When configured, vCenter continuously evaluates total cluster reservations against available capacity. If powering on a new VM would cause total consumption to exceed the allowed threshold, admission control blocks the power-on operation to preserve HA restart guarantees without stranding resources.

---

Host Isolation Response & VM Restart Prioritization

A host enters an Isolated State when its FDM agent loses network heartbeats from the master host, fails to ping its configured network isolation addresses (by default, the default gateway of the management network, or custom addresses specified via das.isolationaddress), but retains active access to shared datastores.

Host Isolation Response Options

When an ESXi host detects it is isolated, it executes its configured Host Isolation Response:

• Disabled: The host leaves all virtual machines running. This is non-disruptive if the VMs are communicating across non-management networks (e.g., separate vMotion/VM data networks), but risks prolonged management disconnection.
• Shut Down and Restart VMs: The host attempts a graceful guest operating system shutdown via VMware Tools for a configurable timeout period (default 300 seconds, controlled by das.isolationshutdowntimeout). If the OS fails to shut down within the window, the host forcefully powers it off. The HA master host then restarts the VMs on healthy nodes.
• Power Off and Restart VMs: The host immediately issues an instantaneous hard power-off to all running virtual machines. The HA master promptly restarts them on surviving cluster nodes.

[!IMPORTANT] vSAN Isolation Response Requirement: For vSAN-backed clusters and VCF environments, Power Off and Restart is the strict recommendation. Graceful shutdown delays VM recovery and can leave storage locks active on vSAN objects, delaying failover. Immediate power-off breaks disk component locks rapidly and prevents split-brain state inconsistencies.

VM Restart Priority & Orchestrated Restarts

vSphere HA assigns priority tiers to virtual machines to dictate recovery sequence during a failover event:

• Priority Levels: Lowest, Low, Medium (Default), High, Highest, and Disabled (do not restart).
• Orchestrated Restart Dependencies: Administrators can establish explicit dependency chains between multi-tier application layers. For example, Database VMs (Tier 1) can be configured with a dependency condition requiring guest heartbeat readiness (via VMware Tools) or a delay timer before Application VMs (Tier 2) are permitted to power on, ensuring complex multi-tier services recover in proper order without manual intervention.

---

Distributed Resource Scheduler (DRS): Mechanics & Automation

vSphere Distributed Resource Scheduler (DRS) dynamically balances CPU and memory utilization across cluster hosts using live vMotion migrations, and automates initial workload placement.

• Manual: Generates initial placement and migration recommendations; requires manual administrator approval.
• Partially Automated: Automatically places VMs upon initial power-on; generates migration recommendations for runtime load balancing.
• Fully Automated: Automatically places VMs at power-on and executes live vMotion migrations to balance cluster load.

The Migration Threshold Slider

In Fully Automated mode, DRS evaluates cluster resource distribution every 300 seconds (5 minutes) and calculates an imbalance metric based on the standard deviation of host utilization. The Migration Threshold slider controls how aggressively DRS acts on these calculations:

• Level 1 (Conservative): Applies only Priority 1 recommendations—mandatory actions required to satisfy host maintenance mode, power management, or hard affinity rule constraints. It does not perform balancing migrations.
• Level 2: Applies Priority 1 and Priority 2 recommendations (extremely significant performance gains).
• Level 3 (Default): Applies Priority 1, 2, and 3 recommendations. Provides an optimal compromise between resource balance and minimizing vMotion network overhead.
• Level 4: Applies Priority 1 through 4 recommendations (moderate performance improvements).
• Level 5 (Aggressive): Applies all recommendations (Priority 1 through 5), executing migrations even for marginal resource balancing gains.

VM-Host and VM-VM Affinity Rules

Affinity rules govern where virtual machines run within a cluster:

• VM-VM Affinity: Keeps specified virtual machines on the same physical host (e.g., tightly coupled microservices communicating via vSwitch to achieve sub-microsecond latency).
• VM-VM Anti-Affinity: Ensures specified virtual machines remain on separate physical hosts (e.g., redundant Active Directory domain controllers, web farm nodes, or database cluster instances) to prevent a single physical host crash from taking down an entire redundant service.
• VM-Host "Must Run On" (Hard Rule): Strictly confines a VM group to a specific host group. DRS cannot violate this rule for load balancing, and vSphere HA will not restart the VM if all hosts in the designated group are down. Used primarily for strict software licensing or physical compliance.
• VM-Host "Should Run On" (Soft Rule): Directs DRS to place VMs on a specified host group whenever possible, but permits DRS and HA to violate the rule during host maintenance or severe outages to ensure the virtual machine remains powered on.

---

Proactive HA, EVC, & Maintenance Mode Workflows

• Proactive HA: Bridges the gap between hardware health diagnostics and hypervisor scheduling. By integrating with server vendor software providers (e.g., Dell OpenManage Integration, HPE OneView), Proactive HA detects impending hardware component degradation—such as predictive fan failures, power supply degradation, or memory uncorrectable ECC error spikes. When triggered, Proactive HA places the degraded host into Quarantine Mode (DRS will not place new VMs on the host, but leaves existing ones unless contention occurs) or Maintenance Mode (DRS automatically evacuates all running VMs to healthy hosts), preventing unexpected host crashes.
• Enhanced vMotion Compatibility (EVC): Enforces a common CPU baseline instruction set across physical hosts within a cluster that feature disparate processor generations (e.g., Intel Ice Lake and Sapphire Rapids). EVC masks out newer CPU feature flags so that virtual machines see a uniform processor architecture, ensuring seamless, non-disruptive vMotion migrations across all hosts. vSphere supports both Cluster-level EVC and Per-VM EVC (which allows a VM to carry its CPU baseline independently across different clusters and vCenter instances).
• Cluster Maintenance Mode Workflow: When placing a host into Maintenance Mode for firmware or software upgrades, DRS automatically initiates vMotion migrations of all active VMs to evacuate the host. For clusters running vSAN storage, the administrator must select an appropriate data evacuation mode:
  • Ensure Accessibility: Default mode; evacuates only data required to maintain VM read/write availability during temporary host reboots.
  • Full Data Migration: Evacuates all storage components to other cluster hosts; required when permanently decommissioning a host.
  • No Data Migration: Does not evacuate storage components; VMs may lose redundancy during the maintenance window.