3.2 vSphere High Availability (HA) & Admission Control

3.2 vSphere High Availability (HA) & Admission Control

High availability is a cornerstone requirement of enterprise infrastructure. vSphere High Availability (HA) protects workloads against physical server outages, operating system crashes, hypervisor isolation, and storage fabric failures. Unlike disaster recovery solutions that operate over hours or days, vSphere HA provides rapid, automated virtual machine restarts within seconds to minutes, operating entirely independently of vCenter Server availability once configured.

---

vSphere HA Agent Architecture & Master Election

vSphere HA is built upon the Fault Domain Manager (FDM) agent. When HA is enabled on a cluster, vCenter pushes and installs the FDM agent (vmware-fdm) into the user world of every ESXi host in the cluster. FDM communicates directly with the ESXi host daemon (hostd) and the VMkernel, executing cluster state machines without requiring an active connection to vCenter's vpxd daemon.

Master Host vs. Slave Hosts

Within an active vSphere HA cluster, hosts operate in one of two roles:

• Master Host: Exactly one host functions as the cluster master (elected dynamically). The master host:
  • Monitors the power state and execution status of all protected virtual machines in the cluster.
  • Monitors the health of all slave hosts via management network heartbeats and datastore heartbeats.
  • Coordinates virtual machine failover restarts when a slave host crashes or isolates.
  • Acts as the primary communication bridge to vCenter Server, reporting cluster status changes to vpxd.
• Slave Hosts: All remaining hosts in the cluster operate as slaves. Slave hosts:
  • Run virtual machines and report local VM power state transitions to the master.
  • Transmit continuous heartbeats to the master over the management VMkernel network.
  • Monitor the master host's heartbeats and initiate a new master election if the master fails.
  • Participate in datastore heartbeating channels.

Primary-Host Election

HA elects a primary host when protection is initialized or the current primary becomes unavailable. Network partitions may have separate primaries. vCenter is needed to configure HA, but installed HA agents can monitor and restart workloads without continuous vCenter availability. For administration, verify agent health and network reachability rather than trying to force an election by changing a host ID.

Multi-Channel Heartbeating & Host Failure Classifications

To prevent erroneous VM restarts and avoid catastrophic split-brain scenarios (where two physical hosts simultaneously write to the same virtual disk), vSphere HA employs dual-channel heartbeating combining management network packets with storage lock checks.

1. Management Network Heartbeating

Hosts exchange point-to-point network heartbeats across the management VMkernel network using UDP port 8182. The master transmits heartbeats to every slave, and slaves transmit heartbeats back to the master at 1-second intervals:

• If the master ceases receiving heartbeats from a slave for 15 consecutive seconds, the master initiates failure domain analysis.
• The master first checks whether the slave is communicating via datastore heartbeats before declaring the host dead.

2. Datastore Heartbeating

When management network heartbeats drop, datastore heartbeating acts as the secondary out-of-band arbitration path:

• vSphere HA automatically designates shared datastores (minimum 2 datastores recommended) for heartbeating.
• Each host creates a hidden directory named .vSphere-HA on the selected datastores, maintaining an active heartbeat lock file (host-XXX-hb).
• Every host updates its heartbeat lock file once every second to prove that the VMkernel is alive and executing I/O, even if its management network is down.

Three Distinct Host Failure Conditions

By cross-referencing management network heartbeats and datastore heartbeats, the master host classifies the failure into one of three specific states:

Host Isolation Responses & Restart Policies

When an ESXi host detects that it can no longer communicate with the master host, it attempts to ping its designated isolation address (by default, the default gateway of the management network; additional addresses can be defined via the advanced option das.isolationaddress). If pings fail, the host declares itself isolated.

Isolation Response Options

The isolated host executes its configured Host Isolation Response:

1. Disabled (Default in some versions): The isolated host leaves all running virtual machines powered on. If the isolated host still has healthy storage connections, workloads continue running uninterrupted. However, if the master attempts to restart these VMs on another host, restart will fail because the isolated host retains the virtual disk file locks.
2. Shut down and restart VMs: The isolated host sends an orderly ACPI shutdown signal to guest operating systems via VMware Tools. If the guest operating system cleanly shuts down within the failover timeout window (default 300 seconds), it releases disk locks, and the master restarts the VM on a surviving host. If the shutdown timer expires before the guest exits, the VM is hard powered off.
3. Power off and restart VMs: The isolated host immediately issues a hard power-off command to all running VMs. This instantly breaks the hypervisor disk locks (.vmdk locks). The master host detects isolation via datastore heartbeats, acquires the released disk locks, and rapidly restarts the virtual machines on surviving cluster hosts. This response releases locks so eligible surviving hosts can attempt restart; detection, storage access, placement and application recovery still determine the interruption.

Virtual Machine Restart Priority & Dependency Chaining

When restarting workloads following a failure, vSphere HA respects configurable priority tiers:

• Restart Priority Levels: Workloads can be assigned to Lowest, Low, Medium (Default), High, or Highest priority tiers. High-priority workloads (such as domain controllers or core database engines) are assigned compute slots and powered on before lower-tier application or web servers.
• VM Restart Delay: Administrators can configure specific delay intervals between tiers (e.g., wait 120 seconds after powering on High-priority database servers before initiating Medium-priority application servers).
• VM-to-VM Dependency Rules: vSphere HA allows administrators to define explicit orchestration chains (e.g., VM-App requires VM-DB to complete guest OS initialization before VM-App powers on).

Cluster Admission Control Policies

Admission Control is the gatekeeping mechanism of vSphere HA. It calculates available compute and memory capacity across the cluster and prevents administrators from powering on virtual machines or expanding reservations if doing so would compromise the cluster's ability to tolerate host failures.

1. Cluster Resource Percentage (Recommended)

The modern, flexible default policy. The administrator specifies the number of host failures the cluster must tolerate (e.g., 1 host failure in a 4-host cluster), and vSphere automatically calculates the required percentage reserve:

• Dynamic Percentage Calculation: In a cluster of 4 identical hosts configured to tolerate 1 host failure, the reserve is automatically set to 25% CPU and 25% Memory.
• If a host is added or removed, vSphere dynamically recalculates the percentage to maintain the desired host failure tolerance without manual administrative intervention.

2. Slot Policy

The legacy admission control model based on "slots":

• Slot Definition: A slot is defined as the largest CPU reservation of any powered-on VM in the cluster combined with the largest memory reservation (plus memory overhead) of any powered-on VM.
• Total cluster capacity is divided by the slot size to determine total available slots. When powered-on VMs consume all available slots, no additional VMs can be powered on.
• The Slot Monster Problem: If a single virtual machine in the cluster is configured with an unusually large memory reservation (e.g., 64 GB reserved while all other VMs reserve 2 GB), the slot size for the entire cluster expands to 64 GB. This severely distorts cluster capacity, causing admission control to block VM power-ons despite massive amounts of unreserved physical RAM.

3. Dedicated Failover Hosts

The administrator designates specific physical ESXi hosts exclusively as standby failover targets:

• Dedicated hosts run no virtual machines during normal operations.
• When a production host fails, the master restarts its VMs on the dedicated failover host.
• Architectural Drawback: Extremely inefficient; expensive compute and memory resources sit completely idle during normal operations.

Performance Degradation Tolerance

Under the Cluster Resource Percentage policy, administrators can configure the Performance Degradation VMs Tolerate setting (0% to 100%):

• A value of 0% requests no tolerated performance degradation in the capacity assessment; it does not guarantee application performance after every failure.
• A value higher than 0% allows the cluster to run slightly overcommitted after a failure, accepting performance degradation in exchange for maximizing normal operational consolidation ratios.

VM Component Protection (VMCP): APD vs. PDL

Traditional vSphere HA protected only against compute host failures. VM Component Protection (VMCP) extends HA to protect against storage connectivity failures between ESXi hypervisors and datastores. Storage path failures fall into two distinct engineering classifications: All Paths Down (APD) and Permanent Device Loss (PDL).

VMCP Recovery Execution Mechanics

PDL is an explicit indication that a device is permanently unavailable to that host. Under an enabled power-off/restart response, HA can attempt recovery on a host that still has access to the VM's storage. It cannot recreate a deleted datastore. If the LUN is lost everywhere, storage recovery or restoration from backup is required.

APD lacks a definitive permanent-loss response. For traditional storage, the default APD timeout is 140 seconds, followed by the configured VMCP failover delay (180 seconds by default), before response decisions. Conservative response requires confidence that another host can restart the VM; aggressive response can also act when that conclusion is unknown. Verify the current settings and datastore type rather than treating 140 seconds as the complete restart timer. See Broadcom KB 313940.

A healthy destination, accessible VM files, released locks and sufficient compatible resources remain necessary for restart. An HA configuration does not replace independent backups, array recovery procedures or an application-specific recovery plan.

Realistic Cluster Failure Scenarios & Troubleshooting

Scenario 1: Total Management Isolation with Active SAN Fabric

Problem: A top-of-rack management network switch experiences a power supply blowout, severing management network uplinks to host esxi-04. All guest VMs on esxi-04 continue servicing customer transactions over their independent virtual switch tenant uplinks. After 15 seconds, vSphere Client alarms trigger Host esxi-04 isolated.

Root Cause Analysis:

• esxi-04 stops transmitting UDP 8182 heartbeats to the master host and cannot ping its management default gateway.
• However, esxi-04 continues writing to its datastore heartbeat lock files on shared Fibre Channel storage.
• Because the cluster isolation response was left at Disabled, esxi-04 leaves its VMs running.
• The master host recognizes via datastore heartbeats that esxi-04 is alive. Because esxi-04 still holds active .vmdk disk locks, the master does not attempt to restart the VMs on other hosts, preventing split-brain corruption.

Resolution Procedure:

1. Restore redundant management switch power.
2. If immediate recovery of esxi-04 is required, configure the isolation response to Power off and restart VMs so that disk locks are released immediately upon isolation, allowing surviving hosts to take over workloads.

Scenario 2: Permanent Storage Loss

If a LUN is accidentally unmapped from one host but remains available on other hosts, a configured VMCP response may enable recovery elsewhere. If the LUN itself is deleted and no surviving host can access the VM files, HA cannot recover the data by restarting a VM. Restore storage access if possible, otherwise use a tested backup/recovery procedure. This distinction separates recovery of access from recovery of destroyed data.

Review Takeaways

• Configured host HA agents can protect workloads while vCenter is unavailable.
• Isolation, network partition and host failure require different interpretations.
• Admission control preserves planned failover capacity but cannot guarantee application recovery time.
• VMCP can restart on a compatible host with usable VM storage; it cannot recreate a deleted LUN.