6.4 AHV High Availability (HA), Acropolis Dynamic Scheduling (ADS), and Affinity Rules

AHV High Availability (HA), Acropolis Dynamic Scheduling (ADS), and Affinity Rules

Quick Summary: Enterprise virtualization requires robust mechanisms to withstand hardware failures, balance dynamic compute demands, and enforce workload placement governance. Nutanix AHV delivers these capabilities natively within the Acropolis Operating System. AHV High Availability (HA) protects workloads against physical server outages through Best Effort and Guarantee reservation modes. Acropolis Dynamic Scheduling (ADS) acts as an intelligent, autonomous cluster scheduler that dynamically mitigates CPU and memory hotspots via non-disruptive Live Migration. Finally, Affinity Rules allow administrators to bind workloads to specific physical hosts for software licensing optimization (VM-to-Host) or separate redundant application instances across distinct nodes (VM-to-VM Anti-Affinity).

---

1. AHV High Availability (HA) Architecture and Failure Detection

High Availability in traditional virtualization relies on an external management server (such as VMware vCenter) to monitor ESXi hosts and issue restart commands. If the management server crashes, HA state can become degraded or unmanageable. In Nutanix AHV, High Availability is a fully distributed, native service embedded into the Acropolis architecture running on every Controller VM.

+-------------------------------------------------------------------------+
|                         Acropolis HA Architecture                       |
+------------------------------------+------------------------------------+
|             Elected Acropolis Leader CVM (Orchestrator)                 |
|   - Monitors ZooKeeper Quorum for Node Heartbeats                       |
|   - Tracks Host Memory Pools & Failover Segment Reservations            |
|   - Issues libvirt Restart Commands upon Node Failure                   |
+------------------------------------+------------------------------------+
                                     |
         +---------------------------+---------------------------+
         | (Continuous Heartbeats)                               | (Heartbeat Missed >60s)
         v                                                       v
+-----------------------------------+   +-----------------------------------+
| Node 01 (Surviving AHV Host)      |   | Node 02 (FAILED AHV Host)         |
| - Hypervisor: Healthy             |   | - Physical Server Crashed         |
| - CVM: Healthy                    |   | - Heartbeat Lost                  |
| - Action: Hosts Restarted VMs     |   | - Action: Declared Dead           |
+-----------------------------------+   +-----------------------------------+

Heartbeating and Failure Mechanics

1. Distributed Health Monitoring: Every AHV host runs a local Acropolis agent that maintains continuous communication with the local CVM and the cluster's elected Acropolis Leader CVM.
2. Failure Thresholds: The Acropolis Leader monitors node liveness using ZooKeeper consensus and network heartbeats. If a physical AHV host stops heartbeating for more than 60 seconds (or if IPMI out-of-band management signals a catastrophic power supply or motherboard failure), the Acropolis Leader declares the host dead.
3. Shared Storage Advantage: Because all VM virtual disks reside on the Distributed Storage Fabric (DSF), storage is universally accessible across the cluster. The Acropolis Leader does not need to unmount, re-zone, or re-attach storage volumes; it simply directs surviving healthy AHV hosts to spawn new QEMU/KVM execution threads and attach to the existing vDisk metadata paths.

---

2. AHV HA Operating Modes: Best Effort vs. Guarantee Mode

AHV provides two distinct High Availability operational configurations, configured easily in Prism Element under Manage VM High Availability:

+-------------------------------------------------------------------------+
|                         AHV HA Operating Modes                          |
+-----------------------+-------------------------------------------------+
| Best Effort Mode      | - Default configuration                         |
|                       | - 0% cluster memory reserved for failover       |
|                       | - Restarts VMs only if free capacity exists     |
|                       | - Risk of stranded unpowered VMs during failure |
+-----------------------+-------------------------------------------------+
| Guarantee Mode        | - Enterprise production standard                |
|                       | - Automatically reserves N+1 or N+2 capacity    |
|                       | - Uses dynamic "Failover Segments" calculation  |
|                       | - Admission Control strictly protects failover  |
+-----------------------+-------------------------------------------------+

1. Best Effort Mode (Default)

• Operational Model: In Best Effort mode, the cluster dedicates zero reserved memory for host failure scenarios. All physical RAM across all nodes can be 100% committed to running virtual machines.
• Failure Behavior: If an AHV host fails, the Acropolis Leader analyzes surviving hosts. If surviving hosts happen to have enough free physical RAM to satisfy the failed VMs' 100% pinned reservations, the VMs restart successfully.
• Risk Factor: If the surviving nodes are heavily utilized and lack sufficient free RAM, some or all failed VMs cannot restart. Those VMs remain in a powered-off state until physical hardware is restored or other workloads are manually shut down.
• Target Environment: Dev/test environments, non-critical lab clusters, or environments where maximum workload density overrides availability guarantees.

2. Guarantee Mode (Production Best Practice)

• Operational Model: In Guarantee mode, the cluster guarantees that sufficient compute and memory capacity is perpetually reserved to restart all protected VMs in the event of one (N+1) or two (N+2) physical host failures.
• Failover Segments Calculation: Rather than requiring administrators to manually designate a static, idle "standby host" (which wastes expensive hardware), AHV calculates Failover Segments. AHV examines the memory capacity of the largest node in the cluster and reserves an equivalent slice of aggregate memory distributed across all cluster nodes.
• Admission Control Enforcement: When Guarantee mode is active, Prism enforces strict Admission Control. If an administrator attempts to create or power on a new VM whose memory allocation would eat into the reserved failover segment, Prism blocks the action with an error: "Cannot power on VM because HA reservation would be violated."

[!IMPORTANT] Enabling HA Guarantee Mode requires that all nodes in the cluster have uniform network configurations and access to identical storage containers, ensuring any surviving host can successfully host any failed VM.

---

3. Acropolis Dynamic Scheduling (ADS)

In dynamic enterprise environments, virtual machine resource consumption fluctuates unpredictably. A batch billing job or midday database query spike can suddenly saturate a physical host's CPU or memory, degrading performance for all co-located workloads.

Acropolis Dynamic Scheduling (ADS) is Nutanix's native, automated workload balancing engine (comparable to VMware DRS, but completely autonomous and built directly into AHV without requiring external appliances).

+-------------------------------------------------------------------------+
|                     Acropolis Dynamic Scheduling (ADS)                  |
+-------------------------------------------------------------------------+
| 1. Continuous Telemetry: ADS samples CPU, memory, and CVM I/O every 10s |
| 2. Contention Detection: Host 01 exceeds 85% CPU / High CPU Ready       |
| 3. Migration Calculation: ADS identifies VM-Batch-02 as ideal candidate |
| 4. Non-Disruptive Live Migration: VM migrated to Host 03 in seconds     |
| 5. Equilibrium Restored: Workload balanced without user intervention    |
+-------------------------------------------------------------------------+

How ADS Operates

• Continuous Anomaly Detection: ADS continuously monitors real-time performance metrics across all physical AHV hosts—including physical CPU utilization, CPU Ready Time (%RDY), physical memory pressure, and local CVM storage throughput.
• Intelligent Migration Planning: When ADS detects compute or storage contention on a host, it calculates an optimization plan. Rather than blindly moving random VMs, ADS evaluates VM resource footprints, memory page dirtiness rates, NUMA topologies, and configured affinity rules to select the candidate that relieves the hotspot with the lowest migration cost.
• Non-Disruptive Live Migration: ADS triggers a native AHV Live Migration:
  1. The VM's memory pages are pre-copied across the high-speed network (10/25/100 GbE) to the target host while the VM continues executing on the source host.
  2. In the final phase, CPU execution pauses for a fraction of a millisecond while the last dirty memory pages transfer.
  3. Execution resumes seamlessly on the target host. Active TCP network connections, database transactions, and user sessions remain uninterrupted.

NUMA and Storage-Aware Scheduling

ADS does not merely look at raw CPU percentages; it is deeply aware of Non-Uniform Memory Access (NUMA) boundaries and storage locality:

• If a multi-vCPU VM is spanning sockets unnecessarily, ADS attempts to migrate co-located workloads to free up an entire NUMA node for the multi-vCPU workload.
• ADS coordinates with the local CVM Stargate service to factor in network fabric bandwidth, ensuring migrations do not congest storage replication pathways.

---

4. Affinity Rules: VM-to-Host and VM-to-VM Anti-Affinity

While ADS automatically distributes workloads for maximum performance, enterprise governance and architectural best practices frequently demand strict control over where specific VMs are permitted to execute.

Nutanix AHV provides two foundational policy mechanisms to govern VM placement: VM-to-Host Affinity and VM-to-VM Anti-Affinity.

+-------------------------------------------------------------------------+
|                          Affinity Rule Types                            |
+------------------------------------+------------------------------------+
|         VM-to-Host Affinity        |      VM-to-VM Anti-Affinity        |
|    "Where can a VM run?"           |   "Who can a VM NOT run with?"     |
+------------------------------------+------------------------------------+
| - Primary Driver: Software License | - Primary Driver: Redundancy / HA  |
| - Oracle / MS SQL Per-Core Rules   | - Domain Controllers (DC01 & DC02) |
| - Hardware Specifics (GPUs, NICs)  | - Web Load Balancers / DB Clusters |
| - Strict vs Non-Strict Modes       | - Prevents Single Point of Failure |
+------------------------------------+------------------------------------+

1. VM-to-Host Affinity Rules (Licensing & Hardware Constraints)

VM-to-Host affinity binds a virtual machine to a specific subset of physical AHV hosts within the cluster.

The Software Licensing Imperative

Enterprise software vendors (most notably Oracle and Microsoft) frequently audit virtualization platforms. Under per-core or per-socket licensing contracts, if a database virtual machine is permitted to freely migrate across all 16 nodes in an enterprise cluster, the vendor may legally demand that the customer purchase licenses for all physical cores across all 16 nodes—costing hundreds of thousands of dollars.

By creating a VM-to-Host affinity rule binding database workloads exclusively to Node 01 and Node 02, the organization strictly bounds its licensing audit scope to those two physical hosts.

Strict vs. Non-Strict Affinity Modes

• Strict Affinity ("Must Run On"): The VM is strictly confined to the designated hosts. If both Node 01 and Node 02 experience hardware failures, AHV HA will never restart the VM on Node 03 or Node 04, protecting the organization from catastrophic software licensing non-compliance.
• Non-Strict Affinity ("Should Run On"): The VM prefers to run on the designated hosts during normal operations. However, if all designated hosts crash, AHV High Availability overrides the rule and restarts the VM on any available surviving host to ensure business continuity.

2. VM-to-VM Anti-Affinity Rules (High Availability & Redundancy)

VM-to-VM anti-affinity rules mandate that two or more specified virtual machines must never run on the same physical host simultaneously.

Architectural Justification

Enterprise applications achieve high availability by deploying redundant pairs:

• Active Directory Domain Controllers (DC01 and DC02).
• Clustered Database Nodes (SQL Always-On Availability Groups SQL-NodeA and SQL-NodeB).
• Clustered Firewalls or Web Proxies.

If ADS or an administrator inadvertently places both DC01 and DC02 onto physical Node 03, and Node 03 suffers an unexpected motherboard failure, both domain controllers crash simultaneously, causing an enterprise-wide authentication outage. Configuring a VM-to-VM Anti-Affinity rule ensures that ADS perpetually keeps DC01 and DC02 on separate physical hosts.

How Anti-Affinity Interacts with HA and ADS

• ADS Enforcement: ADS treats anti-affinity as a hard placement filter. It will never recommend or execute a live migration that would consolidate anti-affinity peers onto a single host.
• HA Failover Behavior: During a host failure, the Acropolis Leader analyzes surviving hosts to find a node that does not host an anti-affinity peer. If the cluster is down to its last host and no alternative exists, AHV prioritizes keeping workloads alive, unless strict isolation policies are programmatically enforced.