2.2 Controller VM (CVM) Architecture, Datapath, and Autopathing

Controller VM (CVM) Architecture, Datapath, and Autopathing

Core Concept: The Controller VM (CVM) is the software engine that powers the Nutanix Distributed Storage Fabric (DSF). Running as a dedicated virtual appliance on each physical host, the CVM abstracts local drives into a shared cluster-wide pool. By coupling hardware-level PCI passthrough with an intelligent local datapath and automated network failover (autopathing), Nutanix delivers ultra-low latency along with non-disruptive high availability.

To pass the NCA exam, administrators must understand how the CVM interacts directly with physical hardware, how guest virtual machine I/O flows through the hypervisor, and what happens behind the scenes during maintenance operations like rolling upgrades.

---

The Controller VM (CVM) Architecture & Hardware Abstraction

In traditional storage architectures, proprietary dual-controller SAN heads process all storage traffic, creating significant scalability bottlenecks. Nutanix eliminates this bottleneck by running a Controller VM (CVM) on every node in the cluster.

+-------------------------------------------------------------------------+
|                              PHYSICAL NODE                              |
|                                                                         |
|  +--------------------+  +--------------------+  +--------------------+ |
|  |    Guest VM 1      |  |     Guest VM 2     |  |   Controller VM    | |
|  |   (User Workload)  |  |   (User Workload)  |  |       (CVM)        | |
|  +---------+----------+  +---------+----------+  +---------+----------+ |
|            |                       |                       |            |
|  Virtual   | SCSI I/O              | SCSI I/O              | Direct     |
|  Hardware  v                       v                       | Control    |
|  ===============================================           |            |
|     Hypervisor (Nutanix AHV, ESXi, Hyper-V)                |            |
|  ===============================================           |            |
|            |                                               |            |
|            +-----> Routes via virbr0 (192.168.5.254) ------>+            |
|                                                            |            |
|  ========================================================  |            |
|    Physical Server Hardware & System Motherboard           |            |
|  ========================================================  |            |
|        |                                                   |            |
|        | PCI Passthrough (Bypasses Hypervisor Kernel)      |            |
|        +==================================================>+            |
|                                                            |            |
|  [ Physical SAS/SATA HBA / NVMe Controller ] <-------------+            |
|  [ Local Solid-State Drives (SSDs) & Hard Drives (HDDs) ]               |
+-------------------------------------------------------------------------+

Purpose and Role of the CVM

The CVM is a hardened virtual appliance running a customized Linux operating system (CentOS/Rocky Linux foundation). It hosts all core AOS daemons—including Stargate, Cassandra, Zookeeper, and Curator. The collection of CVMs across all nodes communicates over the top-of-rack Ethernet network to present a single, pooled storage architecture to hypervisors and guest workloads.

Direct Hardware Control via PCI Passthrough (DirectPath I/O)

Traditional hypervisor architectures require the hypervisor kernel to manage storage device drivers, create file systems, and translate storage commands. This abstraction layer introduces queue contention, CPU overhead, and latency.

Nutanix solves this through PCI Passthrough (known as DirectPath I/O in VMware ESXi and VFIO/PCI Assignment in Nutanix AHV):

• The physical host bus adapter (LSI SAS controller) or direct PCIe NVMe controllers are unbound from the hypervisor kernel.
• The hypervisor assigns these physical PCI devices directly to the CVM virtual machine.
• The CVM boots with native vendor hardware drivers, giving AOS exclusive, bare-metal control over every physical SSD and HDD in that node's drive bays.
• The hypervisor itself has no direct filesystem access to the underlying physical drives; it only accesses storage once the CVM presents it back over standard virtual storage protocols.

CVM Resource Sizing and Memory Reservations

Because the CVM manages data consistency, inline compression, deduplication, caching, and network replication, allocating sufficient compute and memory is critical:

• Default Production Sizing: Typically configured with 32 GB of vRAM and 8 to 12 vCPUs.
• Memory Scaling Requirements: Memory can be expanded to 40 GB, 48 GB, or 64+ GB per CVM when advanced AOS features are enabled, including:
  • In-line and post-process deduplication (requires larger fingerprint cache in RAM).
  • Nutanix Files (File Server VMs leverage CVM memory for file system metadata).
  • High-capacity storage nodes exceeding 40 TB to 80 TB of raw disk capacity.
• 100% Memory Reservation: The hypervisor strictly enforces a 100% memory reservation for the CVM. CVM memory can never be ballooned, reclaimed, or swapped to hypervisor disk, ensuring consistent sub-millisecond storage responses.

Dedicated Internal Networking: virbr0 and 192.168.5.254

To communicate efficiently without congesting physical switches, Nutanix establishes a dedicated, isolated private virtual network between the hypervisor and the local CVM:

• Internal Virtual Bridge: On Nutanix AHV hosts, this private network is configured as a native Linux virtual bridge named virbr0 (or an isolated private vSwitch port group on ESXi).
• Reserved Subnet: Uses the private IP subnet 192.168.5.0/24.
• Addressing: The hypervisor host interface always assumes IP 192.168.5.1, while the local CVM assumes the loopback IP 192.168.5.254.
• Security & Isolation: This internal network has no external physical uplinks; storage traffic exchanged over this bridge never traverses physical top-of-rack switches or external network cables.

---

The AOS Datapath and Data Locality

Nutanix AOS executes a highly optimized storage pipeline designed around Data Locality—the principle that a virtual machine's storage data should reside on the exact physical server where that virtual machine's compute vCPUs are executing.

Write I/O Pipeline and Synchronous Replication

Every write operation generated by a guest VM follows a strict synchronous multi-node path before being acknowledged to the operating system:

+----------+   1. Write    +------------+  2. virbr0   +-----------+
| Guest VM | ------------> | Hypervisor | -----------> | Local CVM |
+----------+               +------------+              +-----+-----+
     ^                                                       |
     |                                       +---------------+---------------+
     | 8. Write Complete                     | 3. Local Write                | 4. Sync Net Copy
     |    (ACK)                              v                               v
+----+-----+                           +-----------+                   +-----------+
| Guest OS |                           | Local SSD |                   | Peer CVM  |
+----------+                           |  (OpLog)  |                   +-----+-----+
                                       +-----------+                         |
                                                                             | 5. Remote Write
                                       +-----------+   7. Remote ACK         v
                                       | Local CVM | <---------------- +-----------+
                                       +-----------+                   | Remote SSD|
                                                                       |  (OpLog)  |
                                                                       +-----------+

1. SCSI Request: The guest virtual machine issues an ordinary SCSI or NVMe write request via its virtual disk controller (virtio-scsi on AHV, pvscsi on ESXi).
2. Hypervisor Forwarding: The hypervisor redirects the write across virbr0 to the local CVM at 192.168.5.254.
3. Local Write Placement: The local CVM's Stargate daemon receives the write and immediately commits it to the local high-speed write cache (OpLog) backed by low-latency NVMe/SSD storage.
4. Synchronous Network Replication: Simultaneously, the local Stargate identifies a remote CVM on a different physical node (based on failure domain and rack awareness rules) and transmits the write payload over the physical 10GbE/25GbE network.
5. Remote Commit: The peer CVM's Stargate commits the replica payload to its own local OpLog.
6. Peer Acknowledgment: The remote CVM returns an acknowledgment (ACK) across the 10GbE/25GbE link to the local CVM.
7. Local Write Acknowledgment: Once both the local write AND the remote replication ACK are safely committed, the local CVM returns a write completion acknowledgment to the hypervisor.
8. Guest Confirmation: The hypervisor signals completion to the guest operating system.

[!NOTE] In Redundancy Factor 2 (RF2), 1 local write + 1 remote replication = 2 total copies. In Redundancy Factor 3 (RF3), 1 local write + 2 remote replications across two separate nodes = 3 total copies. In both cases, the guest OS is not acknowledged until all copies are confirmed on persistent non-volatile media.

Read I/O Pipeline and Data Locality

Read performance is where Nutanix hyperconvergence radically outperforms traditional SAN storage:

• When a guest VM requests a read, the hypervisor passes the request to the local CVM.
• Data Locality: Because the guest VM performed its initial writes locally, the requested data extents are already present on the direct-attached SSDs or in local CVM RAM cache.
• Zero Network Latency: The local CVM fulfills the read directly from local flash or memory. Zero packets traverse the top-of-rack physical switch, eliminating network queuing, switch latency, and bandwidth bottlenecks.

Handling VM Migrations (vMotion / Live Migration)

When an administrator or automated resource scheduler (like AHV Dynamic Scheduling - ADS) migrates a running virtual machine to a different host:

• The VM immediately begins executing compute cycles on the new node.
• Storage data extents initially remain on the previous host's physical disks.
• When the VM issues a read, the new local CVM detects that the required extent is remote. It retrieves the data across the 10GbE/25GbE network from the remote CVM.
• Dynamic Localization: As the new local CVM reads remote extents, it automatically copies ("localizes") those active blocks onto its own local SSD tier in the background. Subsequent reads are satisfied locally once again, restoring 100% data locality without manual intervention.

---

Autopathing and High Availability Redirection

What happens when a CVM is taken offline—for example, during a non-disruptive Nutanix AOS rolling upgrade, a CVM memory upgrade, or an unexpected CVM kernel crash?

Because guest VMs run directly on the hypervisor and rely on the local CVM for storage I/O, Nutanix engineers designed an automated, kernel-level failover mechanism called Autopathing (also called High Availability Redirection).

=================================================================================
NORMAL STATE (Local Datapath)
Guest VM ---> Hypervisor ---> virbr0 (192.168.5.254) ---> Local CVM ---> Local Storage
=================================================================================

=================================================================================
AUTOPATHING FAILOVER (Local CVM Offline / Maintenance)
Guest VM ---> Hypervisor ---> [192.168.5.254 UNRESPONSIVE]
                   |
                   +=====> Physical 10/25GbE NIC =====> Remote CVM (Peer Node)
                                                             |
                                                             v
                                                  Peer Storage / Cluster DSF
=================================================================================

The Autopathing Sequence Step-by-Step

1. Failure Detection: The local CVM powers down, reboots, or crashes. The hypervisor detects that the local IP endpoint (192.168.5.254) is no longer responding to storage heartbeats.
2. Immediate Redirection: The hypervisor's internal routing table dynamically redirects storage packets away from the internal virbr0 bridge and routes them out through the physical network interfaces (br0 on AHV).
3. Peer CVM Takeover: The traffic targets the external IP address of a healthy CVM running on an adjacent node in the cluster.
4. Uninterrupted Service: The remote CVM's Stargate daemon accepts the redirected iSCSI/NFS storage commands, reads and writes data across the Distributed Storage Fabric, and returns acknowledgments over the physical network.
5. Guest VM Transparency: From the perspective of the guest virtual machine, storage I/O never stops. Guest OS disk queues experience a brief, minor sub-second latency increase, but remain far below standard operating system SCSI timeout limits (which default to 30 to 60 seconds in Windows and Linux). No virtual machines crash or reboot.

Automated Failback Upon CVM Restoration

Once maintenance completes or the local CVM successfully reboots:

1. The local CVM initializes its operating system, validates file systems, and starts Genesis.
2. Genesis starts Zookeeper, Cassandra, and Stargate.
3. The local CVM rejoins the cluster quorum and reports healthy status.
4. The hypervisor detects that 192.168.5.254 is responding and healthy.
5. The hypervisor transparently shifts storage routing back to the local virbr0 bridge.
6. Local datapath resumes, and background Curator processes gradually localize new blocks written during the failover period.