4.3 Data Locality Mechanics and Intelligent Lifecycle Management (ILM)

Data Locality Mechanics and Intelligent Lifecycle Management (ILM)

Quick Summary: In traditional virtualization architectures, storage read and write traffic traverses external network switches to reach centralized storage controllers. Nutanix eliminates this network bottleneck through Data Locality—placing virtual machine data blocks directly on the physical storage media of the hypervisor node running that VM. Over 90% of read requests are satisfied locally over a private internal vSwitch. Write operations follow a low-latency synchronous replication pipeline through the local and remote OpLogs before acknowledging completion. When a VM live-migrates to another node, data locality dynamically re-establishes itself on the destination node as active blocks are read. Meanwhile, Intelligent Lifecycle Management (ILM) continuously migrates hot data to flash and demotes cold data to high-capacity storage tiers.

---

1. The Principle and Architectural Necessity of Data Locality

In conventional SAN and NAS architectures, 100% of storage traffic traverses the physical network. Every time a database performs a 4 KB read or an application writes a log file, the request travels across hypervisor network interfaces, physical Top-of-Rack (ToR) switches, storage array director ports, controller CPUs, and back again. Even with high-speed 25 GbE or 32 Gb Fibre Channel fabrics, network hops introduce packet serialization delays, buffer bloat, and contention during peak workload spikes.

Nutanix solves this through the principle of Data Locality:

• Compute and Storage Collocation: In a Nutanix cluster, compute (hypervisor and VMs) and storage (CVM and direct-attached drives) reside inside the exact same physical server node.
• Local Read Acceleration: The local Controller VM intercepts storage requests directly from the local hypervisor kernel via an internal private virtual network (vSwitch). Read operations for data residing on the local node are fulfilled directly from local NVMe or SSD media—zero storage traffic traverses the physical ToR switches.
• Network Bandwidth Conservation: Because 90% to 95% of typical enterprise storage reads are served locally, physical 10/25/100 GbE network interfaces remain idle and unconstrained, reserving full switch backplane bandwidth exclusively for write replication, cluster management, and VM-to-VM application traffic.

Traditional SAN Architecture (100% I/O Across Network Switches)
[VM on Host 1] ===> [Host HBA] ===> [ToR Switch] ===> [SAN Controller] ===> [Disk Shelf]

Nutanix HCI Architecture (Read I/O Stays 100% Local)
+-------------------------------------------------------------+
|                        Physical Node 1                      |
|                                                             |
|  [Guest VM] <=== (Internal vSwitch) ===> [Local CVM/Stargate]|
|                                                  |          |
|                                                  v          |
|                                      [Local NVMe / SSD / HDD]|
+-------------------------------------------------------------+
(Zero physical network hops for local reads!)

---

2. The AOS Write Pipeline: OpLog, Replication, and Extent Store Drain

While read operations benefit immediately from local media, write operations must guarantee data persistence, consistency, and fault tolerance across independent nodes to uphold the configured Redundancy Factor (RF2 or RF3).

Detailed Write Path Workflow

To provide both microsecond write response times and absolute crash consistency, AOS routes writes through a multi-stage distributed pipeline:

                         Guest Virtual Machine
                                  |
                      1. SCSI Write Command
                                  v
                          Local CVM (Stargate)
                                  |
             +--------------------+--------------------+
             |                                         | (Physical 25GbE Network)
    2. Write to Local OpLog                   3. Synchronous Remote Replicas
             |                                         |
             v                                         v
     [Local NVMe/SSD]                          [Remote CVM OpLog]
             |                                         |
             +--------------------+--------------------+
                                  |
                      4. Remote & Local Ack
                                  v
                      5. Write Acknowledged to VM
                                  |
           (Asynchronous Background Drain via Stargate)
                                  v
                 Extent Store (Local Persistent Flash/HDD)

1. SCSI Write Issuance: The guest virtual machine issues a SCSI write command to its virtual disk.
2. Local CVM Interception: The local hypervisor routes the write I/O over the internal loopback or private vSwitch directly to the local CVM's Stargate daemon.
3. Local OpLog Commitment: Stargate analyzes the write size. Random, small block writes (e.g., 4 KB to 32 KB database updates) are written immediately to the local OpLog. The OpLog is a high-speed, persistent write buffer allocated on ultra-fast NVMe or PCIe/SATA SSDs that acts like a distributed battery-backed NVRAM cache.
4. Synchronous Network Replication: Concurrently with the local write, Stargate replicates the exact same write payload across the physical network to the OpLog of one remote CVM (under RF2) or two remote CVMs (under RF3). Replicas are assigned to remote nodes according to failure domain rules (node-, block-, or rack-awareness).
5. Write Completion Acknowledgment (ACK): Once both the local OpLog and the remote OpLog(s) confirm the write has been successfully committed to persistent flash, Stargate returns a write completion acknowledgment to the guest VM. The entire synchronous loop typically executes in under 1 millisecond.
6. Sequential Direct-to-Extent Store Bypass: If the incoming write is a large, contiguous sequential write (e.g., streaming video or large file copy), Stargate intelligently bypasses the OpLog entirely and writes the data directly into the persistent Extent Store, avoiding unnecessary wear and cache pollution on the OpLog.
7. Asynchronous Extent Store Drain: In the background, Stargate aggregates, coalesces, and compresses writes held in the OpLog, draining them sequentially into the persistent Extent Store on local flash or hybrid media, freeing up OpLog space for incoming bursts.

---

3. VM Live Migration Dynamics and Restoring Data Locality

In modern virtualized datacenters, virtual machines frequently migrate across physical hosts for resource re-balancing (e.g., via VMware vSphere vMotion or Nutanix AHV Live Migration) or during non-disruptive rolling hypervisor upgrades.

A fundamental question arises: When a VM live-migrates from Node A to Node B, what happens to its multi-terabyte virtual disk?

The Problem with Bulk Storage Migration

If a storage platform attempted to copy an entire 2 TB virtual disk across the network during a live migration, the migration would take hours, saturate the Top-of-Rack network switches, create severe storage latency spikes, and completely negate the agility of dynamic workload mobility.

The Nutanix Solution: Storage Decoupling and On-Demand Locality

Nutanix decouples virtual machine memory migration from storage placement:

1. Zero Storage Migration at Handoff: During an AHV Live Migration or vMotion, only the VM's active CPU state, registers, and volatile RAM are transferred across the network. The vDisk data remains 100% untouched on Node A's physical drives. The migration finishes in seconds.
2. Immediate Remote Read State: Immediately following migration, the VM begins executing on Node B. When the VM executes a read request for an existing data block, Node B's local CVM intercepts the request. Recognizing that the requested extent resides on Node A, Node B's CVM requests the block from Node A's CVM over the physical network.
3. Dynamic Re-Establishment of Locality: Node A's CVM transmits the extent across the network to Node B's CVM. Node B's CVM returns the data to the VM and simultaneously writes a copy of that extent into its local Extent Store on Node B's SSD tier.
4. 100% Local Subsequent Reads: The next time the VM reads that same extent, Node B's CVM serves the request directly from its own local SSD—data locality for that block has been restored!
5. Immediate Local Writes: Any new write operations issued by the VM on Node B are committed directly to Node B's local OpLog and replicated to a remote node. Node A is no longer involved in new write operations.
6. Preservation of Cold Data: Extents belonging to the VM that are never read or written (e.g., cold archival logs or dormant operating system binaries) are never pulled across the network. They remain safely on Node A or other cluster nodes, preventing unnecessary network traffic and saving local SSD capacity on Node B.

VM Live Migration: Dynamic Locality Re-Establishment

Step 1: VM Migrates from Node A to Node B (Storage stays on Node A)
+-------------------------------+     +-------------------------------+
|            Node A             |     |            Node B             |
| [vDisk Data Stays on Node A]  |     | [VM Starts Running on Node B] |
+-------------------------------+     +-------------------------------+

Step 2: VM on Node B Issues Read -> Node B CVM Pulls Block & Caches Locally
+-------------------------------+     +-------------------------------+
|            Node A             |     |            Node B             |
| [Source Extent]               |====>| [Local CVM Fetches Extent]    |
|                               |     | [Writes Extent to Local SSD]  |
|                               |     | [Returns Data to Guest VM]    |
+-------------------------------+     +-------------------------------+

Step 3: Subsequent Reads Served 100% Locally from Node B's SSD
+-------------------------------+     +-------------------------------+
|            Node A             |     |            Node B             |
| [Old Cold Extent Stays Idle]  |     | [Local CVM Serves Read] <===  |
|                               |     | [Local SSD]                   |
+-------------------------------+     +-------------------------------+

[!NOTE] Shadow Clones: In scenarios where multiple virtual machines across different nodes read from the same base vDisk (such as Virtual Desktop Infrastructure / VDI linked clones or large container base images), Nutanix automatically activates Shadow Clones. The base vDisk is marked read-only and automatically cached across the local SSD tier of every node hosting a consumer VM, delivering instant local read performance without administrative intervention.

---

4. Intelligent Lifecycle Management (ILM) and Multi-Tier Storage

Modern hyperconverged infrastructure must balance extreme performance with economical storage density. To achieve this, Nutanix clusters support hybrid storage topologies combining high-performance solid-state media with high-capacity spinning disks or QLC flash.

The Storage Hierarchy Tiers

• Tier 0 (Flash Tier): High-speed NVMe PCIe SSDs, SAS SSDs, and SATA SSDs. Delivers tens of thousands of IOPS with sub-millisecond latency.
• Tier 1 (HDD / Capacity Tier): High-capacity SATA hard disk drives or dense QLC flash devices. Delivers cost-effective multi-terabyte capacity for archival and infrequently accessed data.

+-------------------------------------------------------------------------+
|                   Intelligent Lifecycle Management (ILM)                |
+-------------------------------------------------------------------------+
|  Tier 0: Flash Tier (NVMe / SATA SSD)                                   |
|  +-------------------------------------------------------------------+  |
|  | [Hot Working Set]  [Active Database Tables]  [Recently Written]   |  |
|  +-------------------------------------------------------------------+  |
|          ^                                                   |          |
|          | (Promote Hot Data on Read)    (Curator Demotes    |          |
|          |                                Cold Extents)      v          |
|  +-------------------------------------------------------------------+  |
|  | [Cold Backups]     [Dormant OS Files]        [Archival Logs]      |  |
|  +-------------------------------------------------------------------+  |
|  Tier 1: Capacity Tier (High-Density HDD / QLC SSD)                     |
+-------------------------------------------------------------------------+

Real-Time vs. Background ILM

Nutanix manages data placement across tiers through two synchronized engines:

1. Real-Time Tiering (Stargate):
   • During active I/O, Stargate dynamically determines whether incoming writes should be committed to Tier 0 flash or streamed to capacity storage based on write size and tier utilization thresholds.
   • If the flash tier crosses a capacity threshold (e.g., 75% full), Stargate begins prioritizing real-time writes and pushes candidate extents to lower tiers.
2. Background ILM via Curator MapReduce:
   • Curator is the distributed cluster analytics coordinator that runs periodic background optimization passes across all nodes.
   • MapReduce Analytics: Curator inspects the Cassandra metadata database, evaluating access timestamps, read frequency, and write age for every extent in the cluster.
   • Cold Extent Demotion: Extents that have not been accessed within a defined period (e.g., cold data) are queued for background demotion. Stargate transfers these extents from the high-cost NVMe/SSD tier down to the high-capacity HDD tier, ensuring the flash tier retains ample free capacity for active working sets.
   • Hot Extent Promotion: If a previously cold extent residing in Tier 1 HDD is suddenly accessed by an application, Stargate serves the read request and immediately schedules the extent for promotion back into the Tier 0 flash tier. Subsequent reads and modifications occur at full solid-state speeds.

By uniting data locality with automated Intelligent Lifecycle Management, Nutanix AOS guarantees that mission-critical virtual machines experience bare-metal NVMe and SSD performance while organizations benefit from the storage economics of high-density capacity media.