4.2 Redundancy Factor (RF2 vs. RF3), Failure Domains, and Fault Tolerance

Redundancy Factor (RF2 vs. RF3), Failure Domains, and Fault Tolerance

Quick Summary: Unlike legacy storage systems that rely on rigid hardware RAID controllers and dedicated hot-spare disks, Nutanix AOS enforces data protection at the software layer through Redundancy Factor (RF). RF2 requires at least 3 nodes and tolerates 1 simultaneous failure by maintaining 2 data copies, while RF3 requires at least 5 nodes and tolerates 2 simultaneous failures by maintaining 3 data copies. Data replicas are intelligently placed across configurable failure domains—including Node-awareness, Block-awareness, and Rack-awareness. Following a component failure, the Curator service orchestrates an automated, distributed parallel rebuild across all surviving nodes, restoring full fault tolerance without waiting for physical hardware replacement.

---

1. Redundancy Factor Architecture: Why AOS Abandoned Hardware RAID

For decades, enterprise datacenters relied on hardware RAID (Redundant Array of Independent Disks) controllers within SAN and NAS arrays to protect against drive failures. While RAID 5 and RAID 6 were effective for smaller spinning disks, modern multi-terabyte drives (12 TB, 16 TB, and beyond) broke the hardware RAID model:

• Catastrophic Rebuild Times: Rebuilding a failed 16 TB drive in a RAID 5 array requires reading every sector on all surviving drives and writing to a single spare disk. This process can saturate controller bandwidth and take 24 to 72 hours.
• Secondary Failure Risk: During an extended multi-day rebuild, surviving drives undergo severe I/O stress. If a second drive experiences an unrecoverable read error (URE) or hardware failure in a RAID 5 group, the entire volume suffers total, unrecoverable data loss.
• Controller as Single Point of Failure: Hardware RAID cards introduce proprietary firmware dependencies, battery backup unit (BBU) failures, and single-chassis controller limits.

The Software-Defined Replication Paradigm

Nutanix AOS eliminates hardware RAID entirely. Physical host HBAs operate in pure IT mode (pass-through / Non-RAID), giving the Controller VM direct block-level access to each physical NVMe, SSD, and HDD drive. Data resilience is managed entirely by software through Redundancy Factor (RF):

• Granular Extent-Level Replication: Rather than mirroring whole disks or striping across rigid controller groups, DSF replicates data at the granular level of individual 1 MB extents across independent cluster nodes.
• Separation of Data and Metadata Resilience: AOS decouples virtual machine data replicas from metadata replicas. Cluster metadata (managed by Cassandra and ZooKeeper) is maintained at a higher replication factor to guarantee cluster quorum and state consistency even under severe multi-node partitions.

---

2. Comparative Analysis: Redundancy Factor 2 (RF2) vs. Redundancy Factor 3 (RF3)

Nutanix clusters operate under two fundamental redundancy models: RF2 and RF3. Administrators define the maximum Redundancy Factor at the cluster level during initial deployment, and can then assign RF2 or RF3 policies to individual Storage Containers.

Redundancy Factor 2 (RF2) - 2 Data Copies, Min 3 Nodes
+-----------------------+     +-----------------------+     +-----------------------+
|        Node 1         |     |        Node 2         |     |        Node 3         |
| +-------------------+ |     | +-------------------+ |     | +-------------------+ |
| | Extent A (Copy 1) | |====>| | Extent A (Copy 2) | |     | | (Free Spare Space)| |
| +-------------------+ |     | +-------------------+ |     | +-------------------+ |
+-----------------------+     +-----------------------+     +-----------------------+

Redundancy Factor 3 (RF3) - 3 Data Copies, Min 5 Nodes
+-------------+ +-------------+ +-------------+ +-------------+ +-------------+
|   Node 1    | |   Node 2    | |   Node 3    | |   Node 4    | |   Node 5    |
| +---------+ | | +---------+ | | +---------+ | | +---------+ | | +---------+ |
| | Extent B| |>| | Extent B| |>| | Extent B| | | | (Spare) | | | | (Spare) | |
| | (Copy 1)| | | | (Copy 2)| | | | (Copy 3)| | | |         | | | |         | |
| +---------+ | | +---------+ | | +---------+ | | +---------+ | | +---------+ |
+-------------+ +-------------+ +-------------+ +-------------+ +-------------+

Detailed Technical Comparison

Cluster RF vs. Container RF

It is critical to distinguish between Cluster RF and Container RF:

• Cluster Redundancy Factor: Sets the maximum allowable protection level for the entire cluster infrastructure, including Cassandra metadata and ZooKeeper consensus engines. A cluster configured as Cluster RF3 can host both RF2 and RF3 storage containers.
• Container Redundancy Factor: Specifies the replication level for guest VM extents stored inside that specific container. On an RF3-enabled cluster, an administrator can assign RF2 to non-critical development containers (to maximize capacity) while assigning RF3 to core database containers (to maximize availability).

[!IMPORTANT] A cluster configured for Cluster RF2 cannot host RF3 storage containers, because the underlying metadata engine and consensus ring only maintain 3 copies of cluster metadata. To support RF3 containers, the cluster must be configured for Cluster RF3 (requiring a minimum of 5 nodes).

---

3. Failure Domains: Node, Block, and Rack Awareness

A failure domain defines the physical boundary of infrastructure that can fail simultaneously without causing data unavailability or data loss. Nutanix AOS supports hierarchical failure domains that insulate workloads against hardware outages.

1. Disk-Level Failure Domain

If a single physical NVMe, SSD, or HDD fails within a node, Stargate detects the I/O error. The drive is marked offline, and the cluster immediately begins re-replicating all extents that resided on that drive to free space distributed across surviving drives on other nodes.

2. Node-Awareness (Default)

Node-awareness is the default failure domain for every Nutanix cluster:

• Placement Rule: When Stargate writes redundant data copies (Copy 1 and Copy 2 under RF2), it guarantees that the copies are written to different physical nodes.
• Protection: If an entire node suffers a motherboard failure, CPU failure, or power supply outage, the surviving node hosting the second copy continues serving read and write I/O immediately.

3. Block-Awareness (Chassis-Awareness)

Many Nutanix hardware platforms (such as the Nutanix NX-3000 and NX-8000 series) use multi-node appliances—for example, a 2U-4Node chassis where four independent server nodes share a single 2U physical enclosure, redundant power distribution units (PDUs), and shared cooling fan modules.

• The Risk: If a physical enclosure loses power or a backplane fails, all four nodes within that chassis lose power simultaneously. Under basic node-awareness, if Copy 1 and Copy 2 happened to be placed on Node 1 and Node 2 in that same chassis, both copies would become unavailable simultaneously!
• The Solution (Block-Awareness): Block-awareness configures DSF to recognize physical chassis enclosures. Stargate ensures that redundant copies of data and metadata are placed on nodes residing in different physical blocks (chassis).
• Prerequisites:
  • For RF2 Block-Awareness: Requires a minimum of 3 physical blocks (chassis).
  • For RF3 Block-Awareness: Requires a minimum of 5 physical blocks (chassis).

Block-Awareness (Chassis-Aware Placement under RF2)
+-----------------------------------+     +-----------------------------------+
|       Block A (Chassis 1)         |     |       Block B (Chassis 2)         |
| +---------------+ +-------------+ |     | +---------------+ +-------------+ |
| | Node 1        | | Node 2      | |     | | Node 3        | | Node 4      | |
| | [Extent A-1]  | |             | |     | | [Extent A-2]  | |             | |
| +---------------+ +-------------+ |     | +---------------+ +-------------+ |
+-----------------------------------+     +-----------------------------------+
                |                                           ^
                +==== Data Replica Guaranteed to Cross =====+
                      Physical Chassis Boundary

4. Rack-Awareness

In large enterprise deployments spanning dozens of server racks, a top-of-rack (ToR) switch failure or entire datacenter PDU failure could take down an entire rack of equipment:

• Rack-Aware Placement: DSF tracks the rack location of every node. Replicas of extents are distributed across distinct physical racks.
• Prerequisites: Minimum of 3 racks for RF2, and 5 racks for RF3.
• Result: An entire server rack can experience a catastrophic power outage or network collapse without causing downtime or data loss for running applications.

---

4. Self-Healing Mechanics and Distributed Parallel Rebuild

In traditional storage architectures, a disk failure initiates a rebuild that relies on a dedicated, idle "hot spare" drive. If a 12 TB drive fails, the spare drive must absorb 12 TB of sequential writes through a single storage controller, creating an acute bottleneck that leaves the array vulnerable for days.

Nutanix AOS revolutionizes fault tolerance through Distributed Self-Healing:

Traditional RAID Rebuild (Single Spare Bottleneck: 24-72 Hours)
[Failed Drive] ===(Reads all surviving drives)===> [Single Hot Spare Disk]

Nutanix Distributed Self-Healing (Parallel Swarm: 10-30 Minutes)
                  +-------------------------+
                  |  Node 1: [Drive Fails]  |
                  +-------------------------+
                               |
      +------------------------+------------------------+
      |                        |                        |
      v                        v                        v
+-------------+          +-------------+          +-------------+
|   Node 2    |          |   Node 3    |          |   Node 4    |
| [Reads Ext] |          | [Reads Ext] |          | [Reads Ext] |
| [Writes Ext]|          | [Writes Ext]|          | [Writes Ext]|
+-------------+          +-------------+          +-------------+

No Dedicated Hot Spares Required

Nutanix clusters do not waste expensive drives as idle hot spares. Instead, free space distributed across all active drives in the cluster serves as dynamic rebuild capacity. If a 4-node cluster has 20 TB of free capacity, that capacity is spread across all 24 or 48 drives in the cluster.

Step-by-Step Distributed Rebuild Lifecycle

When a component fails (e.g., Node 1 suffers a hardware failure in an RF2 cluster):

1. Failure Detection (<30 Seconds): The local Genesis and ZooKeeper daemons identify the missing heartbeats from Node 1. Stargate marks all extents that had a copy on Node 1 as degraded (RF1 state).
2. Curator MapReduce Job Triggering: The Curator service (the distributed analytics and cluster optimization engine) initiates an emergency MapReduce job. Curator scans the distributed Cassandra metadata to locate every degraded extent across the cluster.
3. Distributed Work Distribution: Curator slices the list of degraded extents into parallel task chunks and assigns them to the surviving CVMs across the entire cluster.
4. Parallel Swarm Re-Replication:
   • Node 2 reads surviving extent copies from its local drives and writes them across the network to Node 3.
   • Node 3 reads surviving extents and writes them to Node 4.
   • Node 4 reads surviving extents and writes them to Node 2.
   • Because every node and every physical drive participates in both reading and writing simultaneously, network and disk I/O are fully distributed.
5. Rapid RF Restoration (Minutes, Not Days): Even for multi-terabyte data volumes, the distributed parallel rebuild restores the cluster to full RF2 or RF3 redundancy in 10 to 30 minutes.
6. Return to Full Fault Tolerance: Once the distributed rebuild completes, the cluster is once again fully fault tolerant (capable of surviving another subsequent node or drive failure)—long before a technician arrives with replacement hardware.

[!TIP] When a technician physically replaces a failed drive in a Nutanix node, there is no urgent rebuild trigger. The cluster has already restored full redundancy using distributed free space. Inserting the new drive simply adds fresh raw capacity back into the global Storage Pool, and Curator re-balances data gradually in the background.