7.1 Server High Availability and Cluster Quorum

Server High Availability and Cluster Quorum

Core High Availability Principle: Enterprise server availability is measured in "nines" of uptime—from 99.9% (three nines, permitting up to 8.76 hours of unscheduled downtime annually) to 99.999% (five nines, permitting fewer than 5.26 minutes of downtime per year). Meeting these stringent Service Level Agreements (SLAs) requires eliminating single points of failure (SPOFs) at every architectural layer. Server clustering bonds independent physical or virtual compute nodes into a unified, fault-tolerant operational fabric governed by distributed consensus, automated heartbeat monitoring, and deterministic quorum voting.

Systems administrators preparing for the CompTIA Server+ (SK0-005) certification must master cluster failover architectures, low-latency heartbeat design, mathematical quorum voting algorithms, fencing mechanisms, and multi-site stretch clustering.

---

High Availability Architectures: Active-Active vs. Active-Passive

Clustering solutions aggregate two or more computing nodes running cluster coordination software (such as Windows Server Failover Clustering [WSFC] or Linux Pacemaker/Corosync) to guarantee continuous application availability.

+-----------------------------------------------------------------------------+
|                 High Availability Server Cluster Topologies                 |
|                                                                             |
|   ACTIVE-ACTIVE CLUSTER:                                                    |
|   [Client Traffic] ---> [Load Balancer / VIP]                               |
|                              |             |                                |
|                              v             v                                |
|                     +-------------+   +-------------+                       |
|                     |   Node 01   |   |   Node 02   |                       |
|                     |   Active    |   |   Active    |                       |
|                     | (50% Load)  |   | (50% Load)  |                       |
|                     +-------------+   +-------------+                       |
|                            \                 /                              |
|                      [Heartbeat / Shared Storage]                           |
|                                                                             |
|   ACTIVE-PASSIVE CLUSTER:                                                   |
|   [Client Traffic] ---> [Cluster Virtual IP (VIP)]                          |
|                              |                                              |
|                              v (100% Load)                                  |
|                     +-------------+   +-------------+                       |
|                     |   Node 01   |   |   Node 02   |                       |
|                     |   Active    |===|   Passive   | (Hot Standby)         |
|                     +-------------+   +-------------+                       |
|                            \                 /                              |
|                      [Heartbeat / Shared Storage]                           |
+-----------------------------------------------------------------------------+

Active-Active Clustering

In an Active-Active cluster configuration, all nodes simultaneously process incoming client transactions and execute application workloads.

• Resource Utilization and Throughput: Every server actively contributes compute, memory, and I/O capacity, maximizing return on hardware investment. Common implementations include web application server farms, stateless application microservices, and distributed database engines (such as Oracle Real Application Clusters [RAC] or Microsoft SQL Server Always On Availability Groups with read-intent secondary replicas).
• Failover Dynamics and RTO: Because all nodes are already initialized, running the application stack, and servicing active traffic, failover occurs almost instantaneously. The Recovery Time Objective (RTO)—the maximum tolerable duration of service interruption—is near zero. If Node 01 fails, the network ingress router or load balancer immediately redirects traffic to the surviving active nodes.
• Capacity Headroom ($N+1$ Sizing Rule): Administrators must avoid running active-active nodes at maximum capacity. In a two-node active-active cluster, neither node should continuously exceed 50% compute utilization. If each node runs at 80% capacity and one node fails, the surviving node will be overwhelmed by 160% load, triggering cascading hardware saturation and total cluster failure.

Active-Passive Clustering

In an Active-Passive cluster configuration, one node actively hosts application workloads and services client requests, while one or more secondary nodes remain in an idle standby state.

• Standby Operational Profiles: Standby nodes can operate as Hot Standby (operating system booted, cluster service running, database or application binaries loaded in RAM, awaiting instantaneous failover trigger), Warm Standby (node booted and connected to storage, but application services stopped until failover is invoked), or Cold Standby (hardware powered off, requiring manual boot, storage attachment, and service startup).
• Failover Mechanics and RTO/RPO Impact: Active-passive failover requires the cluster software to detect primary node death, arbitrate cluster ownership, mount shared storage volumes (LUNs), start database engines or application daemons, replay transaction write-ahead logs (WAL) to ensure ACID consistency, and bind the Cluster Virtual IP (VIP) or update DNS records. Consequently, RTO typically ranges from 30 seconds to several minutes. However, the Recovery Point Objective (RPO)—the allowable data loss measured in time—is zero when supported by synchronous shared storage.
• Failback Flapping Prevention: When a failed primary node reboots, automated failback can trigger a failback flap, where services migrate back and forth repeatedly while the recovered node stabilizes. Best practice mandates setting cluster properties to manual failback or establishing a delayed failback window during non-business hours.

---

Heartbeat Network Design and Inter-Node Communication

Clustered nodes maintain real-time awareness of peer health through a dedicated, continuous communication stream known as the Cluster Heartbeat.

+-----------------------------------------------------------------------------+
|                  Cluster Heartbeat Architecture and Fabric                  |
|                                                                             |
|   Node 01                                                   Node 02         |
|   +-----------------------+               +-----------------------+         |
|   | Production NIC        |===[Prod VLAN]==| Production NIC        |         |
|   | (Client Traffic)      |               | (Client Traffic)      |         |
|   +-----------------------+               +-----------------------+         |
|   | Dedicated Heartbeat   |===[Priv VLAN]=| Dedicated Heartbeat   |         |
|   | NIC (Point-to-Point)  |   (Non-Rout)  | NIC (Point-to-Point)  |         |
|   +-----------------------+               +-----------------------+         |
|   | BMC / iDRAC / IPMI    |===[Mgmt VLAN]=| BMC / iDRAC / IPMI    |         |
|   | (STONITH Power Fence) |               | (STONITH Power Fence) |         |
|   +-----------------------+               +-----------------------+         |
+-----------------------------------------------------------------------------+

Heartbeat Fabric Requirements

• Network Isolation: Heartbeat packets must never compete with heavy client production traffic, hypervisor live migration (vMotion), or high-throughput storage I/O (iSCSI/NFS). Saturated links drop UDP packets, causing the cluster engine to falsely believe a peer has died. Heartbeats must traverse a dedicated physical network interface card (NIC), a direct point-to-point cross-over cable (in two-node topologies), or a dedicated, non-routable private Virtual LAN (VLAN).
• Heartbeat Packet Mechanics: Windows Server Failover Clustering (WSFC) transmits heartbeat frames via UDP Port 3343 (unicast or multicast). Linux clusters utilizing Corosync transmit periodic UDP or TCP datagrams containing cluster membership sequence numbers, token rings, and health status flags.
• Heartbeat Thresholds and Timers:
  • Heartbeat Interval (Frequency): How often each node broadcasts a keepalive packet (default: every 1000 ms / 1 second in standard local clusters).
  • Missed Heartbeat Threshold: The number of consecutive missed heartbeats permitted before declaring a node dead (default: 5 to 10 missed heartbeats in local subnets, meaning 5–10 seconds of silence triggers failover).
  • Cross-Subnet and Stretched Clustering Timers: In multi-site or stretched clusters across high-latency WAN links, default timers will trigger false-positive failovers due to intermittent WAN jitter. Administrators must increase the threshold (e.g., 20 to 30 missed heartbeats with 1000 ms intervals, allowing a 20–30 second detection window).

# Windows Server Failover Cluster: Inspecting Heartbeat Thresholds
Get-Cluster | Select-Object SameSubnetDelay, SameSubnetThreshold, CrossSubnetDelay, CrossSubnetThreshold

# Adjusting heartbeat threshold for cross-subnet resilience
(Get-Cluster).CrossSubnetThreshold = 20
(Get-Cluster).CrossSubnetDelay = 1000

---

Cluster Quorum Mechanics: The Voting Algorithm

A cluster cannot operate reliably based solely on peer assumptions. If communications break, nodes must definitively determine which subset possesses the legal authority to run services. This consensus mechanism is called Quorum.

The Mathematical Majority Rule ($50% + 1$)

Quorum is governed by strict majority mathematics. To achieve quorum and remain operational, a cluster must maintain strictly more than half of the total configured votes:

$\text{Quorum Threshold} = \left\lfloor \frac{\text{Total Votes}}{2} \right\rfloor + 1$

If node failures or network partitions reduce the number of active votes below this threshold, the cluster loses quorum. In response, the cluster engine immediately halts all hosted applications and dismounts shared storage volumes. This intentional self-termination prevents corrupted concurrent writes.

+-----------------------------------------------------------------------------+
|                     Quorum Calculation and Vote Totals                      |
|                                                                             |
|   3-Node Cluster (Odd Node Count):                                          |
|   Total Votes = 3 (Node1=1, Node2=1, Node3=1)                               |
|   Quorum Needed = floor(3/2) + 1 = 1 + 1 = 2 Votes                          |
|   * Can tolerate 1 node failure (3 - 1 = 2 votes -> Quorum Maintained)      |
|   * 2 node failures -> 1 vote remaining -> Cluster Halts                    |
|                                                                             |
|   4-Node Cluster without Witness (Even Node Count - FLAWED):                |
|   Total Votes = 4 (Node1=1, Node2=1, Node3=1, Node4=1)                      |
|   Quorum Needed = floor(4/2) + 1 = 2 + 1 = 3 Votes                          |
|   * Can tolerate only 1 node failure (4 - 1 = 3 votes -> Quorum Maintained) |
|   * If 2 nodes fail -> 2 votes remain (2 < 3) -> Cluster Halts!             |
|                                                                             |
|   4-Node Cluster WITH Witness (Even Node Count + Witness - OPTIMAL):        |
|   Total Votes = 5 (4 Nodes + 1 Witness Vote)                                |
|   Quorum Needed = floor(5/2) + 1 = 2 + 1 = 3 Votes                          |
|   * Can tolerate 2 node failures (5 - 2 = 3 votes -> Quorum Maintained)     |
+-----------------------------------------------------------------------------+

• Even vs. Odd Node Topologies: Clusters with an odd number of nodes (3, 5, 7) possess an inherent natural tie-breaker. Clusters with an even number of nodes (2, 4, 6) face a fatal 50/50 split vulnerability if partitioned evenly. Therefore, even-node clusters must always incorporate a Quorum Witness to provide an odd vote total.
• Dynamic Quorum: Modern enterprise clustering platforms (Windows Server, Red Hat Enterprise Linux) implement dynamic quorum management. As nodes are gracefully shut down for scheduled maintenance, the cluster dynamically recalculates total votes downwards, allowing a 5-node cluster to step down to 3 votes, and subsequently 1 vote, surviving sequential planned removals.

---

Quorum Witness Models

A Quorum Witness is an independent voting entity that breaks ties in even-node clusters without running application workloads.

+-----------------------------------------------------------------------------+
|                        Quorum Witness Architecture                          |
|                                                                             |
|   [Node 01: 1 Vote] <=== Heartbeat ===> [Node 02: 1 Vote]                   |
|           \                                     /                           |
|            \                                   /                            |
|             v                                 v                             |
|     +-------------------------------------------------+                     |
|     |              QUORUM WITNESS OPTIONS             |                     |
|     |  1. Disk Witness: Shared SAN LUN (SCSI-3 Lock)  |                     |
|     |  2. File Share Witness: SMB 3.0 Share File Lock |                     |
|     |  3. Cloud Witness: Azure Blob / AWS S3 Blob Lock|                     |
|     +-------------------------------------------------+                     |
|                             |                                               |
|                             v (+1 Tie-Breaker Vote)                         |
|                  Total Cluster Votes = 2 + 1 = 3                            |
+-----------------------------------------------------------------------------+

1. Node Majority (No Witness)

Recommended strictly for clusters with an odd number of nodes (3, 5, 7) situated within a single data center. Every server node contributes exactly one vote. No external storage or third-party server infrastructure is required.

2. Node and Disk Majority (Disk Witness / Quorum Disk)

Deploys a dedicated, small shared storage Logical Unit Number (LUN)—typically 512 MB to 1 GB in size—formatted as NTFS, ReFS, or cluster filesystem. The LUN is provisioned on a shared Storage Area Network (SAN) accessible via Fibre Channel or iSCSI to all cluster nodes.

• Mechanics: The cluster engine places a raw cluster configuration database file on the LUN. The disk witness casts 1 vote. If nodes lose inter-node communication, each node attempts to acquire an exclusive SCSI-3 Persistent Reservation (PR) lock on the disk witness. The node or partition that successfully acquires the lock secures the witness vote and maintains quorum.
• Limitation: Requires expensive shared SAN infrastructure. If the shared storage array itself fails, the disk witness vote is lost.

3. Node and File Share Majority (File Share Witness - FSW)

Assigns an independent server, Network Attached Storage (NAS) appliance, or remote file server hosting a standard Server Message Block (SMB 3.0) share to serve as the witness.

• Mechanics: The file share does not store application data; it maintains a single small lock file (witness.log). When a partition event occurs, surviving nodes attempt to open the file with exclusive write-lock privileges. The partition that holds the lock gains the witness vote.
• Best Practice: The File Share Witness must reside on an independent physical server outside the cluster's compute and storage failure domain. It is the gold standard for multi-site stretch clusters where shared SAN block storage cannot span data centers.

4. Cloud Witness

Leverages public cloud object storage—such as Microsoft Azure Blob Storage or an Amazon Web Services (AWS) S3 bucket—as an external, off-site quorum arbiter.

• Mechanics: The cluster nodes establish secure outbound HTTPS (TCP Port 443) connections to the cloud storage endpoint. The cloud witness writes small timestamped blob files to mediate arbitration. It does not require hosting or patching an external virtual machine.
• Ideal Deployment: Multi-site stretch clusters and disaster recovery (DR) environments spanning two geographic data centers. By placing the witness in the cloud (a neutral third location), an entire data center failure does not sever quorum for the surviving remote site.

---

Preventing Split-Brain Scenarios and Fencing Mechanisms

The most dangerous operational failure in server clustering is the Split-Brain Scenario.

+-----------------------------------------------------------------------------+
|                 Split-Brain Failure vs. Fencing Execution                   |
|                                                                             |
|   UNPROTECTED SPLIT-BRAIN (CATASTROPHIC DATA CORRUPTION):                   |
|   [Node 01: Active] <--- HEARTBEAT SEVERED ---> [Node 02: Active]           |
|           |                                             |                   |
|           v (Writes data block A)                       v (Writes block B)  |
|   +-------------------------------------------------------------+           |
|   |           SHARED STORAGE / DATABASE LUN (CORRUPTED)         |           |
|   +-------------------------------------------------------------+           |
|                                                                             |
|   PROTECTED BY STONITH / FENCING:                                           |
|   [Node 01: Active] <--- HEARTBEAT SEVERED ---> [Node 02: Isolated]         |
|           |                                             |                   |
|           | 1. Detects heartbeat loss                   |                   |
|           | 2. Wins Quorum Arbitration                  |                   |
|           | 3. Sends IPMI/iDRAC "Power Off" command ===>| (Hard Power Off)  |
|           v                                                                 |
|   Safely mounts shared storage without collision!                           |
+-----------------------------------------------------------------------------+

Split-Brain Definition and Consequences

A split-brain condition occurs when the redundant heartbeat networks between cluster nodes fail completely, while the compute nodes themselves remain fully powered and operational.

• The Conflict: Lacking heartbeat keepalives, each side assumes the other has died. In an improperly configured cluster lacking quorum enforcement, both sides attempt to assume active ownership of the identical workload. Both nodes bind the same Cluster Virtual IP (VIP), creating duplicate ARP conflicts on the local network.
• The Catastrophe: Crucially, both nodes attempt to write simultaneously to the underlying shared SAN storage, filesystem, or database volume. Without coordinated distributed metadata locking, uncoordinated block allocations overwrite file allocation tables, corrupt database transaction logs, and cause unrecoverable data loss.

Fencing Mechanisms: STONITH and SCSI-3 PR

To prevent split-brain data corruption, cluster engines implement Fencing—the deterministic isolation and neutralization of malfunctioning or unreachable nodes.

1. STONITH (Shoot The Other Node In The Head)

STONITH is the definitive node-level fencing technique deployed across enterprise clusters (particularly Linux Pacemaker/Corosync fabrics). When a node detects heartbeat loss and secures quorum, it must guarantee with 100% certainty that the uncontactable peer cannot write to storage before mounting resources.

• Mechanics: The surviving node connects to the target node's out-of-band hardware management processor—such as Dell Remote Access Controller (iDRAC), HPE Integrated Lights-Out (iLO), or standard Intelligent Platform Management Interface (IPMI)—or commands an intelligent Switched Power Distribution Unit (ePDU). It issues an immediate hardware power-off or hard reset command.
• Guarantee: Once the physical power rail drops to zero volts, the fenced node is rendered inert. Only then does the surviving node mount shared filesystems.

2. Storage-Level I/O Fencing: SCSI-3 Persistent Reservations (PR)

Storage-level fencing neutralizes nodes at the SAN fabric level rather than the power rail.

• Mechanics: Under the SCSI-3 standard, nodes register an 8-byte reservation key with the storage array controller using the PROUT (Persistent Reserve Out) command. The cluster designates the active node as the exclusive reservation holder.
• Preemption: If Node 02 becomes unresponsive, Node 01 secures quorum and issues a SCSI-3 PREEMPT AND ABORT command to the storage array. The array instantly purges Node 02's reservation key and aborts all pending I/O requests from Node 02's Host Bus Adapter (HBA) World Wide Names (WWNs). Even if Node 02 continues attempting to write, the SAN fabric rejects its commands at the block layer.

# Linux: Querying cluster status and fencing configuration via pcs
pcs status
pcs stonith show

# Linux: Inspecting SCSI-3 persistent reservations on a shared block device
sg_persist --in --report-capabilities --device=/dev/sdb
sg_persist --in --read-keys --device=/dev/sdb

---

Multi-Site and Stretch Clusters (Geo-Clustering)

A Stretch Cluster (also known as a geo-cluster or multi-site cluster) spans two or more geographically separated data centers connected via high-speed optical fiber links, providing automated disaster recovery.

+-----------------------------------------------------------------------------+
|                   Multi-Site Stretch Cluster Architecture                   |
|                                                                             |
|   DATA CENTER A (Primary)                 DATA CENTER B (Secondary)         |
|   +-----------------------+               +-----------------------+         |
|   | Node 01: Vote 1       |               | Node 02: Vote 1       |         |
|   | Local SAN Storage A   |               | Local SAN Storage B   |         |
|   +-----------------------+               +-----------------------+         |
|               \                               /                             |
|                \                             /                              |
|        Inter-Site Metro Link (<5ms RTT Latency; Sync Storage Replication)   |
|                          \                 /                                |
|                           v               v                                 |
|                         +-------------------+                               |
|                         | THIRD SITE / CLOUD|                               |
|                         |  Quorum Witness   |                               |
|                         |     (Vote 1)      |                               |
|                         +-------------------+                               |
+-----------------------------------------------------------------------------+

Latency Constraints and Synchronous Storage Replication

• Round-Trip Time (RTT) Ceiling: In a stretch cluster hosting live, transactional databases or virtual machine clusters, storage blocks must be written to both data centers simultaneously before acknowledging the write to the application (Synchronous Storage Replication). Because light in fiber travels at roughly 200 km per millisecond, physical distance imposes absolute latency floors.
• The 5 ms Latency Rule: Enterprise stretch clusters mandate an end-to-end network round-trip time of less than 5 milliseconds ($<5\text{ ms}$). Latencies exceeding 5 ms degrade database write transactions catastrophically and cause cluster heartbeat packet drops. For distances exceeding 50 to 100 kilometers, organizations must transition to Asynchronous Replication, which introduces non-zero RPO and eliminates automatic cluster-level failover.

The Triangular Third-Site Witness Mandate

In a two-site stretch cluster with one node in Data Center A and one node in Data Center B, placing the Quorum Witness in either Site A or Site B creates a fatal asymmetry:

• If Site A hosts both Node 01 and the Witness (2 votes total), and the WAN link severs, Site A maintains quorum (2 of 3 votes) while Site B shuts down. However, if Data Center A experiences a complete physical disaster (e.g., utility blackout, flood), Site B has only 1 vote out of 3 and cannot come online, defeating the entire disaster recovery investment.
• The Architectural Mandate: The Quorum Witness must reside in an independent third location (a third corporate office or a Cloud Witness). If either primary data center suffers total physical destruction, the surviving data center pairs with the independent Cloud Witness to achieve 2 out of 3 votes, sustaining automated failover without human intervention.

Fault Tolerance Tiers: Component Redundancy vs. Server-Level Redundancy

SK0-005 draws an explicit line between component redundancy and server-level redundancy, and scenario questions hinge on knowing which tier a proposed fix actually buys.

The practical rule is that component redundancy keeps one server alive; server-level redundancy keeps the service alive. A server with dual PSUs, dual NICs, and RAID 6 still goes offline for every kernel patch, every firmware update, and every motherboard replacement — planned downtime that component redundancy cannot address at all. Only a second node can absorb that.

The inverse trap is equally common on the exam: a two-node cluster built on one shared storage array, one top-of-rack switch, or one power feed has server-level redundancy on paper and a single point of failure in practice. This is why single-point-of-failure analysis walks the whole dependency chain — power, cooling, network, storage, hypervisor, and management — rather than stopping at the node count. When a scenario asks how to raise availability for a service that already has redundant PSUs and RAID, the answer is a second node; when it asks about a two-node cluster that failed completely, look for the shared dependency both nodes inherited.