6.5 Failover, Cluster States, and Troubleshooting with cphaprob
Key Takeaways
- Cluster member states include ACTIVE, STANDBY, READY, DOWN, LOST, and INIT, plus problem-flagged variants ACTIVE(!), ACTIVE(!F), ACTIVE(!P), and ACTIVE(!FP) indicating a problem is present but the member still forwards.
- Failover is triggered by a Critical Device (PNOTE) reporting a problem, by loss of connectivity between members (LOST), or by a manual `clusterXL_admin down` command.
- The key diagnostic commands are cphaprob state, cphaprob -a if, cphaprob list, cphaprob syncstat, cphaprob show_failover, cphaprob if, cphaprob roles, and (in Clish) show cluster state.
- READY state means a member recognizes the cluster but cannot act — usually because components are still loading or because it is a higher version in a Multi-Version Cluster upgrade.
- Recommended manual failover is `clusterXL_admin down` / `set cluster member admin down`; the pnote method is documented but not recommended.
Cluster Member States
The cphaprob state command (or show cluster state in Clish) reports each cluster member's current state. The states are precise and the CCSA R82 exam expects you to recognize them:
| State | Meaning | Forwards? |
|---|---|---|
| ACTIVE | Everything is OK; the member is forwarding traffic. | Yes |
| ACTIVE(!) | A problem was detected, but the member still forwards because no other Active member exists (otherwise it goes DOWN). | Yes |
| ACTIVE(!F) | ACTIVE(!) plus freeze state (state sync frozen for troubleshooting). | Yes |
| ACTIVE(!P) | ACTIVE(!) plus this is the Pivot member in Load Sharing Unicast mode. | Yes |
| ACTIVE(!FP) | Pivot member in Unicast mode plus freeze state. | Yes |
| STANDBY | HA mode — waiting for the Active member to fail. | No |
| READY | Member recognizes the cluster but cannot act — components still loading, or a higher version in a Multi-Version Cluster upgrade. | No |
| INIT | Post-boot phase before Full Sync completes. | No |
| DOWN | A Critical Device (PNOTE) reports a problem. | No |
| LOST | The peer member cannot reach this member (e.g., rebooting or sync link down). | No |
Two states often confused:
- READY is not a problem state. The member is healthy enough to recognize the cluster but cannot take over yet — typically because some software component has not finished loading, or because it is running a higher version in a Multi-Version Cluster (MVC) upgrade scenario and is waiting to be promoted. READY sends a message asking whether it can become Active and waits for replies.
- DOWN is a problem state. At least one Critical Device (PNOTE) reported
problem, so the member is excluded from forwarding.
Failover Triggers
A failover is the transfer of the Active role from one member to another. Failover is triggered by:
- Critical Device (PNOTE) reports a problem — Critical Devices monitor interfaces, the fwd daemon, CoreXL, CPSEad, and other essential components. When one fails to report on time or reports a problem, the member's state changes to DOWN and the cluster fails over.
- Connectivity loss — the peer member loses connectivity (the unreachable member shows as LOST), typically during a reboot or a sync-link outage.
- Manual failover — initiated by an administrator.
Manual Failover — Recommended Method
The recommended method is clusterXL_admin down (Expert mode) or set cluster member admin down (Clish). This transitions the member to DOWN cleanly and does not disable Delta Sync, so the new Active member's connection tables stay current. The alternative — using cphaconf set_pnote to manipulate a Critical Device into reporting a problem — is documented but not recommended, because it is more invasive and harder to predict.
The cphaprob Command Family
cphaprob is the primary ClusterXL troubleshooting toolkit. The most important commands:
| Command | What It Shows |
|---|---|
cphaprob state | Member states, assigned load, active PNOTEs, last failover, failover counter. |
cphaprob -a if | Per-interface status including CCP mode and sync interface config. |
cphaprob list | Configured Critical Devices (PNOTEs) and their state. |
cphaprob syncstat | Sync statistics — Full Sync count, Delta Sync errors. |
cphaprob show_failover | Failover statistics. |
cphaprob if / cphaprob roles / cphaprob igmp | Interface status, member roles (Pivot), IGMP membership (Multicast). |
cphaprob state output is dense and worth knowing cold. Key fields include Cluster Mode (HA Primary Up / Active Up, or Load Sharing Multicast / Unicast), ID (member priority in HA, member ID in LS), Unique Address (usually the sync interface IP), Assigned Load (100% for Active, 0% for Standby in HA; 100% for all Active in LS), Active PNOTEs (Critical Devices reporting problem), the last member state change event, the last cluster failover event, and the Cluster failover count — a counter that survives reboot and is synchronized between members, plus the last reset time.
Common ClusterXL Problems
A few recurring problem patterns show up on the exam and in the field:
Members Stuck in INIT or READY
If a member stays in INIT, Full Sync is not completing. Check the sync link: physical connectivity, the sync subnet, switch errors, and CCP mode. cphaprob syncstat shows whether Full Sync is even being attempted. A common cause is a sync interface that is up at Layer 1 but not at Layer 2 (wrong VLAN, port-channel mismatch, switch port disabled).
READY in a healthy cluster usually resolves within seconds. Persistent READY in a Multi-Version Cluster means the higher-version member is waiting to be promoted — this is expected during a controlled upgrade.
Unplanned Failovers and Sync Saturation
If the cluster keeps failing over without an obvious hardware cause, examine the Critical Devices with cphaprob list and watch the failover counter in cphaprob state — a fast-climbing counter indicates instability. A flapping interface, a restarting fwd daemon, or a CoreXL instance dropping offline will all trip failover.
Delta Sync traffic grows with connection churn. A sync link that is too small, shared, or high-latency will fall behind, triggering repeated Full Syncs. Symptoms include connections resetting on failover, members repeatedly cycling through INIT, and high Full Sync counts in cphaprob syncstat. Fix by dedicating an interface to sync, increasing bandwidth, or reducing connection churn.
Policy Install Issues on a Cluster Member
Installing a policy on a cluster object installs it on every member. If one member fails to install (SIC broken, disk full, software mismatch), the policy install reports a partial failure. Verify SIC from SmartConsole to each member and check the per-member install log.
A Troubleshooting Workflow
A reliable workflow when a cluster is misbehaving:
cphaprob state— what is each member's state? Is the cluster mode correct (HA / LS Multicast / LS Unicast)?cphaprob -a if— are the cluster and sync interfaces up? Is the CCP mode what you expect?cphaprob list— are any Critical Devices reporting a problem?cphaprob syncstat— is sync healthy, or is the cluster stuck doing Full Syncs?cphaprob show_failover— how many failovers, and when was the last one?
If the cluster was healthy and recently broke, ask what changed: a policy install, a Gaia upgrade, a switch reconfiguration, or a cable swap. Most cluster outages trace to a recent change rather than a spontaneous hardware failure.
A cluster member is reported as READY by cphaprob state. Which is the most likely cause?
Which cphaprob subcommand lists the configured Critical Devices (PNOTEs) and their current state?
Which cphaprob state field is a counter that survives a member reboot and is synchronized between members?