1.3 Management HA Failover & Troubleshoot (Manual Failover, Sync Status, Collision Resolution)

Management HA Failover Dynamics & Resiliency Scope

Management High Availability (MHA) in Check Point R82 is engineered specifically to ensure control plane availability. It guarantees that security administrators can maintain visibility, modify security policies, and deploy configuration updates even if a primary management appliance experiences a hardware failure, site outage, or maintenance downtime.

Management Plane Redundancy vs. Data Plane Inspection

A foundational concept tested on the CCSE exam is the strict separation between the management control plane (SMS) and the gateway data plane (Security Gateways):

• Gateway Data Plane Continuity: Managed Security Gateways execute compiled policy binaries stored locally in $FWDIR/state/local/fwviews/. If the Primary SMS fails or becomes completely isolated from the network, managed Security Gateways continue inspecting network traffic without disruption. Active firewall connections, VPN tunnels, NAT translations, and IPS inspection engines operate normally.
• Logging Continuity: Gateways automatically fail over security log delivery to the Secondary Log Server/SMS over TCP port 257 if the Primary Log Server fails to acknowledge TCP log ACKs.
• Administrative Failover: To regain policy editing, rule publishing, and gateway policy push capabilities, an administrator must change the Standby SMS to Active status in SmartConsole.

+---------------------------------------------------------------------------------+
|                      FAILOVER & RESILIENCY ARCHITECTURE                         |
|                                                                                 |
|  Primary SMS (Active) [ CRASHED / ISOLATED ]                                    |
|          |                                                                      |
|          X (Management Control Lost)                                            |
|          v                                                                      |
|  Standby SMS (Secondary)  ===>  [ CHANGE TO ACTIVE ]  ===> Active CPM Engine    |
|                                 (SmartConsole HA view)      (Read-Write DB)     |
|                                                                                 |
|  -----------------------------------------------------------------------------  |
|  Managed Security Gateways:                                                     |
|   - Data Plane Traffic: UNINTERRUPTED (Policy runs from local compiled binaries) |
|   - Log Redirection: Transmit logs to Secondary Log Server (TCP 257)            |
+---------------------------------------------------------------------------------+

---

Active/Standby Failover Workflows (SmartConsole)

In Check Point R82, changing a Standby SMS to Active is intentionally a manual administrative action (unless automated by an external orchestrator). This design prevents split-brain scenarios caused by transient network flaps.

Active/Standby vs Primary/Secondary terminology

CCSE questions focus first on Active/Standby continuity of management operations. Promoting Secondary to Primary is a heavier disaster-recovery procedure used when the original Primary is permanently retired.

Workflow 1: Change Standby to Active in SmartConsole

1. Connect SmartConsole to the Standby SMS: Open SmartConsole R82, enter the Standby SMS IP, and authenticate.
2. Open Management High Availability: SmartConsole Menu -> Management High Availability.
3. Change local server to Active: Select the local Standby SMS and change its role to Active (wording may appear as Change to Active / Set Active).
4. Confirm the warning: Ensure the former Active peer is offline or unreachable before continuing, to avoid Collision.
5. Verify: cpstat mgmt -f ha should report local HA Mode as Active and cpprod_util CPPROD_is_mgmt_active should return 1.

+---------------------------------------------------------------------------------+
| SMARTCONSOLE: CHANGE STANDBY TO ACTIVE                                          |
|                                                                                 |
|  Peer Object        Role        Sync Status      Action                         |
|  -----------------------------------------------------------------------------  |
|  SMS-Primary-01     Active      Unreachable      [ Server Down ]                |
|  SMS-Secondary-01   Standby     Sync In Doubt    [ Change to Active ]           |
|                                                                                 |
|  [ WARNING: Making this server Active grants read-write management access.    ] |
|  [ Ensure the peer SMS is offline to avoid Collision. Continue?               ] |
|                                 [ Yes ]  [ Cancel ]                             |
+---------------------------------------------------------------------------------+

Workflow 2: Promote Secondary to Primary (disaster recovery only)

If the original Primary SMS is permanently lost and you must make the surviving Secondary the new Primary (ICA owner), Check Point documents a separate promote procedure on the surviving server:

# On the Secondary that will become Primary (after it is Active)
#$FWDIR/bin/promote_util
#cpstop
# Remove $FWDIR/conf/mgha files that store prior Secondary settings
# Confirm mgmtha licenses use the promoted server IP
#cpstart

After promotion, SmartConsole object cleanup and database install steps are required so gateways and peers reference the surviving management server. Do not confuse this Primary promotion sequence with everyday Active/Standby failover.

---

Resolving Collision Sync Status State

A Collision status occurs when independent changes are published on both Primary and Secondary management servers while database synchronization was broken (e.g., during a network isolation event where an administrator promoted the Secondary SMS, published changes, and then restored connectivity to the original Primary SMS).

+---------------------------------------------------------------------------------+
| SPLIT-BRAIN COLLISION SCENARIO                                                  |
|                                                                                 |
|  Primary SMS (Disconnected)   ---> Admin published Rev 105 ("Objects A, B")    |
|  Secondary SMS (Promoted)     ---> Admin published Rev 106 ("Objects C, D")    |
|                                                                                 |
|  Network Restored ===========> BOTH SERVERS DETECT CONFLICTING REVISIONS         |
|                                SMARTCONSOLE HA STATUS: [ COLLISION (RED) ]      |
+---------------------------------------------------------------------------------+

Step-by-Step Collision Resolution Procedure

1. Identify the Authoritative Server: Consult administrative change tickets to determine which management server holds the valid, authoritative configuration rulebase.
2. Connect SmartConsole to Authoritative Server: Open SmartConsole connected directly to the designated authoritative SMS (e.g., Primary SMS).
3. Open Management HA Panel: Navigate to SmartConsole Menu -> Management High Availability.
4. Execute Overwrite Peer:
   • Right-click the peer SMS object (the non-authoritative server holding stale/conflicting changes).
   • Select Overwrite Peer (or Force Full Sync to Peer).
5. Confirm Overwrite: SmartConsole warns that all unpublished and published changes on the target peer server will be permanently destroyed and replaced.
6. Full Sync Execution: The authoritative SMS initiates a mandatory Full Sync over TCP port 18190, completely overwriting the peer's database. The status transitions to Synchronized (Green).

---

Diagnostic & Troubleshooting CLI Toolkit

When troubleshooting HA synchronization errors, SIC trust failures, or promotion issues, CCSE administrators rely on a suite of native Check Point CLI tools.

Primary CLI Inspection Commands

1. cpstat mgmt -f ha

Displays comprehensive Management HA operational details, including local server role, peer state, synchronization mode, and last sync timestamp:

[Expert@SMS-Primary:0]# cpstat mgmt -f ha

Management High Availability Status:
------------------------------------
HA Mode:            Active
HA State:           OK
Peer Name:          SMS-Secondary-01
Peer Mode:          Standby
Peer State:         OK
Sync Status:        Synchronized
Last Sync Time:     Wed Jul 22 14:35:10 2026
Sync Port:          18190

2. fwm srv status

Inspects the running status of core management daemons (fwm, cpm, cpca):

[Expert@SMS-Primary:0]# fwm srv status

Process Name    PID       State
----------------------------------
fwm             4120      Ready
cpm             4512      Ready (Active Mode)
cpca            3890      Ready

3. cpprod_util CPPROD_is_mgmt_active

Returns a quick binary boolean output indicating whether the local server is operating in Active mode (1) or Standby mode (0):

[Expert@SMS-Primary:0]# cpprod_util CPPROD_is_mgmt_active
1

4. Network Socket Verification (netstat / tcpdump)

Verify that the CPM sync listening socket is active on TCP port 18190:

[Expert@SMS-Primary:0]# netstat -anp | grep 18190
tcp   0   0 0.0.0.0:18190   0.0.0.0:*   LISTEN   4512/cpm

# Monitor active sync traffic between Primary and Secondary SMS
[Expert@SMS-Primary:0]# tcpdump -n -i eth0 port 18190

---

Critical Diagnostic Log Files Analysis

When CLI status commands indicate errors or promotion failures occur, administrators must analyze three primary diagnostic log files:

$FWDIR/log/fwm.elg       # Management server daemon log (HA status, promotion, SIC)
$FWDIR/log/cpd.elg       # Check Point daemon log (SIC transport, TCP 18191 events)
$CPMDIR/logs/cpm.elg     # CPM Java process log (PostgreSQL DB sync, revision diffs)

1. $FWDIR/log/fwm.elg

Tracks higher-level management process events, HA state changes, and promotion sequences. Look for errors such as:

• HA: State changed from Standby to Active (Successful promotion).
• HA Error: Failed to acquire database write lock (Another session or process is locking PostgreSQL).
• SIC Error: Peer certificate validation failed (Expired or invalid SIC certificate).

2. $FWDIR/log/cpd.elg

Records low-level daemon communication and SIC transport protocol events over TCP ports 18191 and 18211. Useful for diagnosing:

• Connection timeouts during initial SIC establishment.
• SSL/TLS handshake failures between management daemons.

3. $CPMDIR/logs/cpm.elg

Captures detailed Java exceptions and PostgreSQL database operations managed by the CPM engine. Key indicators include:

• CPM Sync Engine: Starting Delta Sync payload generation for revision <GUID>.
• PostgreSQL Exception: Unique constraint violation during sync import (Precursor to Collision status).

---

Technical Reference: Troubleshooting Commands & Log Files