1.3 High Availability Architecture, Links & Failover Mechanics

1.3 High Availability Architecture, Links & Failover Mechanics

High Availability Modes: Active/Passive vs. Active/Active

High Availability (HA) on Palo Alto Networks firewalls ensures continuous network connectivity and stateful session preservation in the event of hardware, link, or path failure.

Active/Passive (A/P) Mode

Active/Passive is the recommended, deterministic architecture for the vast majority of enterprise deployments:

• Active Firewall: Actively forwards network traffic, manages sessions, inspects Layer 7 content, runs dynamic routing protocols, and terminates IPsec VPN tunnels.
• Passive Firewall: Sits in standby mode, actively synchronizing session state tables and configuration updates across dedicated HA links. It does not forward data plane traffic.
• Capacity Preservation: 100% throughput capacity is maintained during a failover event, eliminating the risk of resource oversubscription after a peer failure.
• Operational Simplicity: Troubleshooting and route telemetry remain predictable and deterministic.

Active/Active (A/A) Mode

Active/Active mode is deployed primarily in asymmetric routing topologies where external routers load-balance traffic across dual uplinks:

• Both Firewalls Process Traffic: Both appliances simultaneously forward data plane packets and enforce security policies.
• Session Setup vs. Session Owner:
  • Session Setup Peer: The firewall that receives the initial TCP SYN packet. It allocates the session ID, performs route and NAT lookups, and notifies the peer over HA2.
  • Session Owner Peer: The firewall responsible for ongoing Layer 7 Content-ID/App-ID inspection, client/server state tracking, and session closure.
• Virtual MAC & Floating IP: Utilizes Virtual MAC (VMAC) and Floating IP addresses assigned to shared interfaces, allowing either peer to take ownership of default gateway addresses during a failover.
• Asymmetric Routing Handling: Requires a dedicated HA3 link to forward non-session-owner packets directly between peers without re-encapsulation.

---

Dedicated High Availability Links & Port Protocols

PAN-OS clusters communicate through dedicated, point-to-point connections. Understanding the transport protocols, port numbers, and failure behaviors of each link is a core exam requirement.

                     +---------------------------------------+
                     |           Primary Firewall            |
                     +---------------------------------------+
                        |                 |               |
     HA1 Control Link   |                 | HA2 Data Link | HA3 Packet Link
    (TCP 28769 / 28260) |                 |  (Ethertype   | (Active/Active
                        |                 |    0x7261)    |   Transit)
                        v                 v               v
                     +---------------------------------------+
                     |          Secondary Firewall           |
                     +---------------------------------------+
                        ^
                        | Heartbeat Backup (ICMP / Port 28770)
                        |
                 [ Out-of-Band MGT Switch ]

Deep-Dive: HA1 Control Link

The HA1 link manages cluster control plane health:

• Hello Packets: Transmitted every 8,000 ms (default) to verify that the HA process on the peer is operational. The configurable range is 8,000 to 60,000 ms.
• Heartbeats: ICMP pings exchanged between peers every 1,000 ms under the Recommended profile on PA-3400, PA-5400, PA-5500, PA-7000, and PA-7500 Series firewalls, and every 2,000 ms on PA-400 Series, PA-800 Series, PA-220, and VM-Series. The configurable range is 1,000 to 60,000 ms.
• Configuration Synchronization: Replicates committed configurations from the active firewall to the passive peer via TCP port 28769 and TCP port 28260 (SSH/SSL).
• Encryption: HA1 communication can be encrypted using AES-256 for clusters spanning unencrypted external layer 2/3 transport.

Deep-Dive: HA2 Data Link

The HA2 link maintains real-time synchronization of stateful connection data:

• Synchronized Data: Active session state tables, dynamic ARP tables, IPsec security associations (SAs), and GlobalProtect client sessions.
• Zero Fragmentation Requirement: HA2 traffic must never fragment. If intermediate switches traverse the HA2 path, Jumbo Frames (MTU 9216) must be enabled end-to-end. Packet fragmentation causes severe packet drops in session sync updates.

---

High Availability Clustering Prerequisites

Before forming an HA pair, both firewall appliances must satisfy strict hardware and software parity rules:

1. Identical Hardware Platform: Both firewalls must be the exact same model (e.g., two PA-3410s). In virtualized environments (VM-Series), both VMs must have the identical license model, CPU core allocation, and RAM capacity.
2. Identical PAN-OS Version: Both devices must run the exact same PAN-OS maintenance release (e.g., PAN-OS 11.1.2-h3). A temporary version mismatch is supported only during an orchestrated maintenance upgrade window.
3. Identical Hardware Modules: Multi-slot chassis firewalls (e.g., PA-5450, PA-7000 Series) must possess matching Network Processing Cards (NPC), Data Processing Cards (DPC), and Management Processing Cards (MPC) in identical slot locations.
4. Matching Subscription Licenses: Both firewalls must hold active, identical subscription licenses (Threat Prevention, Advanced URL Filtering, WildFire, DNS Security). A license discrepancy triggers high-priority system alerts and may disable synchronization.

---

Device Priority, Preemption, and Preemption Delay

PAN-OS determines active leadership using Device Priority and Preemption settings.

1. Device Priority (Inverted Metric)

Device Priority is configured as an integer ranging from 1 to 255:

• Inverted Value Rule: A LOWER numeric value indicates a HIGHER operational priority.
• Example: A firewall configured with Priority 10 will win active leadership over a peer configured with Priority 100.

2. Preemption Mechanics

• Default State: Preemption is disabled by default in PAN-OS.
• Preemption Disabled (Recommended for Stability): If the primary firewall (Priority 10) reboots, the secondary firewall (Priority 100) becomes Active. When the primary firewall recovers, it remains Passive to avoid unnecessary network flapping.
• Preemption Enabled: If preemption is enabled, a recovered firewall with a lower numeric priority will immediately reclaim the Active role from the currently active peer.

3. Preemption Delay (Crucial Best Practice)

When preemption is enabled, configuring a Preemption Delay timer (e.g., 60 to 180 seconds; default 0 seconds) is critical:

• The Problem: Without preemption delay, a recovered firewall seizes the Active role the instant PAN-OS initializes. At that moment, dynamic routing protocols (BGP and OSPF) have not yet converged, the Forwarding Information Base (FIB) is unpopulated, and ARP tables are empty. This causes massive traffic black-holing for 30 to 90 seconds!
• The Solution: Preemption Delay holds the recovered firewall in a Passive/Tentative state for the configured timer duration, allowing routing tables, BGP peerings, OSPF LSAs, and hardware FIB tables to fully synchronize before cutting over data traffic.

---

Link Monitoring & Path Monitoring

PAN-OS enables proactive, threshold-based health monitoring that initiates failover before a complete device failure occurs.

Link Monitoring

Monitors the physical carrier state of directly connected interfaces or Link Aggregation Groups (LAG / ae interfaces):

• Interfaces are organized into Link Groups.
• Failure Conditions:
  • Any: Failover triggers if any single interface in the group drops.
  • All: Failover triggers only if every interface in the group drops.
• Failure Weight: Each interface can be assigned a numeric weight. If the cumulative weight of failed links exceeds the configured Group Failure Threshold, the firewall enters a Non-Functional state, triggering immediate failover.

Path Monitoring

Detects silent upstream failures where local links remain physically UP, but traffic forwarding across the network is broken:

• Organizes remote IP addresses into Path Groups.
• Proactively pings destinations (e.g., upstream default gateway, root DNS servers) via ICMP echo requests through designated source interfaces.
• Health Parameters: Ping interval (default 200 ms), ping count (default 10). A path is declared failed after 10 consecutive missed pings (2 seconds).
• Threshold Actions: Evaluates Any or All logic or accumulated failure weights against the group threshold.

---

Split-Brain Prevention & Heartbeat Backup

A Split-Brain condition is the most catastrophic failure mode in High Availability clustering.

The Split-Brain Disaster Scenario

1. The primary HA1 control link cable is severed or disconnected.
2. If no secondary control mechanism exists, both firewalls stop receiving heartbeat and hello packets.
3. Each firewall concludes that its peer has suffered a fatal hardware crash.
4. The passive firewall transitions to Active, resulting in a Dual-Active State.
5. Both firewalls assert ownership of identical IP addresses and broadcast Gratuitous ARPs (GARP) across the network. Switches experience MAC-table flapping, IP collisions occur, state tables corrupt, and enterprise network traffic completely collapses.

                         [ HA1 Cable SEVERED ]
                                 X
           Firewall-A (Active) <---> Firewall-B (Passive)
                    \                     /
                     v                   v
           [ Both Firewalls Transition to ACTIVE! ]
                                 |
              [ DUAL-ACTIVE SPLIT-BRAIN DISASTER ]
           - IP Address Conflicts on Gateway Interfaces
           - Gratuitous ARP Flapping across Core Switches
           - Dropped TCP Connections & Corrupted Sessions

The Multi-Layer Mitigation Strategy

1. HA1-Backup Link: Always provision an HA1-Backup link routed across physically disparate switches or dedicated cables. Note: HA1 and HA1-Backup must reside on different, non-overlapping IP subnets.
2. Heartbeat Backup over the Management (MGT) Port:
   • Enables secondary heartbeat transmission across the existing out-of-band management network via ICMP/UDP.
   • If the primary HA1 link fails, the firewalls query the Heartbeat Backup mechanism over the MGT interface.
   • Upon confirming the peer is still operational, the firewall suppresses failover and flags an HA1 failure alarm, completely preventing a dual-active split-brain catastrophe.

---

Failover Triggers, State Machine & Administrative Control

An Active/Passive cluster transitions through defined lifecycle states: Initial -> Non-Functional -> Passive -> Active (with temporary operational states Tentative and Suspended).

Automatic Failover Triggers

• Missed heartbeats and hello packets on the HA1 control link leading to a peer failure declaration, after which the passive peer waits the Promotion Hold Time (default 2,000 ms) before taking over.
• Loss of keepalives on HA1-Backup and Heartbeat Backup simultaneously.
• Link Monitoring failure threshold reached.
• Path Monitoring failure threshold reached.
• Critical software daemon crash (e.g., mgmtsrvr, routed, pan_comm, or dataplane process).

Administrative Manual Failover

To perform scheduled hardware or software maintenance, administrators can gracefully suspend the active firewall:

# Gracefully transition the local active firewall into a suspended state
admin@PA-3410> request high-availability state suspend

# Return the suspended firewall back to functional operational status
admin@PA-3410> request high-availability state functional

---

Essential CLI Verification Commands

# Display comprehensive HA cluster state, link statuses, peer priority, and mode
admin@PA-3410> show high-availability state

# Display real-time status of monitored physical interfaces and link groups
admin@PA-3410> show high-availability link-monitoring

# Display status of ICMP ping probes and upstream path monitoring groups
admin@PA-3410> show high-availability path-monitoring

# Inspect historical HA state transitions, timestamps, and failover reasons
admin@PA-3410> show high-availability transitions