10.3 High Availability Troubleshooting, System Maintenance & Log Forwarding

10.3 High Availability Troubleshooting, System Maintenance & Log Forwarding

High Availability Operational Troubleshooting

Palo Alto Networks High Availability (HA) ensures continuous enterprise network connectivity by pairing two firewalls in either an Active/Passive or Active/Active topology. Operational stability depends on the uninterrupted synchronization of control state, configuration hierarchies, and real-time session tables across dedicated physical clustering links.

High Availability Link Infrastructure

• HA1 (Control Link): Exchanges heartbeats, hello packets, and configuration synchronization. Operates over TCP port 28769 and TCP port 28260 (for cleartext/SSL-secured communications). HA1 maintains cluster state and coordinates failover elections.
• HA1-Backup: Provides an alternate, redundant path for HA1 control traffic. Typically mapped to the dedicated out-of-band Management (MGT) interface or an auxiliary physical port to prevent single-point-of-failure disconnects.
• HA2 (Data Link): Synchronizes real-time stateful connection tables, including active session IDs, TCP sequence numbers, NAT translations, IPSec security associations, and ARP tables. Operates using UDP port 28770 or raw Ethernet (Ethertype 0x7261).
• HA2-Backup: Redundant link for session table replication.
• HA3 (Packet Forwarding Link): Exclusively utilized in Active/Active deployments. Uses a dedicated raw Layer 2 connection (Ethertype 0x7262) to forward asymmetric transit traffic between active peers without re-evaluating slow-path policy.

+---------------------------+                         +---------------------------+
|    Firewall-A (Peer 1)    |                         |    Firewall-B (Peer 2)    |
|                           |=== HA1 (Control Link) ==|                           |
|  - State: Active          |--- HA1-Backup (MGT) ----|  - State: Passive         |
|  - Priority: 100 (Wins)   |=== HA2 (Session Sync) ==|  - Priority: 200          |
|  - Preemption: Enabled    |--- HA2-Backup (Data) ---|  - Preemption: Enabled    |
+---------------------------+                         +---------------------------+
              |                                                     |
              +----------------== Data Paths ==---------------------+ 
                               (Transit Traffic)

Essential HA Diagnostic Commands

# Display overall HA operational state, peer status, and link health
admin@PA-3410> show high-availability state

# Display exhaustive HA configuration, timer values, and failover metrics
admin@PA-3410> show high-availability all

# Verify active session synchronization between cluster peers
admin@PA-3410> show high-availability state-synchronization

# Inspect HA link interface statistics and packet error counters
admin@PA-3410> show high-availability link-monitoring

Testing Controlled Failover

Administrators validate cluster resilience using non-disruptive, software-triggered failover commands:

# Suspend the local firewall, forcing all active traffic to failover to the peer
admin@PA-3410> request high-availability state suspend

# Verify the peer transitions to Active and assumes forwarding duties
admin@PA-3410> show high-availability state

# Return the suspended firewall back to functional status
admin@PA-3410> request high-availability state functional

Split-Brain Failure Scenarios & Recovery Procedures

A split-brain condition occurs when both the primary HA1 link and secondary HA1-backup link fail simultaneously while the data plane interfaces remain physically UP. Because neither peer receives heartbeat keepalives, both firewalls assume the partner has suffered a fatal hardware failure. Consequently, both appliances transition to the Active state.

Operational Impact of Split-Brain

1. Both firewalls broadcast identical gratuitous ARPs, claiming ownership of identical virtual IP and virtual MAC addresses.
2. Upstream and downstream switches experience severe MAC address table flapping, causing asymmetric traffic distribution.
3. Packets belonging to the same TCP session arrive on different firewalls; because session state is un-synchronized, both firewalls drop packets due to out-of-order sequence violations.

Step-by-Step Split-Brain Recovery Procedure

1. Isolate Secondary Data Paths Immediately: Connect to the upstream and downstream access switches and administratively shut down the physical switch ports connecting to the secondary firewall. This stops duplicate MAC advertisements and restores single-path routing.
2. Out-of-Band Administrative Access: Establish a dedicated serial console or isolated out-of-band management connection to the secondary firewall.
3. Suspend Secondary HA: Execute request high-availability state suspend on the secondary unit to force its interfaces into a suspended, non-forwarding state.
4. Restore HA1 Control Links: Diagnose and repair the physical cabling, SFP transceivers, intermediate VLANs, or MTU mismatches on the HA1 and HA1-backup links.
5. Verify Peer Communication: Run show high-availability state on both peers to confirm heartbeat communication and configuration synchronization have recovered.
6. Restore Functional Mode: Re-enable the switch ports and execute request high-availability state functional on the secondary unit.

---

PAN-OS Software Upgrade Procedures

Upgrading production PAN-OS clusters requires strict adherence to dependency rules and structured failover sequences to ensure zero downtime.

Core Upgrade Prerequisites: The Base Image Requirement

PAN-OS software releases follow a hierarchical structure: Major/Minor feature releases (e.g., 11.0.0, 11.1.0) and Maintenance releases (e.g., 11.0.4, 11.1.2-h3).

Exam Trap Alert: Upgrading across feature release versions (such as moving from 10.2.x to 11.0.x or from 11.0.x to 11.1.x) strictly requires downloading the base .0 image (e.g., 11.0.0 or 11.1.0) to the firewall's local disk before the target maintenance release can be installed. The base .0 image does not need to be installed or rebooted, but its file system package must reside locally on disk. Attempting to install a maintenance release without the base image present will fail with an explicit dependency error!

Zero-Downtime HA Upgrade Sequence (Active/Passive)

Step 1: Verify HA Sync & Disable Preemption temporarily on both peers
Step 2: Download Base (.0) and Target Maintenance Images on both peers
Step 3: Install Target Image on PASSIVE Peer & Reboot Passive Peer
Step 4: Verify Passive Peer Health (Operates in Version Mismatch Mode)
Step 5: Suspend Active Peer (`request high-availability state suspend`)
        -> Upgraded Peer becomes ACTIVE (Carries Production Traffic)
Step 6: Verify Production Health on Newly Active Peer
Step 7: Install Target Image on Suspended Peer & Reboot
Step 8: Return Peer to Functional (`request high-availability state functional`)
Step 9: Re-enable Preemption & Verify Synchronized HA Cluster State

1. Pre-Upgrade Validation: Verify configuration sync (show high-availability state). Temporarily disable preemption on both firewalls to prevent unintended failbacks during intermediate reboots.
2. Image Staging: Download the required base image (e.g., PAN-OS 11.1.0) and the target maintenance release (e.g., PAN-OS 11.1.2-h3) onto both Firewall-A and Firewall-B.
3. Upgrade the Passive Peer First: Install the target maintenance release on the Passive firewall and initiate a reboot:

   admin@PA-Passive> request system software install version 11.1.2-h3
   admin@PA-Passive> request restart system
   

4. Verify Health of Upgraded Passive Peer: After the firewall boots, confirm operational stability (show system info). The HA cluster will display a status of Version Mismatch, which is fully supported temporarily during upgrade procedures to maintain session synchronization.
5. Execute Controlled Failover: Suspend the currently Active firewall to shift all production traffic to the newly upgraded firewall:

   admin@PA-Active> request high-availability state suspend
   

6. Validate Production Traffic: Confirm that the newly active firewall processes user sessions, GlobalProtect connections, and App-ID inspection normally.
7. Upgrade the Remaining Peer: Install the target maintenance release on the suspended firewall and reboot it:

   admin@PA-Suspended> request system software install version 11.1.2-h3
   admin@PA-Suspended> request restart system
   

8. Restore Functional Cluster State: After reboot, restore the unit: request high-availability state functional. Re-enable preemption if dictated by network policy, and confirm cluster synchronization.

---

Dynamic Content Updates Lifecycle & Threshold Hold Times

Palo Alto Networks delivers regular security updates via dynamic content packages:

Threshold Hold Times for Operational Stability

While security best practices dictate rapid signature deployment, unvalidated application definitions in mission-critical networks can introduce false positives or alter security policy matching. PAN-OS provides Threshold Hold Times:

• Allows administrators to delay the automatic installation of downloaded content by a configurable window (e.g., 24 to 48 hours).
• Provides an observation buffer during which the global threat research community and peer enterprises validate signature stability.

Managing New App-IDs: The 'Disable New Apps' Safeguard

When Palo Alto Networks introduces new App-IDs in an Applications and Threats update, traffic that previously matched generic protocols (such as web-browsing or ssl) may now match a newly introduced specific App-ID (such as an obscure SaaS tool).

• If the security policy only permits web-browsing, traffic transitioning to the new App-ID will be dropped by the inter-zone-default rule!
• Protection Setting: In Device > Dynamic Updates, enable Disable new apps in content update.
• Newly introduced App-IDs are installed in a disabled state, preserving existing policy matches. Administrators review the Review New Apps report, modify security rules to accommodate approved applications, and manually enable the new App-IDs.

---

Enterprise Log Forwarding & Strata Logging Service

PAN-OS firewalls generate rich telemetry across multiple log types: Traffic, Threat, URL Filtering, WildFire Submissions, Data Filtering, Authentication, Tunnel Inspection, and System logs.

Log Forwarding Profiles

Security policies do not forward logs directly. Instead, an administrator attaches a Log Forwarding Profile to each security rule, dictating where matching event logs are streamed in real time.

[ Security Policy Rule Match ]
              |
              v
[ Log Forwarding Profile ]
              |
              +---> [ Syslog Servers (SIEM) ] (RFC 5424 over TLS / TCP)
              +---> [ SNMP Trap Receivers ]   (SNMP v2c / v3)
              +---> [ HTTP Webhook Receivers] (JSON POST to ServiceNow, Slack, SOAR)
              +---> [ Cortex Data Lake / Strata Logging Service ] (Cloud-Native Fabric)

External Forwarding Destinations

• Syslog: Transmits standard RFC 5424 (TLS/TCP) or RFC 3164 (UDP) formatted records to external SIEM platforms (such as Splunk, QRadar, or Microsoft Sentinel). Supports custom format strings.
• SNMP Traps: Sends alerts for system alarms, environmental thresholds, and high-severity threat detections using SNMP v2c or SNMP v3 with AES/SHA encryption.
• HTTP Webhooks: Formats log data into structured JSON payloads and executes HTTP POST requests targeting external REST APIs (such as triggering automated incident tickets in ServiceNow or alerts in PagerDuty).
• Cortex Data Lake (CDL) / Strata Logging Service:
  • Cloud-native logging infrastructure providing petabyte-scale elastic storage, sub-second query execution, and centralized visibility.
  • Feeds telemetry into Cortex XDR, Strata Cloud Manager (SCM), and Cloud-Delivered Security Services without consuming local firewall disk I/O.
  • Verification Command: request logging-service-forwarding status verifies mutual TLS certificate authentication, cloud gateway connectivity, and queue backlogs.

---

System Diagnostics & Disaster Recovery

Generating Tech Support Dumps

A Tech Support Dump (tech-support-dump) is an exhaustive diagnostic archive required by Palo Alto Networks Technical Support (TAC) for deep root-cause failure analysis. It captures system logs, core crash dumps, running configurations, routing tables, and hardware register telemetry:

# Generate a tech support bundle on the firewall
admin@PA-3410> request tech-support-dump

# Export the generated archive to a remote SCP server
admin@PA-3410> scp export tech-support to user@10.10.10.50:/var/support/PA-3410-TS.tgz

Maintenance Mode for Disaster Recovery

If an unrecoverable system corruption occurs (e.g., sudden power loss during a flash write, lost superuser credentials, or corrupted root partitions), the firewall cannot boot into PAN-OS. Administrators access Maintenance Mode via the physical serial console port (9600 baud, 8 data bits, no parity, 1 stop bit / 8-N-1):

1. Reboot the firewall while observing serial console output.
2. When prompted during the initial bootloader countdown (GRUB/U-Boot), type maint and press Enter.
3. The system boots into an interactive, menu-driven recovery environment:
   • Factory Reset: Wipes local storage media and restores the factory-default PAN-OS image.
   • Image Rollback: Switches the active boot partition back to the previously operational PAN-OS version.
   • File System Check (FSCK): Scans and repairs corrupted disk partition structures.
   • Install Image / Maintenance via TFTP: Installs a fresh PAN-OS software image across the management network.