14.3 Device Maintenance, Backups & Firmware Upgrades

14.3 Device Maintenance, Backups & Firmware Upgrades

Quick Answer: Firebox maintenance relies on two primary backup methods: Configuration Files (.xml), which store human-readable policies and interface settings editable offline in Policy Manager but exclude private keys and certificates; and Backup Images (.fxi), which create encrypted, monolithic system snapshots containing the OS kernel, certificates, private keys, feature keys, and configuration bound to a specific serial number. Disaster recovery at unstaffed branches can be automated using a FAT32 USB Drive Auto-Restore (auto-restore.fxi). FireCluster environments support non-disruptive rolling firmware upgrades (.sysa.dl) where the Backup Master is upgraded and promoted to master before the original master is upgraded. Expired security subscriptions receive a 30-day grace period before signature updates terminate.

---

Configuration Files (.xml) vs. Backup Images (.fxi)

A foundational responsibility of any WatchGuard certified administrator is maintaining robust disaster recovery assets. Fireware provides two distinct backup formats that serve fundamentally different operational objectives: the Configuration File (.xml) and the Backup Image (.fxi).

+-----------------------------------------------------------------------------------+
|                         FIREBOX BACKUP FORMAT COMPARISON                          |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|  [ CONFIGURATION FILE (.xml) ]                                                    |
|  ├── Export Source: Policy Manager (File > Save > To File) or Web UI.             |
|  ├── Content: Policies, aliases, proxy actions, NAT, VPN tunnels, SD-WAN.        |
|  ├── Omissions: NO Private Keys, NO Internal CA, NO Third-Party SSL Certs.        |
|  ├── Portability: Highly portable; can be imported into different Firebox models. |
|  └── Offline Editing: Can be opened and modified offline in Policy Manager.       |
|                                                                                   |
|  [ BACKUP IMAGE (.fxi) ]                                                          |
|  ├── Export Source: Firebox System Manager (Tools > Backup) or Web UI.            |
|  ├── Content: Full OS kernel, bootloader, file system, feature keys, config,     |
|  │            Internal CA keys, 3rd-party SSL certs, local users, DHCP leases.   |
|  ├── Encryption: Strongly encrypted with AES using the administrator passphrase.  |
|  ├── Portability: Hardware-locked to the specific device serial number.           |
|  └── Offline Editing: Monolithic binary; CANNOT be viewed or edited offline.      |
|                                                                                   |
+-----------------------------------------------------------------------------------+

1. Configuration File (.xml)

The configuration file is an XML-encoded plain-text document exported through WatchGuard Policy Manager (File > Save > To File) or downloaded from the Fireware Web UI (System > Configuration File):

• Contents: Defines all logical firewall parameters, including packet filter policies, application proxy actions, network interfaces, secondary IP addresses, static/dynamic routing configurations, Multi-WAN settings, SD-WAN actions, Branch Office VPN topologies, and authentication server definitions.
• Offline Editing: Because it is an XML document, administrators can open, review, edit, and validate the .xml file offline inside Policy Manager without being connected to a live physical Firebox. Configurations can be drafted in advance and committed to the hardware during scheduled maintenance windows.
• Hardware Migration: An .xml configuration file is highly portable. It can be opened in Policy Manager and saved to a different Firebox model (e.g., migrating from a retired Firebox T70 to a new Firebox M390) using the Policy Manager model conversion utility.
• Critical Security Exclusion: For security reasons, the .xml file does NOT contain sensitive cryptographic assets. It excludes the private keys of the internal Firebox Certificate Authority (CA), third-party TLS/SSL web server certificates, Mobile VPN user private keys, and local user passwords. Consequently, restoring an .xml file to a factory-defaulted Firebox will restore the policies, but all custom certificates and private keys must be manually re-imported.

2. Backup Image (.fxi)

A Backup Image is a complete, monolithic binary system snapshot created through Firebox System Manager (Tools > Backup Image) or the Fireware Web UI (System > Backup Image):

• Contents: Captures the entire operational state of the appliance. It includes the complete Fireware OS kernel, bootloader, active configuration, current feature keys, local user authentication databases, active DHCP lease tables, internal Certificate Authority root certificates and private keys, and imported third-party SSL certificates.
• Encryption & Security: Because the .fxi image contains private cryptographic keys and system secrets, Fireware strongly encrypts the image using AES encryption. The administrator must provide the administrative passphrase (or a custom encryption passphrase) during creation. The image cannot be restored without this passphrase.
• Hardware-Locked Serial Number Binding: A .fxi backup image is cryptographically tied to the specific hardware serial number of the Firebox on which it was created. It cannot be restored to a different Firebox appliance under normal circumstances. If an administrator attempts to restore a .fxi image from Firebox A to Firebox B, the restoration process will abort with a serial number mismatch error.
• RMA Exception: If a Firebox suffers a catastrophic hardware failure and WatchGuard Technical Support issues an identical RMA replacement appliance with a new serial number, WatchGuard Support can process the original .fxi backup image through their internal licensing systems to re-key the image for the replacement serial number, allowing a 100% bare-metal restoration.

---

USB Drive Auto-Restore (Zero-Touch Remote Recovery)

Distributed organizations managing dozens or hundreds of remote branch offices, retail storefronts, and teleworker sites frequently face a critical maintenance dilemma: how to recover a corrupted or bricked firewall at a site where no network connectivity exists and no on-site IT technicians are available.

To solve this challenge, Fireware includes the USB Drive Auto-Restore engine, enabling zero-touch bare-metal disaster recovery using a standard USB flash drive.

Preparation and Execution Workflow

sequenceDiagram
    autonumber
    participant IT as Central IT Engineer
    participant USB as FAT32 USB Drive
    participant Remote as Branch Staff (Non-Technical)
    participant Box as Firebox Appliance
    
    IT->>IT: Export encrypted system backup image (.fxi)
    IT->>USB: Rename image to auto-restore.fxi and copy it to the root directory
    IT->>Remote: Ship prepared USB flash drive to remote site
    Remote->>Box: Insert USB drive into Firebox USB port
    Remote->>Box: Power-cycle / Reboot Firebox hardware
    Note over Box: Bootloader detects USB drive during POST
    Box->>USB: Locates /auto-restore.fxi & validates integrity
    Box->>Box: Decrypts image, flashes OS, certificates & config
    Note over Box: Automatic reboot into full production state
    Box-->>IT: Firebox reconnects to central management & BOVPN tunnels

1. USB Drive Preparation: The administrator formats a standard USB flash drive with the FAT32 file system.
2. Image Placement: The administrator copies a valid, encrypted .fxi backup image of that specific Firebox into the root directory of the USB drive and renames the file to: auto-restore.fxi.
3. Deployment: The drive is mailed to the remote site. A non-technical employee inserts the USB drive into the Firebox USB interface and power-cycles the appliance.
4. Automated Boot Detection: During the Power-On Self-Test (POST) boot sequence, the Firebox bootloader scans the USB bus. If it detects a FAT32 partition containing auto-restore.fxi, it pauses normal booting.
5. Image Verification & Flashing: The bootloader verifies the cryptographic signature of the file, verifies that the image serial number matches the appliance hardware, decrypts the contents using the pre-stored system credentials, writes the operating system kernel and configuration to internal flash memory, and reboots.
6. Production Restoration: The Firebox boots directly into full production, re-establishing its IPsec Branch Office VPNs and reconnecting to WatchGuard Cloud or Dimension without requiring any keyboard, monitor, or console interaction.

Security Controls

Because leaving USB auto-restore enabled indefinitely introduces a potential physical attack vector (where a malicious actor with physical access to the chassis could theoretically overwrite the firewall configuration), administrators can disable or restrict USB auto-restore within Fireware Web UI or Policy Manager under System > USB Drive Settings.

---

Fireware OS Upgrade Workflow & Rolling Cluster Upgrades

Maintaining current firmware protects the Firebox against emerging security vulnerabilities, resolves software bugs, and provides updated security inspection engines. WatchGuard distributes operating system upgrades as cryptographically signed installation packages ending in the extension .sysa.dl.

Pre-Upgrade Health Audit

Prior to initiating any operating system upgrade, administrators must execute a strict pre-flight audit:

1. Storage Capacity Verification: Check available flash disk space using Firebox System Manager (Status Report) or the Web UI (System Status). A system with insufficient storage space must have old crash logs, core dumps, and historical diagnostic bundles cleared prior to uploading the new OS image.
2. Feature Key Expiration Check: Ensure that the device's LiveSecurity / Support contract is active. Fireware OS checks the feature key during boot; an expired support license can prevent the appliance from executing newly released major firmware versions.
3. Disaster Recovery Snapshot: Generate and download a fresh .fxi Backup Image immediately prior to flashing firmware. Never attempt a firmware upgrade without a verified, current backup stored off the appliance.

FireCluster Non-Disruptive Rolling Upgrades

In a standalone Firebox deployment, upgrading the operating system requires rebooting the firewall, resulting in 3 to 5 minutes of network downtime. However, in an Active/Passive FireCluster, Fireware executes a non-disruptive rolling upgrade, updating both appliances sequentially with zero dropped connections.

+-----------------------------------------------------------------------------------+
|                FIRECLUSTER NON-DISRUPTIVE ROLLING UPGRADE SEQUENCE                |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|  [ STEP 1: UPLOAD TO BACKUP MASTER ]                                              |
|  • Admin initiates upgrade on Cluster Master.                                     |
|  • Cluster Master transfers .sysa.dl package to Backup Master across sync link.   |
|  • Backup Master flashes firmware to its alternate boot partition.                |
|                                                                                   |
|  [ STEP 2: BACKUP MASTER REBOOTS WITH NEW OS ]                                    |
|  • Backup Master reboots running the updated Fireware OS version.                 |
|  • Backup Master reconnects to cluster across synchronization interfaces.          |
|  • Cluster Master continues processing 100% of network traffic.                   |
|                                                                                   |
|  [ STEP 3: STATEFUL CLUSTER FAILOVER ]                                            |
|  • Cluster Master initiates a controlled, stateful failover to Backup Master.     |
|  • Backup Master (running NEW OS) becomes CLUSTER MASTER.                         |
|  • All active TCP connections, NAT tables & VPN tunnels remain intact.            |
|                                                                                   |
|  [ STEP 4: ORIGINAL MASTER UPGRADE & REBOOT ]                                     |
|  • Original Master (now acting as Backup) receives the .sysa.dl package.          |
|  • Original Master flashes firmware, reboots with NEW OS, and rejoins cluster.    |
|  • Upgrade complete: Both members on new OS with ZERO session drops!              |
|                                                                                   |
+-----------------------------------------------------------------------------------+

During this rolling procedure, the temporary version mismatch between the two appliances is managed by Fireware's cluster compatibility daemon, ensuring that state table replication continues smoothly across the version transition.

---

Feature Key Management & Subscription Lifecycle

The Feature Key is a cryptographically signed text block issued by WatchGuard that unlocks the capabilities of the Firebox hardware. It governs:

• Maximum licensed user counts (on legacy platforms) and interface capacity limits.
• Maximum concurrent Mobile VPN (SSL, IKEv2, L2TP) tunnels and BOVPN tunnels.
• Enabled security subscriptions: Gateway AntiVirus (GAV), IntelligentAV, Intrusion Prevention Service (IPS), WebBlocker, Application Control, spamBlocker, Reputation Enabled Defense (RED), APT Blocker, DNSWatch, Geolocation, Botnet Detection, and ThreatSync. (Data Loss Prevention reached end of life on 26 February 2025 and was removed from Firebox feature keys, so it no longer appears on a current key.)
• The expiration date for each licensed service and the underlying LiveSecurity/Support contract.

Synchronization Methods

• Automatic Feature Key Synchronization: When the Firebox has direct Internet connectivity, it periodically polls WatchGuard licensing servers over HTTPS. When a contract renewal or service add-on is purchased, the Firebox automatically downloads and applies the updated feature key without administrative intervention.
• Manual Feature Key Import: In air-gapped or restricted environments, the administrator logs into the WatchGuard portal, copies the raw cryptographic text block of the feature key, and pastes it into Policy Manager or the Web UI under System > Feature Key.

Subscription Expiration & The 30-Day Grace Period

When a subscription security service reaches its expiration date, WatchGuard provides a built-in 30-day grace period:

• During the 30-Day Grace Period: The expired service (e.g., Gateway AntiVirus or IPS) remains fully operational. It continues to inspect traffic and, critically, continues to download dynamic signature updates from WatchGuard security clouds.
• After the 30-Day Grace Period Expires: The service immediately ceases downloading signature updates. Furthermore, the behavior of firewall policies enforcing that service shifts based on configured fallback actions:
  • For WebBlocker, administrators can configure the policy to either allow all traffic uninspected or deny all web access if the license or cloud server is unavailable.
  • For Gateway AntiVirus and IPS, the engine stops scanning or fails open/closed depending on the global proxy settings.
  • The Firebox generates high-severity system alarms and event logs warning of uninspected traffic.

---

Maintenance Schedule & Task Matrix

The following table outlines recommended routine administrative maintenance procedures across daily, weekly, monthly, and quarterly operational cadences: