5.2 Link Monitor & WAN Health Checks

5.2 Link Monitor & WAN Health Checks

Quick Answer: Fireware Link Monitor tests end-to-end WAN path viability to detect upstream carrier failures where the physical Ethernet cable remains linked but Internet transit is dead. Each External interface can probe up to two targets (the local default gateway and an upstream public IP such as 8.8.8.8 or 1.1.1.1) using Ping (ICMP Echo), TCP connection probes (port 80/443), or DNS queries. Link failure is declared after a user-configured number of Consecutive Failures (default 3) exceeding the Loss Timeout over the Probe Interval. When a primary link recovers, the Failback for Active Connections setting on the Multi-WAN Advanced tab offers exactly two choices: Immediate Failback (stop all existing connections immediately) or Gradual Failback (keep using the failover interface for existing connections until each one completes). Manual failback is an SD-WAN feature, not a Multi-WAN one.

---

The Limitations of Physical Link-Carrier Detection

Every physical network interface on a Firebox monitors its physical hardware carrier signal (Layer 1 and Layer 2 link-state). If an Ethernet cable is disconnected, or if an on-premises cable modem or optical network terminal (ONT) loses electrical power, the Firebox detects an immediate physical link-down state (NIC Link Down). In response, Fireware instantly removes routes associated with that interface from the active forwarding table and initiates failover.

However, in real-world enterprise environments, the vast majority of WAN outages do not involve an unplugged cable at the firewall. Instead, networks experience "gray failures":

• A construction crew cuts a fiber backhaul conduit three miles away from the customer premises.
• The local ISP modem remains powered on and maintains a solid physical Ethernet link with the Firebox, but cannot pass traffic upstream.
• An upstream core router at the ISP experiences a Border Gateway Protocol (BGP) routing loop or table corruption, silently black-holing traffic.
• The ISP's recursive DNS infrastructure fails, preventing clients from resolving domain names even though IP transport remains intact.

Under physical carrier monitoring alone, the Firebox perceives the interface as healthy and continues directing outbound traffic into the black hole. To overcome this fundamental limitation, Fireware incorporates Link Monitor, an active synthetic probing engine that continuously validates end-to-end Layer 3 and Layer 4 path reachability.

---

Link Monitor Target Architecture & Dual-Target Strategy

Link Monitor operates independently on each configured External interface (and can also be assigned to Trusted, Optional, or Custom interfaces for internal path monitoring). In Fireware Web UI (Network > Link Monitor) or Policy Manager (Network > Configuration > Link Monitor tab), administrators define active health check targets.

                  ┌───────────────────────────────┐
                  │       Firebox External        │
                  │       Interface (Eth0)        │
                  └───────────────┬───────────────┘
                                  │
            ┌─────────────────────┴─────────────────────┐
            │ Probe 1: Local Link                       │ Probe 2: Upstream Transit
            ▼                                           ▼
┌──────────────────────┐                    ┌──────────────────────┐
│ ISP Default Gateway  │                    │ Public DNS / Host    │
│ (e.g., 203.0.113.1)  │                    │ (e.g., 8.8.8.8)      │
└──────────────────────┘                    └──────────────────────┘
[Validates First Hop]                       [Validates Full Internet]

The Dual-Target Recommendation

WatchGuard strongly recommends configuring two probe targets for every External interface:

1. Target 1: The ISP Default Gateway: Probing the default gateway IP address assigned to the External interface verifies Layer 2 and Layer 3 health between the Firebox and the immediate service provider edge equipment. If the gateway fails to respond, the local access loop is indisputably offline.
2. Target 2: An Upstream Public Host: Probing a reliable, highly available public destination—such as Google Public DNS (8.8.8.8), Cloudflare DNS (1.1.1.1), Quad9 (9.9.9.9), or an enterprise headquarters data center IP—proves that traffic can traverse the provider's entire core network and successfully reach the public Internet.

Multi-Target Failure Logic

When multiple probe targets are configured on a single interface, Fireware evaluates link status using deterministic logic:

• If all configured targets on an interface fail to respond, the interface is declared Dead (Down), and Multi-WAN failover is initiated.
• If one target fails but another succeeds, Fireware treats the link as Healthy (Up). This prevents a false-positive failover if an individual third-party public DNS server undergoes maintenance or drops ICMP packets, while the ISP connection itself remains fully operational.

---

Link Monitor Probe Mechanisms

Fireware provides three distinct probing mechanisms to validate link viability across different network layers:

1. Ping (ICMP Echo Request)

• Mechanics: The Firebox sends an ICMP Echo Request (type 8) to the target IP address and listens for an ICMP Echo Reply (type 0).
• Advantages: Lightweight, universally supported by network devices, and generates minimal bandwidth overhead.
• Disadvantages: Many public Internet routers and enterprise firewalls deprioritize, rate-limit, or completely drop ICMP traffic during periods of high congestion (ICMP dampening), which can induce false link failure declarations.

2. TCP Connection Probe

• Mechanics: The Firebox transmits a TCP SYN packet to a specific destination IP address and port (typically TCP port 80 for HTTP or TCP port 443 for HTTPS) and waits for a response.
• Pass Condition: If the target responds with a TCP SYN-ACK (connection accepted) or a TCP RST (port reset), the probe is marked successful. Receiving a RST packet proves that the remote host is reachable and that the intermediate Layer 3 routing path is healthy, even if the target web server process is not accepting new connections.
• Advantages: Bypasses ICMP rate-limiting filters on transit carrier backbones; accurately tests real-world TCP transport across the ISP.

3. DNS Query Probe

• Mechanics: The Firebox dispatches a standard RFC 1035 DNS query for a specific Fully Qualified Domain Name (FQDN, such as watchguard.com) to a designated recursive DNS server IP address.
• Pass Condition: The probe passes only if the DNS server returns a valid DNS response (either an A/AAAA record or NXDOMAIN). If the query times out or returns a server failure (SERVFAIL), the probe fails.
• Advantages: Tests not merely IP routing reachability, but the functional ability of downstream users to resolve Internet domain names. Essential for catching ISP DNS infrastructure outages.

---

Failover Timing Parameters & Threshold Calculation

Link Monitor relies on three tightly coupled configuration parameters to govern failure detection latency and prevent premature or erratic link state changes:

   Probe 1 (Fail)         Probe 2 (Fail)         Probe 3 (Fail)
───────┬──────────────────────┬──────────────────────┬─────────────►
       │                      │                      │
       │◄── Probe Interval ──►│◄── Probe Interval ──►│
       │    (e.g., 5 sec)     │    (e.g., 5 sec)     │
       ▼                      ▼                      ▼
 [Loss Timeout: 3s]     [Loss Timeout: 3s]     [Loss Timeout: 3s]
                                                     │
                                                     ▼
                                              LINK DECLARED DEAD
                                              (Failover Triggered)

1. Probe Interval (Seconds): The frequency at which Link Monitor dispatches synthetic probe packets across the interface. The configurable range is 1 to 600 seconds (default is 5 seconds).
2. Loss Timeout (Seconds): The duration the Firebox waits for a response packet before logging the probe as a drop. The configurable range is 1 to 600 seconds (default is 3 seconds). The Loss Timeout must always be less than or equal to the Probe Interval.
3. Consecutive Failures: The exact number of sequential, unacknowledged probes required before Fireware formally transitions the interface state from Healthy to Dead. The configurable range is 2 to 10 attempts (default is 3 consecutive failures).

Calculating Failure Detection Latency

The total elapsed time required for Fireware to declare a link dead and trigger Multi-WAN failover can be calculated mathematically:

$\text{Total Failover Latency} = (\text{Consecutive Failures} - 1) \times \text{Probe Interval} + \text{Loss Timeout}$

Under default Fireware settings:

• Probe Interval = 5 seconds
• Loss Timeout = 3 seconds
• Consecutive Failures = 3 $\text{Total Failover Latency} = (3 - 1) \times 5\text{ s} + 3\text{ s} = 13\text{ seconds}$

If a catastrophic fiber cut occurs, the Firebox will detect the failure and transition outbound traffic to the secondary WAN in exactly 13 seconds.

Anti-Flapping Logic (Recovery Threshold)

To prevent route flapping—where an unstable WAN connection repeatedly cycles between up and down states, causing routing instability and dropped connections—Fireware enforces a recovery threshold. An interface that has been marked Dead will not be restored to active service until it achieves a consecutive sequence of successful probe responses (by default, matching the consecutive failure count). This guarantees circuit stability before traffic is routed back.

---

Fallback Behaviors & Connection State Management

When a previously failed primary External interface completes its recovery threshold and transitions back to the Healthy state, the Firebox must execute a fallback procedure. Because the primary and backup interfaces possess different public IP addresses, switching traffic back to the primary link has profound implications for active TCP sessions and NAT state tables.

Fireware provides three distinct fallback modes configured under the Multi-WAN settings:

               ┌─────────────────────────────────────┐
               │    Primary Link Recovers Health     │
               └──────────────────┬──────────────────┘
                                  │
        Network > Multi-WAN > Advanced tab:
        "Failback for Active Connections"
                                  │
       ┌──────────────────────────┴──────────────────────────┐
       │                                                     │
┌──────┴───────┐                                      ┌──────┴───────┐
│ Immediate    │                                      │ Gradual      │
│ Failback     │                                      │ Failback     │
└──────┬───────┘                                      └──────┬───────┘
       │                                                     │
       ▼                                                     ▼
Stops all existing connections                Existing connections keep using
immediately; everything moves                 the failover interface until each
back to the primary interface.                one completes; new flows use primary.

1. Immediate Failback

• Operational Mechanics: The instant the primary link is declared healthy, Fireware immediately reroutes all outbound traffic back to the primary interface.
• Connection Teardown Impact: Outbound TCP sessions that were actively communicating across the backup link undergo abrupt state invalidation. Because their egress NAT IP address suddenly shifts from the backup public IP to the primary public IP, remote Internet servers drop the out-of-sequence packets, causing active file downloads, VPN tunnels, and VoIP calls to drop immediately. Clients must re-establish connections from scratch.
• Use Case: Environments where the backup connection is severely constrained in bandwidth or carries exorbitant metered per-megabyte costs (e.g., satellite or emergency cellular), making immediate evacuation of the backup circuit mandatory regardless of session disruption.

2. Gradual Failback (Recommended)

• Operational Mechanics: When the primary link recovers, Fireware dynamically splits traffic handling:
  • New Outbound Connections: All newly initiated TCP handshakes and UDP flows are immediately assigned to the restored primary interface.
  • Existing Active Connections: Sessions that were established across the backup interface during the outage are permitted to remain on the backup interface until they naturally terminate or time out.
• Connection Teardown Impact: Zero abrupt session drops. Users engaged in active VoIP calls, video conferences, or large software updates continue uninterrupted on the backup circuit until completion, while all new web browsing and application requests smoothly populate the restored primary link.
• Use Case: Standard enterprise environments where user productivity and application session continuity take precedence over instantaneous circuit evacuation.

[!IMPORTANT] Multi-WAN failback has only these two options. A third behavior — holding traffic on the failover path until an administrator intervenes — exists in Fireware, but it belongs to SD-WAN, not Multi-WAN. An SD-WAN action that uses the Failover method offers No failback, Immediate failback, and Gradual failback, and when you select Gradual failback or No failback you can trigger a manual failback from the SD-WAN Status page in the Web UI or from Firebox System Manager. Section 5.3 covers SD-WAN in detail. Multi-WAN failover events are always logged, and you can attach a notification method to them.

Choosing Between the Two Multi-WAN Options

Immediate Failback suits metered or severely constrained backup circuits, where evacuating the backup link quickly matters more than session continuity. Gradual Failback suits ordinary business networks, where a dropped VoIP call or a reset file transfer is more disruptive than a few extra minutes of backup-circuit usage.

---

Link Monitor Configuration & Verification Reference

The following table outlines the essential configuration attributes and operational trade-offs for Link Monitor: