13.3 WatchGuard Cloud Visibility & Diagnostic Tools

13.3 WatchGuard Cloud Visibility & Diagnostic Tools

Quick Answer: WatchGuard Cloud is a multi-tenant SaaS management and monitoring platform that eliminates on-premises logging servers by providing scalable cloud log retention, global threat dashboards, alerting, and cross-platform ThreatSync (XDR) correlation. When troubleshooting complex on-premises network failures, Fireware OS provides essential embedded diagnostic tools directly in the Web UI and Firebox System Manager: TCP Dump captures raw packets on any interface using Berkeley Packet Filter (BPF) syntax and exports .pcap files for Wireshark analysis; Diagnostic Tasks execute Ping (with custom source interfaces and Don't Fragment MTU testing), Traceroute, and DNS lookups; and the Diagnostic Log File (Support Snapshot) bundles sanitized configuration files, kernel dumps, process tables, and connection states for WatchGuard Technical Support analysis.

---

WatchGuard Cloud Visibility & Monitoring Architecture

While on-premises solutions like WatchGuard Dimension provide outstanding local log aggregation, they require organizations to maintain virtual server hardware, manage hypervisor resources, expand SAN storage, and maintain database integrity. WatchGuard Cloud delivers a cloud-native software-as-a-service (SaaS) architecture hosted globally in Amazon Web Services (AWS) that eliminates all on-premises server overhead.

+-----------------------------------------------------------------------------------+
|                         WATCHGUARD CLOUD SAAS ARCHITECTURE                        |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|  [ Distributed Enterprise / MSP Client Sites ]                                    |
|  Firebox A (T45)        Firebox B (M390)        AuthPoint MFA / Endpoint EDR      |
|       │                        │                        │                         |
|       └────────────────────────┼────────────────────────┘                         |
|                                ▼ Outbound Encrypted TLS (TCP Port 443)            |
|  +-----------------------------------------------------------------------------+  |
|  | WATCHGUARD CLOUD MULTI-TENANT FABRIC (AWS Global SaaS Infrastructure)       |  |
|  |                                                                             |  |
|  |  [ Multi-Tenant Account Partitioning ]                                      |  |
|  |  • Tier 1: Service Provider (MSP Global Fleet Dashboard)                    |  |
|  |  • Tier 2: Subscriber Accounts (Individual Client Visibility)               |  |
|  |                                                                             |  |
|  |  [ Cloud Visibility Engine ]              [ ThreatSync (XDR) Engine ]       |  |
|  |  • Real-Time Traffic & Threat Dashboards  • Correlates Network + Endpoints  |  |
|  |  • Executive & Compliance Reporting      • Automated Incident Isolation     |  |
|  |  • Cloud Alerting & Webhooks              • Global Threat Intelligence      |  |
|  |                                                                             |  |
|  |  [ Elastic Cloud Data Lake ]                                                |  |
|  |  • Retention Tiers: 1-Day (Basic) / 30-Day (Total) / Up to 365-Day Add-On   |  |
|  +-----------------------------------------------------------------------------+  |
|                               │                                                   |
|                               ▼ HTTPS Browser Access                              |
|                  Administrators, MSPs, & Auditors                                 |
|                                                                                   |
+-----------------------------------------------------------------------------------+

Cloud Architecture and Multi-Tenancy

WatchGuard Cloud was designed from the ground up to support both enterprise IT departments and Managed Service Providers (MSPs):

• Hierarchical Multi-Tenancy: MSPs can access a multi-tier management console divided into Service Provider (global view across all managed clients) and Subscriber (isolated tenant views dedicated to individual customer organizations). Role-Based Access Control (RBAC) ensures client administrators only see their own telemetry.
• Zero-Touch Outbound Connection: Fireboxes connect outbound to WatchGuard Cloud over standard HTTPS (TCP port 443). Because the connection originates outbound from the Firebox, administrators do not need to open inbound firewall ports or configure static public IP addresses on remote branch firewalls.
• ThreatSync (XDR): ThreatSync unifies event telemetry from network security appliances (Firebox) and endpoint security agents (WatchGuard EPDR). It correlates network threat events (such as an internal machine contacting a known botnet command-and-control server) with endpoint processes (identifying the specific malicious .exe or PowerShell script executing on that machine) and can automatically isolate the infected host from the network.
• Scalable Data Retention Tiers: Unlike on-premises storage arrays that can run out of disk space, WatchGuard Cloud offers elastic storage tiers based on appliance licensing:
  • Basic Security Suite: Includes 1 day of cloud reporting.
  • Total Security Suite: Includes 30 days of comprehensive cloud log retention and historical reporting.
  • Cloud Log Retention Licenses: Optional add-on subscriptions that extend storage up to 365 days to meet long-term compliance mandates.

WatchGuard Cloud vs. WatchGuard Dimension Comparison

---

On-Device Packet Capture: TCP Dump Utility

When standard log records indicate that traffic is being dropped or misrouted, but do not provide low-level protocol visibility, network engineers turn to packet capture. Fireware OS contains an embedded, full-featured TCP Dump utility directly accessible through the Fireware Web UI (under System Status > Diagnostics > Network Statistics > Diagnostic Tasks) and Firebox System Manager (Tools > Diagnostic Tasks).

+-----------------------------------------------------------------------------------+
|                         FIREWARE TCP DUMP PACKET CAPTURE                          |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|  1. Configuration:                                                                |
|     • Interface:  [ eth1 - Trusted        ▼ ]                                     |
|     • Filter:     [ host 10.0.1.50 and port 443                           ]       |
|     • Max Packets:[ 5000                  ▼ ]                                     |
|                                                                                   |
|  2. Execution:                                                                    |
|     [ START CAPTURE ] ──> Kernel Packet Sniffer Captures Raw Frames               |
|                                    │                                              |
|                                    ▼ Circular Memory Buffer                       |
|     [ STOP CAPTURE ]  ──> Freezes Buffer & Closes Capture Handle                  |
|                                    │                                              |
|                                    ▼                                              |
|  3. Export:                                                                       |
|     [ DOWNLOAD PCAP ] ──> Saves file: firebox-capture-eth1.pcap                   |
|                                    │                                              |
|                                    ▼ Opened in Wireshark                          |
|     Forensic Analysis: TCP Handshake, Window Size, TLS SNI, RST Flags             |
|                                                                                   |
+-----------------------------------------------------------------------------------+

Interface Selection and Multi-Zone Capture

The Fireware TCP Dump engine can bind to virtually any physical or logical Layer 3 interface on the appliance:

• Physical Interfaces: External interfaces (eth0), Trusted networks (eth1), Optional DMZ networks (eth2).
• Logical and Virtual Interfaces: 802.1Q VLAN sub-interfaces (e.g., vlan10, vlan20), Link Aggregation Groups (lag1), and Route-Based BOVPN Virtual Interfaces (bvif1).
• Loopback and System: Capture internal kernel forwarding events.

Berkeley Packet Filter (BPF) Syntax

Running an unfiltered packet capture on a gigabit production interface would immediately flood the memory buffer and consume heavy CPU resources. Fireware utilizes standard Berkeley Packet Filter (BPF) syntax, allowing engineers to isolate exact conversational streams:

# Capture all traffic to or from a specific host:
host 192.168.1.100

# Capture specific TCP application traffic between a host and a subnet:
host 10.0.1.25 and (port 80 or port 443)

# Capture DNS queries and responses while ignoring all other traffic:
udp port 53

# Capture traffic across an entire subnet except for administrative SSH:
net 10.0.20.0/24 and not port 22

# Capture ICMP error messages and ping requests:
icmp

Exporting and Analyzing in Wireshark

Once the capture is executed, the Firebox writes the raw frame data into an internal buffer. The administrator clicks Stop Capture and then clicks Download. Fireware generates a standard .pcap (or .cap) packet capture file that can be opened directly in Wireshark or other network protocol analyzers.

This enables engineers to perform advanced protocol forensics:

• Verifying three-way TCP handshakes (SYN, SYN-ACK, ACK) to determine if a remote server is responding.
• Diagnosing unexpected TCP Reset (RST) flags generated by intermediate upstream routers.
• Examining TLS Client Hello Server Name Indication (SNI) headers before decryption.
• Troubleshooting MTU/MSS fragmentation issues and packet retransmissions.

[!CAUTION] Packet Capture Performance Safety: The TCP Dump sniffer operates in promiscuous/inspection mode within the Fireware networking stack. Capturing traffic without filters on high-throughput interfaces can spike CPU utilization and degrade appliance forwarding performance. Always apply restrictive BPF filters and limit capture file sizes when diagnosing production networks.

---

Fireware Network Diagnostic Validation Tasks

Before launching a deep packet capture, network administrators can rapidly validate Layer 3 connectivity, path latency, and name resolution using the built-in Diagnostic Tasks suite in Fireware Web UI and FSM.

+-----------------------------------------------------------------------------------+
|                         FIREWARE DIAGNOSTIC TASKS SUITE                           |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|  [ PING (ICMP Echo) ]                                                             |
|  • Validates Layer 3 reachability to target IP or FQDN                            |
|  • Source Interface Selection: Bind probe to External, Trusted, or BOVPN VIF      |
|  • Don't Fragment (DF) Bit: Probes Path MTU to diagnose packet fragmentation      |
|  • Packet Size & Count: Tests packet loss under varying payload loads             |
|                                                                                   |
|  [ TRACEROUTE (Path Hop Analysis) ]                                               |
|  • Maps hop-by-hop Layer 3 routing path from Firebox to destination               |
|  • Identifies intermediate ISP latency spikes and packet loss                     |
|  • Detects routing loops and verifies Multi-WAN path selection behavior           |
|                                                                                   |
|  [ DNS LOOKUP (Name Resolution Validation) ]                                      |
|  • Queries configured Network DNS servers directly from Firebox kernel            |
|  • Resolves A, AAAA, MX, and PTR records                                          |
|  • Identifies internal DNS server failures vs. external upstream resolution drops |
|                                                                                   |
+-----------------------------------------------------------------------------------+

Ping (ICMP Echo) with Advanced Options

The Fireware Ping utility is significantly more powerful than standard workstation ping utilities because it allows engineers to manipulate packet attributes:

• Source Interface / IP Binding: By default, ping packets originate from the interface closest to the destination according to the routing table. However, engineers can explicitly force the ping to originate from a specific internal Trusted interface, an Optional DMZ interface, or a secondary WAN IP. This is essential when testing whether a remote BOVPN endpoint accepts packets from a specific internal subnet.
• Do Not Fragment (DF) Flag: Setting the DF flag instructs the Firebox to set the Don't Fragment bit in the IP header. By varying packet size (e.g., from 1400 to 1500 bytes), engineers can accurately determine the exact Path Maximum Transmission Unit (Path MTU) across an IPsec VPN tunnel or PPPoE WAN connection, diagnosing packet black-holing caused by MTU mismatch.
• Packet Size and Count: Allows administrators to transmit large buffers (e.g., 8,192 bytes) or high-count streams (e.g., 100 packets) to detect intermittent packet loss or buffer bloat on congested links.

Traceroute (Layer 3 Hop-by-Hop Path Discovery)

The Traceroute utility sends a series of UDP or ICMP probes with incrementing Time-To-Live (TTL) values to map the path taken by packets exiting the Firebox:

• Detecting Routing Asymmetry and Loops: Confirms whether outbound traffic is traversing the expected ISP gateway in Multi-WAN environments.
• Pinpointing WAN Failures: Identifies the precise intermediate router or autonomous system (AS) where traffic is being dropped.

DNS Lookup

The DNS Lookup utility queries the Firebox's configured Network DNS servers to resolve hostnames:

• Verifies whether the Firebox itself can resolve critical FQDNs used in subscription service updates (e.g., updates.watchguard.com).
• Tests DNS resolution speed and identifies misconfigured internal Active Directory DNS forwarders.

---

Diagnostic Log Files & Support Log Bundles (Support Snapshot)

When a network issue defies routine troubleshooting—such as an unexplained kernel panic, an intermittent proxy daemon crash, high CPU utilization without apparent traffic, or complex routing engine corruption—administrators must escalate the incident to WatchGuard Technical Support (TAC).

WatchGuard support engineers cannot diagnose deep operating system issues from basic Traffic Monitor text alone. Instead, they require a comprehensive Diagnostic Log File, commonly referred to as a Support Snapshot or Support Log Bundle.

Generating the Support Snapshot

Administrators can generate and download the support snapshot through two methods:

1. Fireware Web UI: Navigate to System Status > Diagnostics > Diagnostic File, and click Download.
2. Firebox System Manager (FSM): Select the Status Report tab, click Support Log, or select Tools > Diagnostic Log.

+-----------------------------------------------------------------------------------+
|                         SUPPORT SNAPSHOT ARTIFACT BUNDLE                          |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|  support_snapshot_[Serial]_[Timestamp].tgz                                        |
|  │                                                                                |
|  ├── config.xml (Sanitized Configuration File)                                    |
|  │   └── Full firewall policy rules, aliases, and interfaces                      |
|  │       (NOTE: All passphrases, PSKs, and private keys are cryptographically stripped)|
|  │                                                                                |
|  ├── kernel_logs.txt (Kernel dmesg & Crash Dumps)                                 |
|  │   └── Hardware initialization, memory faults, driver crashes, NIC events       |
|  │                                                                                |
|  ├── system_processes.txt (ps, top, and Memory Allocation)                        |
|  │   └── Daemon CPU utilization, memory leaks, slabinfo, thread locks             |
|  │                                                                                |
|  ├── network_state.txt (Interface Registers & Driver Stats)                       |
|  │   └── CRC errors, frame drops, duplex mismatches, hardware buffer overruns    |
|  │                                                                                |
|  ├── routing_rib_fib.txt (Kernel Routing Tables & Dynamic Routing)                |
|  │   └── Complete Layer 3 route table, policy-based routes, OSPF/BGP RIB/FIB      |
|  │                                                                                |
|  ├── conntrack_table.txt (Active Stateful Connection Sessions)                    |
|  │   └── Current NAT state tables, active TCP/UDP sessions, proxy session pools  |
|  │                                                                                |
|  └── vpn_engine_state.txt (IPsec SA & IKE Daemon Status)                          |
|      └── Phase 1 IKE SAs, Phase 2 Child SAs, SPI registers, DPD keepalive state   |
|                                                                                   |
+-----------------------------------------------------------------------------------+

What the Support Snapshot Contains

The downloaded archive contains a complete, time-stamped diagnostic snapshot of the Firebox operating system environment:

• Sanitized System Configuration (config.xml): Contains the full active configuration file, including all policies, proxy actions, NAT rules, and network interface settings. Crucially, Fireware automatically sanitizes sensitive credentials—administrative passphrases, Pre-Shared Keys, and private keys are removed or hashed, ensuring security credentials are not exposed to third parties.
• Kernel Logs & Hardware Information (dmesg): Records low-level Linux kernel events, driver initialization sequences, hardware sensor readings (temperature, fan speeds, power supplies), and kernel panic backtraces.
• Process & Memory State Tables: Contains snapshots of active Linux processes, memory thread allocations, CPU utilization per core, and memory slab allocations, allowing support engineers to identify memory leaks or hung proxy processes.
• Hardware Interface Statistics: Provides low-level Ethernet controller registers, detailing CRC errors, frame alignment errors, collisions, and ring buffer drops that indicate physical cable or switch port failures.
• Routing & State Tables: Dumps the full kernel Forwarding Information Base (FIB), dynamic routing database, active connection tracking (conntrack) tables, and NAT translation mappings.
• VPN Diagnostic State: Records active Phase 1 and Phase 2 Security Parameter Indices (SPIs), sequence counters, and negotiation state machines.