10.1 Network Watcher Diagnostic Tools

Network Watcher Diagnostic Tools

Azure Network Watcher is a regional, cloud-native monitoring and network diagnostic service designed to inspect, monitor, and diagnose conditions in Azure Software-Defined Networking (SDN) infrastructure. In modern enterprise environments, diagnosing connectivity failures requires distinguishing between software-defined network boundary drops (NSGs), routing path misconfigurations (UDRs and BGP propagation), platform-level translation barriers (NAT/Firewalls), and in-guest operating system faults.

Understanding which Network Watcher diagnostic tool to apply at each phase of a troubleshooting workflow—and whether a given capability requires a dedicated VM extension or operates purely against the Azure management control plane—is a core objective for the AZ-700 exam.

1. Network Watcher Architecture & Regional Enablement

Unlike global management services such as Azure Monitor, Azure Network Watcher operates within a strict regional boundary:

• Automatic Regional Enablement: Azure automatically creates a Network Watcher instance whenever a virtual network is created in an Azure subscription and region. These instances reside in a system-managed resource group named NetworkWatcherRG (for example, NetworkWatcher_eastus in the NetworkWatcherRG resource group).
• Resource Provider: Network Watcher relies on the Microsoft.Network/networkWatchers resource provider. If this resource provider is disabled or deregistered, Network Watcher tools cannot interact with virtual networks in that region.
• Regional Scope Constraint: Network Watcher diagnostic tools can only target resources (VMs, NICs, subnets, VNets) located in the same geographic region as the Network Watcher instance.
• Agentless vs. Agent-Based Tools:
  • Agentless (Control-Plane / SDN-based): Tools such as IP Flow Verify, Next Hop, Effective Security Rules, and Effective Routes query Azure's underlying SDN fabric directly. They do not require any software agent inside the virtual machine, and they function even if the VM is deallocated or the guest OS is unresponsive.
  • Agent-Based (Data-Plane / In-Guest): Tools such as Packet Capture and Connection Monitor require the Azure Network Watcher Agent VM Extension (AzureNetworkWatcherExtension on Windows or NetworkWatcherAgentLinux on Linux) to capture raw frame buffers and generate synthetic Layer 4/7 test traffic.

2. IP Flow Verify & Next Hop Deep Dive

When a network engineer needs to determine whether a traffic flow is failing due to security filtering or incorrect routing, IP Flow Verify and Next Hop represent the first line of rapid diagnosis.

IP Flow Verify

IP Flow Verify tests whether a packet is permitted or denied based on the complete Layer 3/4 5-tuple parameters. It evaluates all Network Security Groups (NSGs) applied at both the Subnet level and the Network Interface (NIC) level, alongside Azure default rules.

+-----------------------------------------------------------------------------+
|                        IP FLOW VERIFY INPUT & OUTPUT                        |
|                                                                             |
|   INPUT PARAMETERS:                                                         |
|   - Target Resource: Target VM / Target Network Interface (NIC)             |
|   - Direction: Inbound OR Outbound                                          |
|   - Protocol: TCP OR UDP                                                    |
|   - Local IP & Port: e.g., 10.1.0.4 : 443                                   |
|   - Remote IP & Port: e.g., 198.51.100.25 : 51234                           |
|                                                                             |
|   EVALUATION ENGINE:                                                        |
|   - Subnet NSG Inbound/Outbound Rules (Priority 100-4096)                   |
|   - NIC NSG Inbound/Outbound Rules (Priority 100-4096)                      |
|   - Default Platform Rules (Priority 65000-65500)                           |
|                                                                             |
|   OUTPUT RESULT:                                                            |
|   - Access: 'Access Allowed' OR 'Access Denied'                             |
|   - Matched Rule: Exact name of the NSG rule that made the decision         |
|   - Resource Group / NSG Name: Exact resource ID of the enforcing NSG       |
+-----------------------------------------------------------------------------+

[!IMPORTANT] Scope Limitations of IP Flow Verify: IP Flow Verify only evaluates NSG security rules. It does not evaluate Azure Firewall policies, User-Defined Routes (UDRs), OS-level firewalls (such as Windows Defender Firewall or Linux iptables/nftables), or Load Balancer NAT rules. A result of Access Allowed means the packet passes Azure NSG filters, but it does not guarantee end-to-end delivery if an OS firewall or route table drops the packet.

Next Hop

Next Hop determines the next hop type and IP address for traffic destined to a specific target IP from a source virtual machine. Azure evaluates routes using the Longest Prefix Match (LPM) algorithm across three route sources: User-Defined Routes (UDRs), BGP-propagated routes (ExpressRoute / VPN Gateway), and System Default routes.

# Azure CLI: Verifying Next Hop for a VM targeting an on-premises database
az network watcher show-next-hop \
  --resource-group RG-Networking \
  --vm VM-Frontend-01 \
  --source-ip 10.1.0.4 \
  --dest-ip 192.168.100.50

3. Effective Security Rules & Effective Routes

Complex enterprise networks often feature multi-tier NSGs (applied to both subnets and individual NICs) alongside multiple route tables and BGP route advertisements. Diagnosing configuration drift across these layers requires inspecting the Effective state as computed by Azure SDN.

+-----------------------------------------------------------------------------+
|                   EFFECTIVE SECURITY RULES EVALUATION MODEL                 |
|                                                                             |
|   [ INBOUND TRAFFIC FLOW ]                                                  |
|   1. Evaluates Subnet NSG Rules (Ordered by Priority: 100 -> 4096)          |
|      - If explicitly DENIED -> Packet Dropped Immediately                   |
|      - If explicitly ALLOWED -> Moves to Step 2                             |
|   2. Evaluates NIC NSG Rules (Ordered by Priority: 100 -> 4096)             |
|      - If explicitly DENIED -> Packet Dropped Immediately                   |
|      - If explicitly ALLOWED -> Packet Reaches VM NIC                       |
|                                                                             |
|   [ OUTBOUND TRAFFIC FLOW ]                                                 |
|   1. Evaluates NIC NSG Rules (Ordered by Priority: 100 -> 4096)             |
|      - If explicitly DENIED -> Packet Dropped Immediately                   |
|      - If explicitly ALLOWED -> Moves to Step 2                             |
|   2. Evaluates Subnet NSG Rules (Ordered by Priority: 100 -> 4096)          |
|      - If explicitly DENIED -> Packet Dropped Immediately                   |
|      - If explicitly ALLOWED -> Packet Egresses Subnet                      |
+-----------------------------------------------------------------------------+

Effective Security Rules Inspection

• Consolidated View: Displays all security rules impacting a specific NIC, grouping them by Subnet association, NIC association, and Default rules.
• Overriding Logic: Identifies when an NSG rule at the subnet level is nullified or blocked by an NSG rule at the NIC level (or vice versa).
• Expanded ASGs: Automatically resolves Application Security Groups (ASGs) into their underlying private IP addresses, showing exact matching criteria.

Effective Routes Inspection

• Active Forwarding Table: Displays the live Forwarding Information Base (FIB) of the target NIC.
• State Flags:
  • Active: The route currently in effect for traffic matching the prefix.
  • Invalid: A route that is overridden by a more specific prefix (Longest Prefix Match) or disabled due to an invalid next hop IP (e.g., an NVA IP that is unreachable or outside the subnet).
• BGP Propagation Verification: If Propagate gateway routes is disabled on a route table, BGP routes will be absent from Effective Routes, explaining why on-premises subnets are unreachable.

4. Connection Troubleshoot & Connection Monitor

While IP Flow Verify and Next Hop test static SDN policy and route tables, Connection Troubleshoot and Connection Monitor test dynamic, active data-plane packet delivery.

+-----------------------------------------------------------------------------+
|               CONNECTION TROUBLESHOOT VS CONNECTION MONITOR                 |
|                                                                             |
|   Capability             Connection Troubleshoot    Connection Monitor      |
|   --------------------   -----------------------    ------------------      |
|   Execution Model        Ad-hoc / Instantaneous     Continuous / Scheduled  |
|   Probing Protocols      TCP, ICMP                  TCP, ICMP, HTTP, HTTPS  |
|   Multi-Cloud / On-Prem  No (Azure VM source only)  Yes (Hybrid Agents)     |
|   Hop-by-Hop Topology    Yes (Single trace)         Yes (Historical graphs) |
|   Automated Alerting     No                         Yes (Azure Monitor)     |
|   Configuration Storage  None (Ephemeral test)      ARM Resource / Workspace|
+-----------------------------------------------------------------------------+

Connection Troubleshoot

• Instantaneous Point-in-Time Test: Probes Layer 4 connectivity from a source Azure VM (with Network Watcher Agent) to a destination (Azure VM, FQDN, URI, or IPv4 address).
• Diagnostic Output:
  • Status: Reachable or Unreachable.
  • Round-Trip Latency: Minimum, average, and maximum latency in milliseconds.
  • Hop-by-Hop Breakdown: Identifies every intermediate network hop, including virtual network peering, Azure Firewall, NVAs, and gateway endpoints.
  • Root Cause Detection: Returns explicit fault reasons such as UserDefinedRouteReason, NetworkSecurityRuleReason, or GuestFirewallReason (detecting when an OS-level firewall blocks SYN packets).

Connection Monitor (V2 Unified Architecture)

Connection Monitor provides unified, continuous connectivity monitoring across hybrid, multi-cloud, and Azure cloud networks.

• Test Groups: Logical groupings of sources, destinations, and test configurations.
• Test Configurations: Specifies protocol (TCP, ICMP, HTTP/HTTPS), test frequency (e.g., every 30 seconds), target port, and performance thresholds (acceptable latency and packet loss %).
• Diagnostic Alerts: Directly triggers Azure Monitor action groups when round-trip latency exceeds baseline SLAs or packet loss exceeds threshold limits.

5. Packet Capture on Virtual Machines and Scale Sets

When packet loss, protocol negotiation failures, or payload corruption cannot be identified through high-level logs, Network Watcher Packet Capture allows security and network engineers to capture raw network frames directly on Azure VMs and Virtual Machine Scale Sets (VMSS).

+-----------------------------------------------------------------------------+
|                     PACKET CAPTURE CONFIGURATION MODEL                      |
|                                                                             |
|   [ Target VM / VMSS Instance ] <--- Network Watcher Agent VM Extension    |
|                 |                                                           |
|                 |--- Captures Raw Frames (.cap)                             |
|                 |                                                           |
|                 +-----------------------------------+                       |
|                 |                                   |                       |
|                 v                                   v                       |
|   [ Azure Storage Account ]               [ Local VM Disk Path ]            |
|   - Blob Container: 'network-captures'    - Windows: C:\captures\trace.cap  |
|   - URI: https://mystorage.blob...        - Linux: /var/captures/trace.cap  |
|   - Centralized SecOps analysis           - Rapid local debugging           |
+-----------------------------------------------------------------------------+

Core Configuration Parameters & Safeguards

1. Storage Destination:
   • Azure Storage Account: Writes .cap files directly to a designated blob container. Recommended for multi-VM captures and centralized Wireshark analysis.
   • Local File: Writes to a local file system path on the target VM.
   • Dual Storage: Both blob storage and local file paths can be configured simultaneously.
2. Capture Filters (Protocol, IP, and Port):
   • Protocol: Filter by TCP, UDP, or Any.
   • Local / Remote IP Address: Limit captures to specific communication partners.
   • Local / Remote Port: Limit capture to target ports (e.g., port 443 or 1433) to prevent capturing massive background traffic.
3. Safety & Resource Safeguards:
   • Maximum Duration (seconds): Default is 18,000 seconds (5 hours); session stops automatically once reached.
   • Maximum Capture Size (bytes): Default is 1,073,741,824 bytes (1 GB); capture terminates when limit is reached to protect disk and storage quotas.
   • Bytes Per Packet (Truncation / Slicing): Limits the number of bytes captured per frame (e.g., capturing the first 128 bytes captures L3/L4 headers while omitting sensitive payload data and conserving storage).