10.3 Network Troubleshooting Scenarios & End-to-End Connectivity

Network Troubleshooting Scenarios & End-to-End Connectivity

Enterprise Azure networking environments combine hybrid connectivity (ExpressRoute, S2S VPN), hub-and-spoke topologies, stateful security appliances (Azure Firewall, NVAs), and private PaaS integrations (Private Endpoints). When connectivity fails in these multi-layered topologies, network engineers must execute systematic, structured root-cause analysis.

This section details five critical real-world troubleshooting scenarios frequently tested on the AZ-700 exam: asymmetric routing, Private Endpoint DNS resolution failure, SNAT port exhaustion, BGP route manipulation, and IPsec VPN negotiation failures.

1. Scenario 1: Asymmetric Routing Across Stateful Firewalls & Dual NVAs

Asymmetric routing occurs when an outbound network packet traverses one path (e.g., through a stateful firewall or NVA), but the corresponding return packet takes a different path (e.g., bypassing the firewall or hitting a different NVA instance in a scale set).

+-----------------------------------------------------------------------------+
|                   ASYMMETRIC ROUTING BREAKDOWN & REMEDIATION                |
|                                                                             |
|   FAILURE TOPOLOGY (ASYMMETRIC DROP):                                       |
|   1. Spoke VM (10.1.0.4) sends TCP SYN to On-Premises (192.168.1.50).       |
|   2. UDR on Spoke routes packet -> Azure Firewall / NVA (10.100.0.4).       |
|   3. Firewall creates state table entry: [10.1.0.4:51234 -> 192.168.1.50:443]|
|   4. On-Premises server receives SYN and replies with TCP SYN-ACK.          |
|   5. On-Premises gateway has a direct route or ExpressRoute path to Spoke   |
|      VNet (10.1.0.0/16) that BYPASSES the Azure Firewall.                  |
|   6. Spoke VM receives SYN-ACK from an unexpected path, or if return hits a|
|      different NVA instance, the NVA drops the SYN-ACK (no state entry).   |
|                                                                             |
|   REMEDIATION 1: Source NAT (SNAT) on NVA / Firewall                        |
|   - NVA replaces Source IP (10.1.0.4) with NVA's own IP (10.100.0.4).       |
|   - On-Premises server MUST reply directly to NVA (10.100.0.4).             |
|   - Symmetric path is guaranteed across all connection states.              |
|                                                                             |
|   REMEDIATION 2: Symmetric Route Tables (UDRs) & GatewaySubnet Routing      |
|   - Associate a UDR to 'GatewaySubnet' routing Spoke CIDRs -> NVA IP.       |
|   - Enforces bidirectional symmetry without requiring SNAT.                 |
+-----------------------------------------------------------------------------+

Diagnostic Workflow:

1. Inspect Effective Routes on both the source VM NIC and the GatewaySubnet.
2. Verify whether GatewaySubnet has a UDR directing return traffic back through the NVA.
3. Check NVA / Azure Firewall diagnostic logs (AzureDiagnostics or AZFWNetworkRule) for dropped packets marked with invalid TCP state or out-of-order ACK flags.

2. Scenario 2: Split-Brain DNS Failures for Private Endpoints

When organizations deploy Azure Private Endpoints for PaaS resources (e.g., Azure SQL Database, Azure Blob Storage), Azure automatically assigns a private IP within the virtual network. To ensure seamless access, Azure registers the resource in an Azure Private DNS Zone (e.g., privatelink.blob.core.windows.net).

The Problem: Hybrid DNS Resolution Failure

On-premises clients connecting to an Azure Storage account (mystorage.blob.core.windows.net) resolve the public IP address instead of the Private Endpoint private IP (10.1.5.4), receiving an HTTP 403 Forbidden or connection timeout.

+-----------------------------------------------------------------------------+
|                   HYBRID PRIVATE ENDPOINT DNS RESOLUTION                    |
|                                                                             |
|   [ On-Premises Client ]                                                    |
|             | (1) Queries: mystorage.blob.core.windows.net                  |
|             v                                                               |
|   [ On-Premises DNS Server (Active Directory / BIND) ]                      |
|             |                                                               |
|             | (2) Conditional Forwarder: 'blob.core.windows.net' ->         |
|             |     Targets Inbound Endpoint IP (10.100.0.132)                |
|             v                                                               |
|   [ Azure Private DNS Resolver: Inbound Endpoint (10.100.0.132) ]           |
|             |                                                               |
|             | (3) Queries linked Private DNS Zone:                          |
|             |     privatelink.blob.core.windows.net                         |
|             v                                                               |
|   [ Azure Private DNS Zone ]                                                |
|   - Record: mystorage.privatelink.blob.core.windows.net -> 10.1.5.4         |
|             |                                                               |
|             | (4) Returns Private IP: 10.1.5.4                              |
|             v                                                               |
|   [ Client Connects Directly to 10.1.5.4 via ExpressRoute / S2S VPN ]       |
+-----------------------------------------------------------------------------+

Diagnostic Checklist for Private Endpoint DNS:

1. Do NOT Target 168.63.129.16 from On-Premises: The Azure recursive resolver IP (168.63.129.16) is a virtual non-routable link-local IP accessible only from within Azure virtual networks. On-premises DNS servers cannot forward queries directly to 168.63.129.16.
2. Deploy Azure Private DNS Resolver: Provision an Inbound Endpoint in a dedicated subnet inside the hub VNet. Configure on-premises DNS conditional forwarders to forward queries for blob.core.windows.net (or privatelink.blob.core.windows.net) to the Inbound Endpoint private IP.
3. Virtual Network Links: Verify that the Azure Private DNS Zone has an active Virtual Network Link associated with the Hub VNet and all relevant Spoke VNets.

3. Scenario 3: SNAT Port Exhaustion on Load Balancers & NAT Gateways

When virtual machines inside Azure initiate outbound connections to external public endpoints (APIs, third-party services, package repositories), Azure performs Source Network Address Translation (SNAT) to map the VM's private IP to a public IP address.

The Failure Mechanism: Pre-Allocated SNAT Starvation

• Standard Load Balancer Default Allocation: When outbound rules are configured on a Standard Load Balancer without custom allocation, Azure allocates a fixed, static number of SNAT ports (typically 1,024 ports per VM instance) from each assigned public IP.
• Exhaustion Condition: If an application opens hundreds of concurrent outbound connections in rapid succession, it exhausts its 1,024-port quota. Additional outbound connection attempts are dropped, resulting in intermittent connection timeouts and high latency.

+-----------------------------------------------------------------------------+
|                   SNAT PORT CAPACITY & ALLOCATION MATRIX                    |
|                                                                             |
|   Architectural Feature      Standard Load Balancer   Azure NAT Gateway     |
|   ------------------------   ----------------------   -----------------     |
|   SNAT Allocation Model      Static (Pre-allocated)   Dynamic (On-Demand)   |
|   Ports Per Public IP        64,000                   64,000                |
|   Concurrency per Subnet VM  Fixed (e.g., 1,024 max)  Dynamic (Up to 64,000)|
|   Max Public IPs Attached    Configurable             Up to 16 IPs (1M+ pts)|
|   Availability Zone Resilient Yes                     Yes (Zone-Redundant)  |
|   Subnet Association Model   NIC Backend Pool         Direct Subnet Binding |
+-----------------------------------------------------------------------------+

Diagnostic Metrics & Remediation:

1. Monitor Azure Metrics:
   • Used SNAT Ports and Allocated SNAT Ports (Load Balancer / NAT Gateway).
   • Failed SNAT Connections (Load Balancer) / Dropped Packets (NAT Gateway).
   • A spike in Failed SNAT Connections while Used SNAT Ports approaches Allocated SNAT Ports confirms SNAT port exhaustion.
2. Best Practice Remediation: Deploy an Azure NAT Gateway on the subnet. NAT Gateway provides fully dynamic, on-demand SNAT port allocation across up to 16 Public IPs (yielding up to 1,024,000 concurrent flows), eliminating fixed per-VM port starvation.

4. Scenario 4: BGP Routing Loops & AS-Path Prepending Issues

In hybrid architectures utilizing BGP over ExpressRoute and Site-to-Site VPN, incorrect BGP attribute propagation can lead to routing loops, suboptimal asymmetric routing, or complete route withdrawal.

+-----------------------------------------------------------------------------+
|                   BGP ROUTE SELECTION & AS-PATH PREPENDING                  |
|                                                                             |
|   AZURE BGP BEST PATH SELECTION CRITERIA (IN ORDER):                        |
|   1. Longest Prefix Match (LPM) — e.g., /24 preferred over /16              |
|   2. User-Defined Routes (UDR) — Overrides BGP routes                       |
|   3. BGP Local Preference (if configured)                                   |
|   4. Shortest AS-Path Length (Evaluates number of ASNs in AS_PATH)          |
|   5. Origin Type (IGP preferred over EGP, EGP preferred over Incomplete)    |
|   6. Lowest Multi-Exit Discriminator (MED)                                  |
|                                                                             |
|   ACTIVE / PASSIVE FAILOVER VIA AS-PATH PREPENDING:                         |
|   - Primary Link (ExpressRoute): Advertises ASN 65001 (AS-Path Length = 1)   |
|   - Backup Link (S2S VPN): Advertises ASN 65001 65001 65001 (Length = 3)    |
|   - Azure Result: Selects ExpressRoute because AS-Path is shorter.          |
|   - Failover: If ExpressRoute drops, Azure automatically routes via VPN.    |
+-----------------------------------------------------------------------------+

Common BGP Pitfalls for AZ-700:

• Reserved ASNs: Azure reserves ASNs 65515 through 65520 for internal gateway infrastructure (VPN Gateway, ExpressRoute Gateway, Azure Route Server). Attempting to configure an on-premises BGP peer with ASN 65515 causes peering session establishment failure.
• Route Leaking: If an on-premises edge router or Azure NVA re-advertises routes learned from Azure back into Azure over a second circuit, a routing loop occurs. Enforce strict BGP route filtering on customer premises equipment (CPE).
• ExpressRoute Route Limits: Standard SKU ExpressRoute circuits support a maximum of 4,000 routes; Premium SKU supports up to 10,000 routes. If on-premises edge routers advertise prefixes exceeding this quota, the BGP session is instantly dropped by Microsoft Enterprise Edge (MSEE) routers.

5. Scenario 5: VPN Gateway Tunnel Negotiation Failures

Site-to-Site (S2S) VPN connectivity issues between an Azure Virtual Network Gateway and an on-premises firewall/router typically occur during cryptographic handshake negotiation.

+-----------------------------------------------------------------------------+
|                     IPSEC / IKE NEGOTIATION PHASES                          |
|                                                                             |
|   PHASE 1: IKE SA (Main Mode)                                               |
|   - Purpose: Authenticates peers and creates secure management channel.     |
|   - Negotiated Parameters:                                                  |
|     * Pre-Shared Key (PSK) — MUST MATCH EXACTLY                             |
|     * IKE Version (IKEv1 vs IKEv2)                                          |
|     * Encryption Algorithm: AES256 / AES128                                 |
|     * Integrity / Hash: SHA256 / SHA1                                       |
|     * Diffie-Hellman (DH) Group: Group 14 (2048-bit) / Group 24             |
|     * IKE SA Lifetime: Default 28,800 seconds                               |
|   - #1 Failure Cause: Pre-Shared Key (PSK) typo or crypto proposal mismatch|
|                                                                             |
|   PHASE 2: Child IPsec SA (Quick Mode)                                      |
|   - Purpose: Establishes encrypted data-plane tunnel for customer payload.  |
|   - Negotiated Parameters:                                                  |
|     * IPsec Encryption: AES256 / AES-GCM                                   |
|     * IPsec Integrity: SHA256                                               |
|     * Perfect Forward Secrecy (PFS) Group: PFS2048 / None                   |
|     * IPsec SA Lifetime: Default 27,000 seconds / 7,200 seconds             |
|     * Traffic Selectors (Local & Remote Address Prefixes)                   |
|   - #1 Failure Cause: Traffic Selector / Policy-Based mismatch              |
+-----------------------------------------------------------------------------+

Diagnostic Steps & Azure Logs:

1. Check Connection Status:
   • Run Get-AzVirtualNetworkGatewayConnection or az network vpn-connection show.
   • Status Connecting indicates Phase 1 or Phase 2 failure; NotConnected indicates unreachable peer IP or platform shutdown.
2. Enable VPN Diagnostic Logs in Log Analytics:
   • Ingest IKEDiagnosticLog, RouteDiagnosticLog, and TunnelDiagnosticLog.
   • Query IKEDiagnosticLog for INVALID_PAYLOAD_TYPE (incorrect PSK), NO_PROPOSAL_CHOSEN (crypto suite mismatch), or TS_UNACCEPTABLE (traffic selector mismatch).
3. Policy-Based vs. Route-Based Traffic Selectors:
   • Azure Route-Based gateways negotiate wildcards (0.0.0.0/0 <-> 0.0.0.0/0).
   • If connecting to a legacy Policy-Based device expecting specific subnet pairs (e.g., 10.1.0.0/24 <-> 192.168.1.0/24), enable UsePolicyBasedTrafficSelectors: $true on the Azure Connection resource.