12.3 Layer 3, IP Services, and End-to-End Diagnostics

Layer 3, IP Services, and End-to-End Diagnostics

Quick Summary: Once physical and data link integrity is verified, troubleshooting shifts to Layer 3 IP routing, core network services (DHCP and DNS), and end-to-end application reachability. Aruba AOS-CX switches provide sophisticated diagnostic utilities—including extended ICMP ping, traceroute path validation, VRF-aware routing table inspection, DHCP relay/snooping telemetry, and automated Network Analytics Engine (NAE) agents. Mastering these tools enables network engineers to resolve complex inter-VLAN routing failures, isolate transit drops, and definitively distinguish between underlying network latency and application server processing delays.

---

ICMP Diagnostic Utilities: Ping and Traceroute Mechanics

Internet Control Message Protocol (ICMP) tools provide baseline verification for end-to-end Layer 3 reachability.

The Mechanics of AOS-CX Ping

The ping command transmits ICMP Echo Request packets (Type 8, Code 0) to a target IP address and listens for ICMP Echo Reply packets (Type 0, Code 0).

switch# ping 10.20.1.50
PING 10.20.1.50 (10.20.1.50) 100(128) bytes of data.
108 bytes from 10.20.1.50: icmp_seq=1 ttl=64 time=1.23 ms
108 bytes from 10.20.1.50: icmp_seq=2 ttl=64 time=0.98 ms
108 bytes from 10.20.1.50: icmp_seq=3 ttl=64 time=1.04 ms
108 bytes from 10.20.1.50: icmp_seq=4 ttl=64 time=1.11 ms
108 bytes from 10.20.1.50: icmp_seq=5 ttl=64 time=0.95 ms

--- 10.20.1.50 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 0.950/1.062/1.230/0.098 ms

Advanced Ping Parameters in AOS-CX

A standard ping uses default parameters (5 packets, 100 bytes payload, egress interface IP as source, default VRF). In complex enterprise campus networks, standard ping is insufficient. AOS-CX provides advanced flags:

switch# ping <ip-address> [repetitions <count>] [datagram-size <bytes>] [timeout <seconds>] [source <ip|interface>] [vrf <vrf-name>] [do-not-fragment]

1. Specifying Source IP / Interface (source <ip|interface>):
   • By default, the switch uses the IP address of the physical egress port or SVI closest to the destination as the packet source.
   • Sourcing pings from a specific SVI or Loopback interface (ping 10.50.1.1 source 10.10.1.1) tests whether return routing exists back to that specific subnet, and validates that transit Access Control Lists (ACLs) permit traffic originating from client subnets.
2. Path MTU Discovery and Isolating MTU Black Holes (datagram-size + do-not-fragment):
   • When a packet exceeds an interface MTU and the Do Not Fragment (DF) bit is set in the IP header, the transit router drops the packet and returns an ICMP Destination Unreachable - Fragmentation Needed (Type 3, Code 4) message.
   • If an intermediate firewall silently discards ICMP Type 3 messages, an MTU Black Hole is created: standard small pings succeed, but large application data packets (such as database transfers, SSL handshakes, or file downloads) freeze and drop indefinitely.
   • To test the exact path MTU on AOS-CX, set the DF bit and sweep the packet size: switch# ping 10.20.1.50 datagram-size 1500 do-not-fragment
3. VRF Awareness (vrf <name>):
   • SVI interfaces and management interfaces often reside in isolated Virtual Routing and Forwarding (VRF) instances (such as mgmt or custom tenant VRFs). Sourcing a ping without specifying vrf <name> executes against the default VRF table, resulting in false failures when testing out-of-band management targets.

Interpreting Ping Results

AOS-CX ping output looks like Linux ping output: one line per reply with its ttl and round-trip time, followed by a summary of packets transmitted, received, and lost.

• Replies received: the target and the return path work for that source and VRF.
• 100% loss: the target is down, something in the path drops ICMP, or the target has no route back to the source address.
• Partial loss: congestion, rate limiting of ICMP, or an intermittent physical fault.
• "Destination unreachable" or "time to live exceeded" messages: a router in the path returned an ICMP error (no route, an ACL, or a routing loop).

The Mechanics of Traceroute

While ping confirms end-to-end reachability, traceroute identifies the exact hop-by-hop layer 3 path packets traverse.

switch# traceroute 10.30.1.10 vrf default
traceroute to 10.30.1.10 (10.30.1.10), 30 hops max, 60 byte packets
 1  10.1.1.1 (10.1.1.1)  1.124 ms  1.012 ms  0.985 ms
 2  10.254.1.2 (10.254.1.2)  2.341 ms  2.115 ms  2.089 ms
 3  10.30.1.10 (10.30.1.10)  3.452 ms  3.210 ms  3.184 ms

• How It Works: Traceroute sends UDP probe packets (or ICMP Echo Requests) with an initial Time-to-Live (TTL) of 1. The first transit router decrements TTL from 1 to 0, discards the packet, and transmits an ICMP Time Exceeded (Type 11, Code 0) message back to the switch, identifying Hop 1. Traceroute then sends probes with TTL=2 to discover Hop 2, TTL=3 for Hop 3, until reaching the destination.
• Diagnostic Interpretations:
  • Asterisks (* * *): An intermediate router does not respond to ICMP TTL expired messages or a firewall blocks the probes. If subsequent hops respond, traffic is transiting normally (router simply deprioritized ICMP generation). If all subsequent hops display * * *, the packet is being dropped at or immediately after the last responding hop.
  • Repeating IP Addresses: If Hop 4 and Hop 5 oscillate repeatedly between two IP addresses (10.0.0.1 -> 10.0.0.2 -> 10.0.0.1), a routing loop exists between those two routers.

---

IPv4 Routing Table and VRF Diagnostics

When Layer 3 forwarding fails, the administrator must inspect the switch's Routing Information Base (RIB) using show ip route.

switch# show ip route
Displaying ipv4 routes selected for forwarding
'[x/y]' denotes [preference/metric]

0.0.0.0/0, vrf default
        via 10.0.1.1, [1/0], static
10.10.0.0/24, vrf default
        via vlan10, [0/0], connected
10.20.0.0/24, vrf default
        via 10.0.1.2, [110/20], ospf
10.30.0.0/16, vrf default
        via 10.0.1.3, [110/30], ospf

Key Routing Diagnostic Checks

1. Route Presence and Longest Prefix Match (LPM):
   • Verify that an entry matching the target destination exists. Remember: the switch matches destination IP addresses against the longest prefix length (most specific subnet mask).
   • If traffic to 10.20.0.50 fails, check if a specific /24 or /32 route exists that overrides a broader /16 route or the default route (0.0.0.0/0).
2. Administrative Distance (AD) and Route Selection:
   • Connected: AD 0 (Active SVI/interface in up/up state)
   • Static Route: AD 1
   • eBGP: AD 20
   • OSPF: AD 110
   • iBGP: AD 200
   • If a route is missing from show ip route, check if the interface is down or if an alternate routing source with a lower AD was selected.
3. Next-Hop Reachability and Recursive Lookups:
   • For static routes, verify that the configured next-hop IP is reachable and resolves in the ARP cache (show arp). If the next-hop interface is down, AOS-CX removes the static route from the active forwarding table.
4. VRF Routing Table Isolation:
   • AOS-CX maintains completely separate routing tables for each VRF instance.
   • To view routes in the management VRF, run: show ip route vrf mgmt
   • To view routes in a dedicated tenant VRF, run: show ip route vrf <vrf-name>

---

Domain Name System (DNS) Troubleshooting

DNS translates human-readable Fully Qualified Domain Names (FQDNs) into routable IP addresses. In enterprise networks, users frequently report "the network is down" when underlying IP routing is completely healthy but DNS name resolution has failed.

The Two-Step DNS Diagnostic Test

To determine whether a problem is an IP network fault or a DNS failure, execute the two-step ping test from the client or switch CLI:

• Step 1: Ping the remote service by its raw IP address (ping 198.51.100.25). If this succeeds, Layer 1, 2, and 3 network transport is proven fully functional.
• Step 2: Ping the remote service by its FQDN (ping intranet.company.com). If this fails with "Host name lookup failure" or "Unknown host", the failure is isolated to DNS.

AOS-CX DNS Verification Commands

switch# show ip dns
DNS Name Resolution is Enabled
Domain Name : corp.local
Name Servers:
    10.1.10.10 (vrf default)
    10.1.10.11 (vrf default)
Domain Search List:
    corp.local
    datacenter.corp.local

Common DNS Troubleshooting Points:

• Verify DNS server reachability: Ping the configured DNS server IP (ping 10.1.10.10). If unreachable, check routing and firewalls.
• Firewall Port 53 Filtering: DNS operates over UDP port 53 (and TCP port 53 for large transfers). Ensure intermediate firewalls and switch ACLs permit outbound UDP 53 to authorized enterprise DNS servers.
• VRF Assignment: In AOS-CX, verify which VRF the DNS server belongs to. If the DNS server is reachable via the management port, it must be configured in vrf mgmt.

---

Dynamic Host Configuration Protocol (DHCP) Troubleshooting

DHCP dynamically assigns IPv4 addresses, subnet masks, default gateways, and DNS server addresses to client devices.

The Four-Step DHCP DORA Sequence

Client                                          Switch SVI / Relay                     DHCP Server
  |                                                     |                                   |
  |--- 1. Discover (L2 Broadcast: 255.255.255.255) ---->|                                   |
  |                                                     |--- 1. Relayed Discover (Unicast)->|
  |                                                     |                                   |
  |                                                     |<-- 2. Relayed Offer (Unicast) ----|
  |<-- 2. Offer (L2 Unicast or Broadcast) --------------|
  |                                                     |                                   |
  |--- 3. Request (L2 Broadcast: 255.255.255.255) ----->|                                   |
  |                                                     |--- 3. Relayed Request (Unicast) ->|
  |                                                     |                                   |
  |                                                     |<-- 4. Relayed ACK (Unicast) ------|
  |<-- 4. ACK (L2 Unicast or Broadcast) ----------------|

Troubleshooting APIPA (169.254.x.x)

When a Windows or macOS workstation fails to complete the DHCP DORA exchange, it self-assigns an Automatic Private IP Addressing (APIPA) address in the 169.254.0.0/16 range.

• Diagnostic Conclusion: Seeing a 169.254.x.x address is conclusive evidence that the client workstation transmitted a DHCP Discover but received no valid DHCP Offer or ACK from a server.

DHCP Relay (ip helper-address) Diagnostics

Because DHCP Discover and Request packets are Layer 2 broadcasts, routers and multilayer switches terminate them and do not forward them across VLAN boundaries. When the DHCP server resides on a centralized server subnet or in a data center, the client's default gateway SVI must act as a DHCP Relay Agent:

switch# configure terminal
switch(config)# interface vlan 20
switch(config-if-vlan)# ip helper-address 10.1.10.5

• How It Works: When the SVI receives a broadcast DHCP Discover on UDP port 67, it converts the broadcast into a unicast UDP packet destined for 10.1.10.5. It inserts its own SVI IP address into the giaddr (Gateway IP Address) field of the DHCP payload so the DHCP server knows which subnet pool to allocate from.
• Verification Commands: show ip helper-address lists the configured helper addresses, and show dhcp-relay shows whether the relay agent is enabled plus its valid and dropped request/response counters.

DHCP Snooping Diagnostics

DHCP Snooping is a Layer 2 security mechanism on access switches that prevents unauthorized (rogue) DHCP servers from distributing fraudulent IP configurations and default gateway addresses to clients.

• Trusted vs. Untrusted Ports:
  • Untrusted Ports (Default): Host-facing access ports. The switch permits incoming DHCP client requests (Discover, Request), but drops any incoming DHCP server responses (Offer, ACK, NAK)!
  • Trusted Ports: Uplink ports connected toward legitimate DHCP servers. The switch permits server responses.
• Classic Troubleshooting Pitfall: An administrator enables DHCP Snooping globally and on user VLANs, but forgets to configure dhcp-snooping trust on the uplink ports! The access switch drops all incoming DHCP Offers from the legitimate server, causing every client on the switch to fall back to 169.254.x.x APIPA addresses.
• Diagnostic Commands:
  • show dhcp-snooping: Displays enabled VLANs and trusted ports.
  • show dhcp-snooping statistics: Displays drop counters for untrusted server packets.
  • show dhcp-snooping binding: Displays the active database of learned client MACs, leased IPs, and lease durations.

---

Isolating Network Latency vs. Server / Application Delay

When users complain that "the application is slow," the network engineer must definitively prove whether the root cause is network transport latency or server application processing delay.

Client Workstation                                                       Server Endpoint
      |                                                                         |
      |--- 1. TCP SYN --------------------------------------------------------->|
      |<-- 2. TCP SYN-ACK (Network RTT = SYN to SYN-ACK) -----------------------|
      |--- 3. TCP ACK (Handshake Complete) ------------------------------------>|
      |                                                                         |
      |=== 4. HTTP GET / Database Query =======================================>|
      |    [ Server Processing Time / Application Delay / Database Query Run ]  |
      |<== 5. HTTP 200 OK / Data Payload (Time to First Byte - TTFB) ===========|

1. TCP Three-Way Handshake Analysis (Network RTT)

The duration of the TCP three-way handshake (the time elapsed between the client sending the initial SYN and receiving the server's SYN-ACK) reflects the pure network round-trip time (RTT) across physical cables, transceivers, switches, and routers.

• Because the server's operating system kernel automatically generates the SYN-ACK at the transport layer without invoking the underlying application, the TCP handshake duration is independent of server application load.
• If the TCP handshake completes in 2 to 5 milliseconds, the network transport path is fast and healthy.

2. Application Processing Time (Time to First Byte - TTFB)

The time elapsed between the client completing its application request (e.g., sending an HTTP GET or SQL query) and the server transmitting the first byte of response data (HTTP 200 OK) represents server processing latency.

• If the TCP handshake takes 2 ms, but the server takes 4,500 ms to return the first byte of data, the delay is 100% attributable to server-side bottlenecks (e.g., database table locks, server CPU throttling, disk I/O wait, or unoptimized application code)—not the campus network!

3. Aruba Network Analytics Engine (NAE)

Aruba AOS-CX switches feature the Network Analytics Engine (NAE), a built-in telemetry framework that automates network performance analysis:

• State Database Integration: NAE scripts run locally in Python directly on the switch, querying the state database (OVSDB) in real time.
• Automated Root-Cause Correlation: NAE continuously tracks interface buffer drops, CPU spikes, OSPF adjacency flaps, and route churn. When an anomaly occurs, NAE automatically captures diagnostic snapshots, executes targeted CLI commands, and flags the event in the Web UI, eliminating manual post-incident reconstruction.

---

Summary of AOS-CX Layer 3 CLI Diagnostic Commands