14.1 Diagnostic Utilities, Conditional Debugging & Path MTU Discovery

14.1 Diagnostic Utilities & Path MTU Discovery

Core Blueprint Focus: Cisco 350-401 ENCOR v1.2 topic 4.1 requires candidates to diagnose network problems using tools such as debugs, conditional debugs, traceroute, ping, SNMP, and syslog. MTU and PMTU behaviour is not a named leaf topic, but it underpins topic 4.1 diagnosis and the tunnel overhead arithmetic in topic 2.2.b. Candidates must master ICMP message types and codes, extended ping parameters, traceroute port mechanics, Path MTU Discovery (PMTUD RFC 1191), and TCP Maximum Segment Size (MSS) adjustment.

In mission-critical enterprise environments, isolating Layer 3 forwarding and transport-layer failures requires a rigorous understanding of diagnostic protocols. The Internet Control Message Protocol (ICMP), defined in RFC 792 for IPv4 and RFC 4443 for IPv6, operates as the foundational control plane signaling mechanism for IP networks. Diagnostic utilities like Ping and Traceroute leverage specific ICMP message types, code fields, and IP header options to evaluate reachability, hop-by-hop latency, path MTU limitations, and physical transmission integrity.

1. ICMP Architecture, Types & Codes

ICMP messages are encapsulated directly within IP datagrams (IPv4 Protocol 1, IPv6 Next Header 58) without Layer 4 transport headers. An ICMP header contains an 8-bit Type field, an 8-bit Code field, a 16-bit Checksum, and variable Header Data depending on the specific type.

+-----------------------------------------------------------------------------------+
|                             ICMPv4 PACKET ENCAPSULATION                           |
+-----------------------------------------------------------------------------------+
| 0                   1                   2                   3                     |
| 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1                   |
|+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                  |
||     Type      |     Code      |          Checksum             |  ICMP Header     |
|+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  (8 Bytes)       |
||           Identifier          |        Sequence Number        |                  |
|+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                  |
||     Original IP Header + First 8 Bytes of Original Payload    |  (For Error      |
||     (Used for Error Diagnostics and Session Correlation)      |   Messages)      |
|+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                  |
+-----------------------------------------------------------------------------------+

Comprehensive ICMP Types and Diagnostic Codes Reference

2. Extended Ping Mechanics & Data Pattern Diagnostics

While basic CLI ping (ping <target-ip>) sends 5 ICMP Echo Requests of 100 bytes each with a 2-second timeout, Extended Ping provides interactive, granular control over Layer 3 headers, packet sizing, timing, and payload bit composition.

+-----------------------------------------------------------------------------------+
|                         EXTENDED PING DIAGNOSTIC CAPABILITIES                     |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|  [ Diagnostic Capability ]         [ Operational & Troubleshooting Purpose ]      |
|  --------------------------------  ---------------------------------------------  |
|  Source Address / Interface        Forces packet to originate from loopback/SVI;  |
|                                    validates asymmetric routing & ACL rules.      |
|  Don't Fragment (DF) Bit Set       Prevents intermediate fragmentation; isolates  |
|                                    exact path MTU bottlenecks when sweeping size. |
|  Sweep Range of Sizes              Transmits sequence of sizes (e.g., 1400-1500)  |
|                                    to pinpoint maximum allowable frame size.      |
|  Type of Service (ToS) / DSCP      Marks IP precedence/DSCP bits to test QoS      |
|                                    queuing, policing, and shaping policies.       |
|  Data Pattern (e.g., 0x5555)       Stresses physical media to detect clocking     |
|                                    slips, serial framing errors, and bit drops.   |
|  Validate Reply Data               Bitwise verification that echoed payload data  |
|                                    was not corrupted in transit.                  |
+-----------------------------------------------------------------------------------+

Character Codes in Cisco IOS Ping Output

When executing ping commands in Cisco IOS-XE, the CLI returns specific single-character status codes:

• ! (Exclamation point): Each exclamation point indicates receipt of a valid ICMP Echo Reply (Type 0 Code 0).
• . (Period): Indicates the network server timed out while waiting for a reply (default 2 seconds). Typically indicates packet drop, route loss, or silent firewall filtering.
• U (Unreachable): An ICMP Destination Unreachable message (Type 3) was received. Common causes include missing routes (Code 0), failed ARP (Code 1), or ACL drop (Code 13).
• M (Could not fragment): An ICMP Type 3 Code 4 message was received, indicating the packet exceeded MTU and the DF bit was set.
• Q (Source Quench): An ICMP Type 4 message was received requesting the sender slow down transmission (deprecated in modern networks).
• & (TTL expired): An ICMP Type 11 message was received indicating the packet's TTL expired in transit before reaching the target.
• ? (Unknown packet type): Malformed or unrecognized packet response.

Data Pattern Bit-Stressing

Serial lines, microwave links, and older physical PHY transceivers rely on clock recovery derived from bit transitions. Extended ping allows network engineers to specify custom 16-bit hexadecimal payload patterns:

• 0x0000 (All Zeros) / 0xFFFF (All Ones): Tests line coding and zero/one density enforcement (such as B8ZS or AMI).
• 0x5555 (0101010101010101 in binary): Generates alternating bit transitions to stress clock synchronization circuits.
• 0xAAAA (1010101010101010 in binary): Complements alternating bit patterns to isolate physical transceiver framing skew.

3. Traceroute Internal Mechanics: UDP vs ICMP vs TCP

Traceroute maps the Layer 3 hop-by-hop forwarding path toward a destination by systematically exploiting the Time to Live (TTL) field in the IPv4 header (or Hop Limit in IPv6).

+-----------------------------------------------------------------------------------+
|                         TRACEROUTE OPERATIONAL EXECUTION                          |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|  Source Router                                                                    |
|  [ 10.1.1.1 ]                                                                     |
|       |                                                                           |
|       |-- Probe 1: UDP Port 33434, TTL=1 ----> [ Router R1 (10.1.1.2) ]           |
|       |                                        - Decrements TTL: 1 -> 0           |
|       |                                        - Discards packet                  |
|       |<-- ICMP Type 11 Code 0 (Time Exceeded) | (Returns R1 IP: 10.1.1.2)        |
|       |                                                                           |
|       |-- Probe 2: UDP Port 33435, TTL=2 ----> [ R1 ] ---> [ Router R2 (10.2.2.2) ]|
|       |                                                    - Decrements TTL: 1->0 |
|       |                                                    - Discards packet      |
|       |<-- ICMP Type 11 Code 0 (Time Exceeded) ----------- | (Returns R2 IP)      |
|       |                                                                           |
|       |-- Probe 3: UDP Port 33436, TTL=3 ----> [ R1 ] ---> [ R2 ] ---> [ Target ] |
|       |                                                               [10.3.3.3]  |
|       |                                                               - Port 33436|
|       |                                                                 is closed!|
|       |<-- ICMP Type 3 Code 3 (Port Unreachable) ---------------------| (Stops)   |
+-----------------------------------------------------------------------------------+

Comparison of Traceroute Implementations

1. Cisco IOS-XE / Unix / Linux Traceroute (UDP-Based):
   • Transmits UDP datagrams targeting high, invalid destination ports starting at UDP 33434 and increments the port for each subsequent probe (33435, 33436...).
   • Starts with $\text{TTL}=1$. Intermediate routers decrement TTL to 0, drop the datagram, and return ICMP Type 11 Code 0 (Time Exceeded).
   • When the probe reaches the destination host, the host inspects the high UDP port. Because no application is listening on port 33434+, the destination returns an ICMP Type 3 Code 3 (Port Unreachable).
   • Receipt of ICMP Type 3 Code 3 signals to the traceroute utility that the destination has been reached, terminating the probe sequence.
2. Microsoft Windows Tracert (ICMP-Based):
   • Transmits ICMP Echo Requests (Type 8 Code 0) with incrementing TTL values instead of UDP.
   • Intermediate hops return ICMP Type 11 Code 0 (Time Exceeded).
   • The target destination returns an ICMP Echo Reply (Type 0 Code 0), terminating the trace.
3. TCP Traceroute (tcptraceroute / traceroute tcp):
   • Transmits TCP SYN packets targeting well-known open services (such as TCP port 80 or 443).
   • Used when stateful firewalls or intermediate ACLs silently drop UDP and ICMP diagnostic traffic.
   • Intermediate routers still decrement TTL and return ICMP Type 11 Code 0. When reaching the target, the target returns a TCP SYN-ACK or TCP RST, confirming end-to-end reachability through the firewall.

Traceroute Response Symbol Decoder

4. Path MTU Discovery (PMTUD), MTU Black Holes & TCP MSS Clamping

Standard Ethernet frames carry an IP Maximum Transmission Unit (MTU) of 1500 bytes. However, enterprise overlay networks, WAN tunnels, and encryption protocols add encapsulation headers that reduce the effective payload MTU available for transit packets.

Encapsulation Overhead Breakdown

+-----------------------------------------------------------------------------------+
|                         ENCAPSULATION OVERHEAD COMPARISON                         |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|  Encapsulation Protocol     Header Overhead      Resulting Max IP Payload (MTU)   |
|  ------------------------   -----------------    ------------------------------   |
|  Standard Ethernet          0 Bytes              1500 Bytes                       |
|  802.1Q VLAN Tagging        4 Bytes              1500 Bytes (1504B L2 Frame)      |
|  MPLS Unicast Tag           4 Bytes / Label      1500 Bytes (1504B-1508B L2 Frame)|
|  PPPoE                      8 Bytes              1492 Bytes                       |
|  GRE Tunnel                 24 Bytes             1476 Bytes                       |
|  VXLAN (Underlay UDP)       50 Bytes             1450 Bytes (Requires Jumbo 1550B)|
|  IPsec ESP (Transport)      ~38-52 Bytes         ~1448-1462 Bytes                 |
|  IPsec ESP (Tunnel Mode)    ~50-73+ Bytes        ~1427-1450 Bytes                 |
|  GRE over IPsec             ~74-97+ Bytes        ~1403-1426 Bytes                 |
+-----------------------------------------------------------------------------------+

The Mechanics of Path MTU Discovery (RFC 1191)

Path MTU Discovery (PMTUD) is an automated mechanism defined in RFC 1191 (IPv4) and RFC 8201 (IPv6) that determines the smallest MTU among all links on the path between source and destination:

1. The sending host sets the Don't Fragment (DF) bit (bit 1 of the 3-bit Flags field in the IPv4 header) on all outgoing datagrams.
2. If a router along the path receives a packet whose size exceeds the MTU of the egress interface, the router cannot fragment the packet because $\text{DF}=1$.
3. The router discards the packet and sends an ICMP Type 3 Code 4 (Destination Unreachable: Fragmentation Needed and DF Set) message back to the source IP address.
4. Modern routers compliant with RFC 1191 include the Next-Hop MTU value in the low-order 16 bits of the ICMP header.
5. Upon receiving the ICMP Type 3 Code 4 message, the source host updates its internal Path MTU routing cache for that destination and retransmits the data using packets equal to or smaller than the reported Next-Hop MTU.

MTU Black Hole Failures

An MTU Black Hole occurs when an intermediate security appliance, firewall, or ISP edge router drops ICMP Type 3 Code 4 messages (often due to misconfigured security policies that indiscriminately filter all ICMP messages).

+-----------------------------------------------------------------------------------+
|                             MTU BLACK HOLE FAILURE MODE                           |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|  Host A                               Tunnel Router R1                Host B      |
|  [10.1.1.1]                            [MTU = 1400]                  [10.2.2.2]   |
|      |                                      |                            |        |
|      |-- TCP 3-Way Handshake (SYN/ACK) ---->|===========================>|        |
|      |   (Small packets ~60 bytes, DF=1)    |   (Handshake Succeeds!)    |        |
|      |<-- Handshake Complete ---------------|<===========================|        |
|      |                                      |                            |        |
|      |-- Large TLS / HTTP Data (1500B, DF=1)|                            |        |
|      |   ==================================>| (Exceeds MTU 1400B)        |        |
|      |                                      | - Packet Dropped!          |        |
|      |                                      | - Sends ICMP Type 3 Code 4 |        |
|      |                                      |   ====X (Dropped by ISP!)  |        |
|      |                                                                            |
|      | * Host A NEVER receives ICMP error. Retransmits 1500B packet indefinitely. |
|      | * SYMPTOM: SSH/Telnet connects, but HTTP/HTTPS page loads and SCP hang!    |
+-----------------------------------------------------------------------------------+

TCP MSS Clamping (ip tcp adjust-mss)

To permanently eliminate MTU black holes caused by blocked ICMP messages without requiring end-host reconfiguration, routers at the tunnel boundary perform TCP Maximum Segment Size (MSS) Clamping via the ip tcp adjust-mss interface configuration command.

During the initial TCP three-way handshake, hosts advertise their Maximum Segment Size in the TCP Options field of SYN and SYN-ACK packets. The relationship between IP MTU and TCP MSS is: $\text{TCP MSS} = \text{IP MTU} - (\text{IP Header [20 bytes]} + \text{TCP Header [20 bytes]}) = \text{IP MTU} - 40$

For IPv6 (where base IP header is 40 bytes): $\text{TCP MSS (IPv6)} = \text{IPv6 MTU} - (\text{IPv6 Header [40 bytes]} + \text{TCP Header [20 bytes]}) = \text{IPv6 MTU} - 60$

When ip tcp adjust-mss <bytes> is configured on an interface (such as a GRE tunnel or WAN interface), the router intercepts transit TCP SYN and SYN-ACK packets. If the host's advertised MSS is larger than the configured adjust-mss value, the router rewrites the MSS field in the TCP header down to the configured value and recalculates the TCP checksum on the fly. Both endpoints are forced to segment their data payloads to fit cleanly within the tunnel MTU, preventing fragmentation and avoiding ICMP dependencies.

• Standard GRE Tunnel (MTU 1476): $\text{MSS} = 1476 - 40 = \mathbf{1436}$ (ip tcp adjust-mss 1436)
• IPsec / GRE over IPsec Tunnel (MTU 1400): $\text{MSS} = 1400 - 40 = \mathbf{1360}$ (ip tcp adjust-mss 1360)

5. Cisco IOS-XE Diagnostic CLI Configuration & Verification

1. Extended Ping CLI Execution Examples

Router-1# ping
Protocol [ip]: 
Target IP address: 198.51.100.10
Repeat count [5]: 50
Datagram size [100]: 1450
Timeout in seconds [2]: 1
Extended commands [n]: y
Source address or interface: Loopback0
Type of service [0]: 184
Set DF bit in IP header? [no]: yes
Validate reply data? [no]: yes
Data pattern [0xABCD]: 0x5555
Loose, Strict, Record, Timestamp, Verbose[none]: 
Sweep range of sizes [n]: y
Sweep min size [36]: 1400
Sweep max size [18024]: 1500
Sweep interval [1]: 20
Trying 198.51.100.10 [1400..1500], sending 50, [1400..1500]-byte ICMP Echos to 198.51.100.10, timeout is 1 seconds:
Packet sent with the /Set DF bit
!!!!!!!!!!!!!!!!!!!!MMMMMMMMMM
Success rate is 66 percent (20/30), round-trip min/avg/max = 4/8/15 ms

2. Advanced Traceroute CLI Execution

Router-1# traceroute 198.51.100.10 source loopback 0 numeric ttl 1 10 probe 2 port 33434
Type escape sequence to abort.
Tracing the route to 198.51.100.10
VRF info: (none)
  1 10.1.12.2 2 msec 1 msec
  2 10.1.23.3 4 msec 3 msec
  3 10.1.34.4 8 msec 6 msec
  4 * *
  5 198.51.100.10 12 msec 11 msec

3. Configuring Interface MTU, IP MTU, and TCP MSS Clamping

! Configure physical WAN Interface MTU
interface GigabitEthernet0/0/0
 description WAN-Uplink-to-ISP
 mtu 1500
 ip address 203.0.113.2 255.255.255.252
!
! Configure GRE Tunnel Interface with MTU and MSS Clamping
interface Tunnel100
 description DMVPN-Overlay-to-DataCenter
 ip address 10.255.0.1 255.255.255.0
 tunnel source GigabitEthernet0/0/0
 tunnel destination 198.51.100.1
 !
 ! Set Layer 3 MTU to 1400 to account for GRE + IPsec overhead
 ip mtu 1400
 !
 ! Clamp IPv4 TCP MSS to 1360 (1400 - 40 bytes IP/TCP header)
 ip tcp adjust-mss 1360
 !
 ! Clamp IPv6 TCP MSS to 1340 (1400 - 60 bytes IPv6/TCP header)
 ipv6 tcp adjust-mss 1340

4. Verification and Diagnostic Inspection Commands

Router-1# show ip interface Tunnel100
Tunnel100 is up, line protocol is up
  Internet address is 10.255.0.1/24
  Broadcast address is 255.255.255.255
  Address determined by setup command
  MTU is 1400 bytes
  Helper address is not set
  Directed broadcast forwarding is disabled
  Outgoing access list is not set
  Inbound  access list is not set
  Proxy ARP is enabled
  Local Proxy ARP is disabled
  Security level is default
  Split horizon is enabled
  ICMP redirects are never sent
  ICMP unreachables are always sent
  ICMP mask replies are never sent
  IP fast switching is enabled
  IP Flow subblocks initialization is done
  TCP subblocks initialization is done
  TCP MSS Clamping is enabled (adjust-mss 1360)

6. Conditional Debugging

ENCOR v1.2 topic 4.1 names its tools explicitly: "Diagnose network problems using such as debugs, conditional debugs, traceroute, ping, SNMP, and syslog." Conditional debugging earns its own mention because an unconditioned debug on a production device is one of the fastest ways to cause an outage.

6.1 Why Unconditioned Debugs Are Dangerous

A debug is process-switched output generated by the CPU. On a busy router, debug ip packet with no filter can generate tens of thousands of messages per second, and if console logging is enabled each one is written synchronously out a 9600-baud UART — the exact CPU-starvation failure analysed in Section 14.2. The device becomes unreachable, and you cannot type undebug all because the console is saturated.

Three rules before any debug on a production device:

1. Disable console logging (no logging console) and read from the buffer (logging buffered 8192 debugging) or a monitored VTY (terminal monitor).
2. Scope the debug with a condition or an ACL. Never run a bare debug ip packet.
3. Know your escape. undebug all (u all) disables every debug; no debug condition all clears every condition.

6.2 The debug condition Family

debug condition installs a global filter that applies to all subsequent debug output. Set the condition first, then enable the debug.

! Restrict all subsequent debug output to one interface
Router# debug condition interface GigabitEthernet0/0/1
Condition 1 set

! Or restrict to traffic involving one IP address
Router# debug condition ip 10.20.30.40
Condition 2 set

! Now enable the debug - output appears only for the matching condition
Router# debug ip ospf adj

! Inspect and clear
Router# show debug condition
Condition 1: interface Gi0/0/1 (1 flags triggered)
Condition 2: ip 10.20.30.40 (1 flags triggered)
Router# no debug condition all
Router# undebug all

The behaviour that trips candidates up: multiple conditions are combined with OR, not AND. Setting debug condition interface Gi0/0/1 and debug condition ip 10.20.30.40 produces output for traffic on Gi0/0/1 or involving 10.20.30.40, not only traffic that satisfies both. If you need a genuine AND, use an ACL-filtered debug instead.

6.3 ACL-Filtered Packet Debugging

access-list 150 permit ip host 10.20.30.40 host 172.16.9.9
access-list 150 permit ip host 172.16.9.9 host 10.20.30.40
!
Router# debug ip packet 150 detail

Listing both directions matters — a one-way ACL shows you only half the conversation and makes a working return path look like a failure.

The critical caveat: debug ip packet displays only process-switched packets. Traffic forwarded by CEF in the fast path never reaches the CPU, so it never appears in the output. A silent, empty debug ip packet therefore usually means the traffic is being forwarded correctly in hardware, not that it is absent. Disabling CEF to force packets into the process path is a genuine outage on a production device and must never be done to satisfy a debug.

6.4 The IOS-XE Answer: Platform Conditional Debug and Packet Trace

Because of the CEF limitation, modern IOS-XE platforms expose a conditional-debug framework that inspects the data plane without punting traffic to the CPU. This is the correct tool on a Catalyst 8000 or ASR 1000.

! 1. Define the condition
Router# debug platform condition interface GigabitEthernet0/0/1 both
Router# debug platform condition start

! 2. Enable packet tracing through the Forwarding Information Agent
Router# debug platform packet-trace packet 128 fia-trace

! 3. Reproduce the problem, then stop and read
Router# debug platform condition stop
Router# show platform packet-trace summary
Pkt  Input            Output           State  Reason
0    Gi0/0/1          Gi0/0/2          FWD
1    Gi0/0/1          internal0/0/rp:0 PUNT   11  (For-us data)
2    Gi0/0/1          Gi0/0/2          DROP   8   (Ipv4Acl)

Router# show platform packet-trace packet 2
Router# clear platform packet-trace statistics

show platform packet-trace summary gives the single most valuable answer in WAN troubleshooting: was the packet forwarded, punted, or dropped — and by which feature. A DROP with reason Ipv4Acl ends an ACL argument immediately, and a PUNT reason explains CPU spikes without guesswork.

Exam framing: if a stem describes needing to see the fate of specific traffic on a production router without impacting other flows, the answer is a condition-scoped debug or packet-trace, never a bare debug ip packet.