10.3 Systematic Troubleshooting & Embedded Packet Capture (EPC)

10.3 Systematic Troubleshooting & Embedded Packet Capture (EPC)

Core Blueprint Focus: When an enterprise user cannot connect to a corporate wireless network, blaming the Wi-Fi infrastructure is common. However, wireless onboarding is a complex, multi-protocol transaction spanning 802.11 radio negotiation, 802.1X/RADIUS authentication, DHCP IP acquisition, DNS name resolution, and Layer 3 gateway routing. Domain 6.0 of the Cisco WLCOR 350-101 exam requires network engineers to isolate faults methodically using a structured 5-stage troubleshooting framework, master IEEE 802.11 Reason Codes and Status Codes, and leverage Embedded Packet Capture (EPC) directly on the Catalyst 9800.

---

1. The Structured 5-Stage Client Connection Lifecycle

Every wireless connection traverses five distinct, sequential phases. A failure at any individual stage halts the onboarding pipeline. Troubleshooting without identifying the exact failure stage leads to random, ineffective configuration changes.

+-----------------------------------------------------------------------------------------+
|                    STRUCTURED 5-STAGE WIRELESS ONBOARDING LIFECYCLE                     |
|                                                                                         |
|  +-----------------------------------------------------------------------------------+  |
|  | STAGE 1: 802.11 ASSOCIATION (Physical & Radio Layer)                              |  |
|  | Frames: Probe Req/Resp -> 802.11 Open Auth Req/Resp -> Assoc Req/Resp             |  |
|  | Common Failures: PMF policy mismatch, capability mismatch, AP capacity limits       |  |
|  +-----------------------------------------------------------------------------------+  |
|                                            |                                            |
|                                            v                                            |
|  +-----------------------------------------------------------------------------------+  |
|  | STAGE 2: 802.1X / SECURITY (Identity & Cryptographic Keys)                         |  |
|  | Frames: EAPOL-Start -> EAP-Identity -> RADIUS Access-Request -> TLS -> 4-Way M1-M4|  |
|  | Common Failures: RADIUS timeout, bad server certificate, expired cert, bad creds |  |
|  +-----------------------------------------------------------------------------------+  |
|                                            |                                            |
|                                            v                                            |
|  +-----------------------------------------------------------------------------------+  |
|  | STAGE 3: DHCP IP ADDRESSING (Network Layer Bootstrap)                             |  |
|  | Frames: DHCP Discover -> DHCP Offer -> DHCP Request -> DHCP ACK (DORA)            |  |
|  | Common Failures: Scope exhausted, missing ip helper, VLAN trunk pruned, APIPA IP  |  |
|  +-----------------------------------------------------------------------------------+  |
|                                            |                                            |
|                                            v                                            |
|  +-----------------------------------------------------------------------------------+  |
|  | STAGE 4: DNS RESOLUTION (Service Discovery)                                       |  |
|  | Frames: DNS Query (UDP 53) -> DNS Response                                        |  |
|  | Common Failures: Unreachable DNS server, missing DNS IP in DHCP, DNS ACL drop     |  |
|  +-----------------------------------------------------------------------------------+  |
|                                            |                                            |
|                                            v                                            |
|  +-----------------------------------------------------------------------------------+  |
|  | STAGE 5: DEFAULT GATEWAY & DATA PATH (End-to-End Reachability)                    |  |
|  | Frames: ARP Request/Reply -> ICMP / TCP SYN                                       |  |
|  | Common Failures: ARP proxy failure, duplicate IP (SISF), Client Exclusion list    |  |
|  +-----------------------------------------------------------------------------------+  |
+-----------------------------------------------------------------------------------------+

Stage 1: 802.11 Association Phase

The client and Access Point negotiate physical and radio capabilities over the air:

• Diagnostic Indicators: Client does not appear in show wireless client summary, or appears momentarily and vanishes.
• Root Causes:
  • 802.11w PMF Mismatch: If the WLAN requires Protected Management Frames (security pmf mandatory) but the client does not advertise the required capability, the controller rejects association for the robust-management-frame policy. Decode the status from the capture instead of assigning an incorrect fixed 0x001d label.
  • Cipher Suite Mismatch: WPA2 (AES-CCMP) vs WPA3 (SAE / GCMP-256) incompatibility.
  • Low RSSI Suppression: The AP is configured with an association RSSI threshold (e.g., rejecting frames weaker than $-75\text{ dBm}$ to prevent sticky clients).

Stage 2: 802.1X / Security Negotiation Phase

The client negotiates identity and authentication credentials via the Extensible Authentication Protocol (EAP):

• Diagnostic Indicators: Client displays State: Authenticating in show wireless client detail.
• Root Causes:
  • RADIUS Server Timeout: Controller cannot reach Cisco ISE over UDP ports 1812/1813 due to routing failure, firewall drops, or a mismatched RADIUS shared secret key.
  • Untrusted Server Certificate: The RADIUS server presents an X.509 certificate signed by an enterprise CA that is not installed in the client's trusted root store.
  • NTP / Clock Desynchronization: If the Catalyst 9800 or Cisco ISE clocks are out of synchronization by more than a few minutes, certificate validation fails immediately.
  • EAPOL-Key M3 Timeout: The AP transmits Message 3 of the 4-way handshake, but the client fails to return Message 4 within the timeout window (often caused by client NIC driver bugs), triggering Reason Code 15.

Stage 3: DHCP IP Address Acquisition Phase

Once Layer 2 security is established, the client initiates the Dynamic Host Configuration Protocol DORA exchange:

• Diagnostic Indicators: Client displays State: Associated or State: UP, but displays an IPv4 address of 0.0.0.0 or a link-local Automatic Private IP Addressing (APIPA) address (169.254.x.x).
• Root Causes:
  • DHCP Scope Exhaustion: The IP pool for the client VLAN on the DHCP server has run out of available leases.
  • Missing ip helper-address: The Layer 3 default gateway for the wireless VLAN lacks an ip helper-address <DHCP_IP> command to convert client broadcast Discovers into unicast relay packets.
  • VLAN Trunk Pruning: In FlexConnect local switching, the client VLAN is omitted from the switch trunk port connecting to the AP. In central switching, the client VLAN is missing or pruned from the trunk interface connecting to the Catalyst 9800.
  • Option 82 Mismatch: The controller injects DHCP Option 82 (circuit ID / remote ID) sub-options, but the upstream DHCP server is configured to reject Option 82 from untrusted relay agents.

Stage 4: DNS Name Resolution Phase

The client attempts to resolve corporate hostnames and external fully qualified domain names (FQDNs):

• Diagnostic Indicators: Client can ping its default gateway and direct IP addresses (e.g., 8.8.8.8), but cannot browse intranet URLs or access web applications.
• Root Causes:
  • DHCP server passed invalid or unreachable DNS server IP addresses.
  • Access Control Lists (ACLs) on the WLC or firewall block outbound UDP/TCP port 53 traffic.

Stage 5: Default Gateway & Application Reachability Phase

The client attempts to establish Layer 3 reachability to its default gateway and enterprise servers:

• Diagnostic Indicators: Client has a valid IP address, but cannot ping its local gateway; ARP requests go unanswered.
• Root Causes:
  • Client Exclusion List (Blacklisting): If a client fails 802.1X authentication multiple times consecutively (default 3 attempts), the Catalyst 9800 blacklists the client MAC address for a configurable period (default 60 seconds), dropping all subsequent frames.
  • IP Device Tracking (SISF) Duplicate IP Collision: If another device responds to an ARP probe for the client's assigned IP address, the controller's Switch Integrated Security Features drop the client session.
  • Upstream Layer 2 Loop / Spanning Tree Inconsistency: Upstream switch blocks the wireless VLAN due to BPDU guard or root guard triggers.

---

2. IEEE 802.11 Reason Codes vs. Status Codes

When diagnosing client disconnections or association rejections via packet captures or controller logs, network engineers must differentiate between Reason Codes and Status Codes:

• Reason Codes: Indicate why an established 802.11 association or authentication was terminated. They are carried exclusively inside Disassociation and Deauthentication management frames. Either the AP or the client station can transmit a Reason Code.
• Status Codes: Indicate the outcome (success or specific reason for failure) of a connection request. They are carried exclusively inside Authentication Response and Association / Reassociation Response management frames. Status Codes are transmitted exclusively by the Access Point (authenticator).

+-----------------------------------------------------------------------------------------+
|                     802.11 REASON CODES VS STATUS CODES SUMMARY                         |
|                                                                                         |
|  REASON CODES (Disconnection / Teardown)     STATUS CODES (Request Outcome / Admission) |
|  Carried in: Deauth / Disassoc frames        Carried in: Assoc / Auth Response frames   |
|  Sent by: AP or Client Station               Sent by: Access Point ONLY                 |
|                                                                                         |
|  Code 1:  Unspecified reason                 Code 0:  Successful                        |
|  Code 2:  Previous auth no longer valid      Code 1:  Unspecified failure               |
|  Code 3:  Deauth: STA leaving BSS            Code 10: Cannot support requested caps     |
|  Code 4:  Disassoc due to inactivity         Code 12: Assoc denied: outside standard    |
|  Code 6:  Class 2 frame from nonauth STA     Code 17: Assoc denied: AP out of resources |
|  Code 7:  Class 3 frame from nonassoc STA    Code 18: Assoc denied: unsupported rates   |
|  Example: code 17 = AP capacity; PMF mismatch = robust-management policy violation     |
|  Code 16: Group Key Handshake timeout        Code 34: Assoc denied: poor RSSI / QoS     |
|  Code 23: IEEE 802.1X auth failed                                                       |
+-----------------------------------------------------------------------------------------+

Key IEEE 802.11 Reason Codes in Detail

• Reason Code 1 (Unspecified Reason): Generic disconnection. Frequently generated when an AP is rebooted or administratively reset.
• Reason Code 2 (Previous Authentication No Longer Valid): The AP received a frame from a client whose security association expired or was purged from the controller's active table.
• Reason Code 4 (Disassociated Due to Inactivity): The client stopped transmitting data frames for longer than the AP's configured client idle timeout (default 300 seconds), prompting the AP to tear down the session to reclaim memory.
• Reason Code 6 (Class 2 Frame Received from Nonauthenticated STA): The AP received an Association Request or Reassociation Request before the client completed 802.11 Open Authentication.
• Reason Code 7 (Class 3 Frame Received from Nonassociated STA): The AP received data frames from a client that had not completed Layer 2 association.
• Reason Code 15 (4-Way Handshake Timeout): The AP transmitted Message 1 or Message 3 of the WPA2/WPA3 4-way handshake, but the client failed to return Message 2 or Message 4 within the timeout window (typically 1,000 ms, repeated 3 times). This is the hallmark signature of client driver incompatibility, wrong PSK pre-shared key, or severe RF packet corruption.
• Reason Code 16 (Group Key Handshake Timeout): The client failed to acknowledge the AP's Group Temporal Key (GTK) rotation frame.
• Reason Code 23 (IEEE 802.1X Authentication Failed): The RADIUS server returned an Access-Reject, or the client failed certificate validation.

Key IEEE 802.11 Status Codes in Detail

• Status Code 0 (Successful): The association or authentication request was accepted without error.
• Status Code 17 (Association Denied: AP Lacks Resources / Capacity): The AP rejected the client association because its internal memory tables, CPU capacity, or maximum client limit (e.g., max 200 clients per radio) was reached. In addition, when WMM Admission Control (WMM-AC) is active and the Available Admission Capacity (AAC) is exhausted, new voice calls are rejected with Status Code 17.
• Status Code 18 (Association Denied: Unsupported Data Rates): The client station's association request did not include support for all mandatory (basic) operational data rates advertised in the AP's beacon.
• PMF policy rejection: When a WLAN requires IEEE 802.11w PMF and a client lacks the required capability, association is rejected for robust-management-policy incompatibility. Use Wireshark’s decoded status and the applicable standard/release documentation; hexadecimal 0x001d must not be taught as the universal “PMF required” status.

---

3. Embedded Packet Capture (EPC) on Catalyst 9800

Embedded Packet Capture (EPC) provides hardware-accelerated wire-level packet sniffing natively on the Catalyst 9800. Rather than deploying physical network TAPs or configuring switchport analyzer (SPAN) sessions, EPC captures packets traversing controller interfaces and stores them in local memory buffers.

EPC Capture Points on Catalyst 9800

• Physical Uplink Interfaces (e.g., GigabitEthernet0/0/1): Captures traffic traversing the wired network, including outer CAPWAP control/data packets, RADIUS UDP exchanges, DHCP relay packets, and inter-WLC mobility tunnels.
• Wireless Management Interface (WMI): Captures control-plane traffic terminating directly on the controller.
• Client MAC Filter: Intercepts traffic to and from a specific wireless station across the control and data paths.

Buffer Management & Sizing

EPC captures frames into local RAM buffers configured as either:

• Linear Buffer: Captures packets sequentially until the allocated buffer size is exhausted, then halts capture automatically.
• Circular Buffer: Continuously writes packets; when full, it overwrites the oldest frames, retaining the most recent network activity.

Step-by-Step EPC Configuration, Verification & Wireshark Export

! =====================================================================
! Step 1: Define an Access List to isolate specific diagnostic traffic
! =====================================================================
C9800(config)# ip access-list extended ACL-EPC-DIAG
C9800(config-ext-nacl)# permit udp any any eq 1812     ! RADIUS Authentication
C9800(config-ext-nacl)# permit udp any any eq 1813     ! RADIUS Accounting
C9800(config-ext-nacl)# permit udp any any eq 67       ! DHCP Server Relay
C9800(config-ext-nacl)# permit udp any any eq 68       ! DHCP Client
C9800(config-ext-nacl)# permit udp any any eq 5246     ! CAPWAP Control
C9800(config-ext-nacl)# permit ip host 10.10.40.115 any ! Specific client IP
C9800(config-ext-nacl)# exit

! =====================================================================
! Step 2: Define the capture point on the physical uplink interface
! =====================================================================
C9800# monitor capture CAP-WIFI interface GigabitEthernet0/0/1 both

! =====================================================================
! Step 3: Associate the filter, set buffer size, and packet limits
! =====================================================================
C9800# monitor capture CAP-WIFI access-list ACL-EPC-DIAG
C9800# monitor capture CAP-WIFI buffer size 50
C9800# monitor capture CAP-WIFI limit packet-count 10000
C9800# monitor capture CAP-WIFI buffer circular

! =====================================================================
! Step 4: Start the capture
! =====================================================================
C9800# monitor capture CAP-WIFI start
Starting the capture process for CAP-WIFI

! =====================================================================
! Step 5: Reproduce the client failure, then stop the capture
! =====================================================================
C9800# monitor capture CAP-WIFI stop
Capture process stopped for CAP-WIFI

! =====================================================================
! Step 6: Live buffer inspection directly on the controller CLI
! =====================================================================
C9800# show monitor capture CAP-WIFI buffer brief
C9800# show monitor capture CAP-WIFI buffer dump

! =====================================================================
! Step 7: Export capture file off-box for deep Wireshark decoding
! =====================================================================
C9800# monitor capture CAP-WIFI export tftp://10.1.1.50/captures/onboarding_issue.pcap
! Or export via SFTP:
! C9800# monitor capture CAP-WIFI export sftp://admin:pass@10.1.1.50/onboarding.pcap

! =====================================================================
! Step 8: Remove the capture point when complete
! =====================================================================
C9800# no monitor capture CAP-WIFI

5-Stage Wireless Troubleshooting Reference Matrix