10.2 Operational Troubleshooting CLI Commands & Packet Diagnostics

10.2 Operational Troubleshooting CLI Commands & Packet Diagnostics

Operational Troubleshooting Architecture

Troubleshooting complex traffic drops, state machine discrepancies, and NAT translation failures on high-throughput enterprise firewalls requires surgical precision. Because PAN-OS firewalls process hundreds of thousands of concurrent connections at wire speed, attempting broad packet captures or unbounded diagnostic logging on production hardware risks saturating management CPU queues, consuming packet buffers, and destabilizing active sessions.

To provide safe, line-rate visibility, PAN-OS provides the Dataplane Packet Diagnostic Framework (debug dataplane packet-diag). This framework executes directly within the Data Plane subsystem, using hardware-enforced Packet Filters to isolate traffic before passing targeted frames to the diagnostic logging daemon (pan_packet_diag.log) or the multi-stage Packet Capture (PCAP) engine.

                                [ Ingress Packet Stream ]
                                            |
                                            v
                     +----------------------------------------------+
                     |     Hardware Packet Filter Engine (1-4)      |
                     |  - Matches Source / Dest IP & Ports          |
                     |  - Matches Protocol & Ingress Interface      |
                     +----------------------------------------------+
                                    |                |
                             [ No Match ]       [ Match! ]
                                    |                |
                                    v                +------------------------------+
                            (Normal Forwarding)      |                              |
                                                     v                              v
                                         +-----------------------+      +-----------------------+
                                         |   Flow Basic Engine   |      |  4-Stage PCAP Engine  |
                                         | - Traces Session Setup|      | - rx (Driver Ingress) |
                                         | - Traces Route & NAT  |      | - firewall (Session)  |
                                         | - Traces Policy Match |      | - transmit (Egress)   |
                                         | - Traces App-ID Shift |      | - drop (Discards)     |
                                         +-----------------------+      +-----------------------+
                                                     |                              |
                                                     v                              v
                                         [ pan_packet_diag.log ]        [ Local PCAP Buffers ]
                                         (View via CLI Console)         (Export via SCP/Web UI)

---

The Advanced Packet Diagnostic Framework (debug dataplane packet-diag)

The packet diagnostic framework operates through a rigorous, three-step operational workflow: Define Filters, Enable Logging or Captures, and Analyze Output.

Step 1: Configuring Hardware Packet Filters

Before enabling any diagnostic logging or packet capture, an administrator MUST define packet filters. A packet filter instructs the Data Plane Network Processing Engine (NPE) to intercept only traffic that matches specific Layer 3 and Layer 4 criteria. Up to four independent filter indexes (Index 1 through 4) can be active simultaneously:

# Define Filter Index 1 matching specific source, destination, and port
admin@PA-3410> debug dataplane packet-diag set filter match source 10.1.1.50 destination 198.51.100.25 destination-port 443 protocol 6

# Optionally define Filter Index 2 matching reverse return traffic
admin@PA-3410> debug dataplane packet-diag set filter match source 198.51.100.25 destination 10.1.1.50 source-port 443 protocol 6

# Enable the configured packet filters
admin@PA-3410> debug dataplane packet-diag set filter on

# Verify active filter configuration and operational status
admin@PA-3410> debug dataplane packet-diag show setting

Exam Trap Alert: Defining a packet filter does not automatically activate it! The filter only takes effect after explicitly executing debug dataplane packet-diag set filter on. If an administrator skips this command, pan_packet_diag.log will record zero events, and packet captures will generate empty zero-byte files.

Step 2: Flow Basic Debugging (flow basic)

Flow Basic is the most powerful diagnostic logging tool in PAN-OS. It produces cycle-by-cycle logs tracing packet traversal through each phase of the 6-stage lifecycle: ingress validation, session matching, route lookup, NAT translation, security policy evaluation, App-ID classification, Content-ID profile attachment, and egress transmission.

# Enable Flow Basic diagnostic logging
admin@PA-3410> debug dataplane packet-diag set log feature flow basic

# Activate dataplane logging
admin@PA-3410> debug dataplane packet-diag set log on

# View the diagnostic log in real time as traffic flows
admin@PA-3410> tail follow yes mp-log pan_packet_diag.log

# Alternatively, view the entire captured log buffer with search filters
admin@PA-3410> view-log dataplane-log pan_packet_diag.log

Deconstructing a Real Flow Basic Log Trace

Below is an annotated excerpt from pan_packet_diag.log tracing a newly initiated connection:

== 2026/09/02 12:40:15:234 ==
Packet received at ingress stage, interface ethernet1/1, vsys 1
Packet info: len 64, 10.1.1.50:51240 -> 198.51.100.25:443, proto 6 (TCP SYN)
Flow lookup: session not found, entering slow path for session setup
Zone protection check: zone 'Trust', flood checks passed, syn-cookie inactive
Route lookup: dest 198.51.100.25, nexthop 203.0.113.1, egress interface ethernet1/2, egress zone 'Untrust'
NAT policy lookup: matched rule 'Corp-Outbound-NAT', type DIPP, translated source 203.0.113.5:14208
Security policy lookup: from 'Trust' to 'Untrust', pre-nat-ip 198.51.100.25, proto 6, port 443
Security policy match: matched rule 'Allow-Internet-HTTPS', action ALLOW
Allocated session: id 104523, c2s flow (10.1.1.50:51240 -> 198.51.100.25:443), s2c flow (198.51.100.25:443 -> 203.0.113.5:14208)
App-ID: initial protocol match 'ssl', service 'service-https'
Forwarding packet to egress stage, interface ethernet1/2

If the connection were dropped, the trace would explicitly identify the terminating stage:

• Security policy match: matched rule 'inter-zone-default', action DENY, dropping packet
• Route lookup failed: no route to host 198.51.100.25, drop packet
• Zone protection violation: SYN flood rate exceeded, dropping packet

Step 3: Deactivating and Clearing Diagnostic Settings

Always clean up diagnostic features immediately after troubleshooting to preserve storage and processing capacity:

# Deactivate logging and packet filters
admin@PA-3410> debug dataplane packet-diag set log off
admin@PA-3410> debug dataplane packet-diag set filter off

# Clear active filter configurations and delete existing log files
admin@PA-3410> debug dataplane packet-diag clear filter all
admin@PA-3410> debug dataplane packet-diag clear log log-filter

---

The Dataplane Packet Capture (PCAP) Engine

While management interface packet captures (tcpdump) capture only administrative traffic on the MGT port, the PAN-OS Dataplane Packet Capture Engine captures transit data traffic directly from the fast-path processing hardware. Captures operate across four distinct pipeline stages:

# Configure capture stages, output file names, and byte capture limits
admin@PA-3410> debug dataplane packet-diag set capture stage rx file rx-stage.pcap bytes 1514
admin@PA-3410> debug dataplane packet-diag set capture stage firewall file fw-stage.pcap
admin@PA-3410> debug dataplane packet-diag set capture stage transmit file tx-stage.pcap
admin@PA-3410> debug dataplane packet-diag set capture stage drop file drop-stage.pcap

# Enable the packet capture engine
admin@PA-3410> debug dataplane packet-diag set capture on

# Verify active capture status and packet counts
admin@PA-3410> debug dataplane packet-diag show setting

# Turn off packet capture after reproducing the issue
admin@PA-3410> debug dataplane packet-diag set capture off

Exporting Packet Captures

Once captured, PCAP files reside in the internal dataplane storage buffer. They can be exported via SCP or TFTP for analysis in Wireshark, or downloaded directly via the PAN-OS Web Interface (Monitor > Packet Capture):

# Export capture file to a remote SCP server
admin@PA-3410> scp export capture file drop-stage.pcap to admin@10.10.10.50:/var/pcap/

# Export capture file to a remote TFTP server
admin@PA-3410> tftp export capture file tx-stage.pcap to 10.10.10.50

Exam Trap Alert: The drop packet capture stage captures packets dropped by the PAN-OS dataplane software engines (such as security policy denies, threat prevention blocks, or route lookup failures). It cannot capture packets dropped by physical hardware PHY faults, Ethernet framing errors (FCS/CRC errors), or ASIC ring-buffer overruns before the frame reaches the dataplane software. Those errors must be identified using show interface hardware ethernet1/1.

---

Session Management & Operational Flag Interpretation

The PAN-OS session table provides complete state visibility into every active connection passing through the firewall.

Filtering and Displaying Sessions

# List active sessions matching specific source and destination filters
admin@PA-3410> show session all filter source 10.1.1.50 destination 198.51.100.25 state active

# Filter sessions by specific application
admin@PA-3410> show session all filter application salesforce-base

# Display exhaustive state information for a single session ID
admin@PA-3410> show session id 104523

Dissecting show session id Output

Session           104523
        c2s flow:
                source:      10.1.1.50 [Trust]
                dst:         198.51.100.25
                proto:       6
                sport:       51240           dport:      443
                state:       ACTIVE          type:       FLOW
                src user:    corp\jdoe
        s2c flow:
                source:      198.51.100.25 [Untrust]
                dst:         203.0.113.5
                proto:       6
                sport:       443             dport:      14208
                state:       ACTIVE          type:       FLOW
                src user:    unknown
        start time:                           Wed Sep  2 12:40:15 2026
        timeout:                              3600 sec
        time to live:                         3588 sec
        total byte count(c2s):                8412 bytes
        total byte count(s2c):                24108 bytes
        layer7 processing:                    completed
        url filtering enabled:                True
        session flags:                        strict-ip-check proxy decrypt flow_fwd
        application:                          office365-enterprise
        rule:                                 Allow-Enterprise-SaaS

Critical Operational Session Flags

Modifying Timeouts and Clearing Sessions

# Clear a single problematic session by its ID
admin@PA-3410> clear session id 104523

# Clear all active sessions matching a specific source IP address
admin@PA-3410> clear session all filter source 10.1.1.50

# Clear all sessions matching a specific application
admin@PA-3410> clear session all filter application ms-update

Exam Trap Alert: Executing clear session all without any filter parameters will immediately terminate every active session on the entire firewall, dropping all enterprise user traffic, IPsec VPN tunnels, and administrative sessions! Always append filter parameters when clearing sessions on production appliances.

---

Global Counters Analysis (show counter global filter delta yes)

PAN-OS maintains internal operational counters tracking every packet parsing decision, policy evaluation, error condition, and buffer transition. These counters increment continuously from system boot.

To diagnose transient drops or unexplained packet losses, administrators use Differential Global Counters (filter delta yes). This command establishes a baseline and reports only the counters that incremented during the observation window:

# Step 1: Run the delta command once to establish the initial baseline
admin@PA-3410> show counter global filter delta yes

# Step 2: Reproduce the problematic traffic on the client workstation (e.g., attempt connection)

# Step 3: Run the delta command again filtered by severity 'drop'
admin@PA-3410> show counter global filter delta yes severity drop

# Filter global counters by category aspect (packet, flow, session, system)
admin@PA-3410> show counter global filter delta yes aspect pkt

Interpreting Critical Drop Counters

Global counters:
Elapsed time since last query: 12.450 seconds

name                             value     rate     severity  category  aspect    description
------------------------------------------------------------------------------------------------
flow_policy_deny                 45        3        drop      flow      session   Session denied by security policy
flow_parse_l4_drop               2         0        drop      flow      pkt       Packet dropped: L4 header corrupt or invalid
flow_zone_protect_syn_drop       1200      96       drop      flow      pkt       Packets dropped: Zone protection SYN flood
flow_fwd_no_route                14        1        drop      flow      route     Packets dropped: No route to destination
session_alloc_failure            0         0        drop      flow      session   Session allocation failed: memory limit
buffer_alloc_fail                0         0        drop      sys       buffer    Packet buffer allocation failure

• flow_policy_deny: Indicates packets were blocked by a security policy rule (either an explicit deny rule or the default inter-zone-default rule).
• flow_fwd_no_route: Indicates the routing table (FIB) lacks a valid route to reach the packet's destination IP address. The firewall discards the packet during Stage 2 route lookup.
• flow_zone_protect_syn_drop: Indicates the Zone Protection Profile attached to the ingress zone is actively dropping incoming SYN packets because packet rates exceeded the maximal threshold.
• session_alloc_failure: The firewall has exhausted its maximum licensed concurrent session capacity. New connections are dropped until existing sessions terminate.
• buffer_alloc_fail: Indicates critical global packet buffer exhaustion, typically caused by unmitigated volumetric flood attacks or extreme bandwidth saturation.

---

Diagnostic Framework Comparison