11.4 Troubleshooting Media Quality & Call Failures

11.4 Troubleshooting Media Quality & Call Failures

Diagnosing media impairments in enterprise collaboration networks requires isolating whether a failure originates in signaling negotiation, network routing, firewall inspection, physical transmission, or QoS queuing. Because voice and video separate the control plane (SIP/H.323 signaling) from the data plane (RTP/RTCP media streams), a call may set up successfully with normal ringback and connection states while the media stream fails completely or suffers severe degradation. This section covers structured diagnostic workflows, verification commands, and packet analysis techniques.

1. One-Way Audio Diagnostic Workflows

One-way audio (where Caller A hears Caller B, but Caller B hears silence) is among the most frequent and disruptive collaboration issues. Because RTP media travels over two independent unidirectional UDP streams (one in each direction), any failure affecting only one transmission path causes one-way audio.

+-------------------------------------------------------------------------+
|                        ONE-WAY AUDIO ROOT CAUSES                        |
|                                                                         |
|   1. NAT / PAT Traversal Failure:                                       |
|      SDP c= line advertises unroutable private RFC 1918 IP address.     |
|                                                                         |
|   2. Asymmetric Routing & Stateful Firewalls:                           |
|      RTP forward path traverses Firewall A; return path hits Firewall B |
|      without an established state table session -> DROPPED.             |
|                                                                         |
|   3. Firewall UDP Port Blocking:                                        |
|      Firewall blocks dynamic RTP port ranges (UDP 16384 - 32767).       |
|                                                                         |
|   4. IP Phone Default Gateway Misconfiguration:                         |
|      Phone can communicate with local CUCM via directed subnet, but     |
|      lacks default route to transmit RTP to remote branch subnet.       |
|                                                                         |
|   5. CUBE Interface Media Binding Failure:                              |
|      CUBE binds RTP to internal management loopback instead of WAN      |
|      interface facing the carrier SIP trunk.                            |
+-------------------------------------------------------------------------+

1. NAT Traversal & SDP IP Leakage

When an endpoint sits behind a Network Address Translation (NAT) router, the router translates the Layer 3 IP header. However, standard Layer 3 NAT does not rewrite the embedded payload of application-layer SIP messages. Inside the SIP message, the Session Description Protocol (SDP) payload contains the connection data (c=IN IP4 <ip-address>) and media description (m=audio <port> RTP/AVP ...). If the internal endpoint advertises its private RFC 1918 address (e.g., 10.1.1.50) in the SDP c= line, the external remote party attempts to transmit return RTP to 10.1.1.50—an unroutable address that is immediately discarded across the Internet or carrier WAN.

• Remediation: Deploy a Cisco Unified Border Element (CUBE) operating in Media Flow-Through mode with address hiding, or configure Session Traversal Utilities for NAT (STUN/TURN/ICE).

2. Asymmetric Routing & Stateful Firewalls

Signaling packets (SIP over TCP/UDP 5060) may follow a primary routed path through an enterprise firewall. When the firewall inspects the outbound SIP INVITE, it permits the session. However, due to dynamic routing protocols (e.g., OSPF/BGP unequal-cost paths), return RTP media may be routed through a secondary border firewall. Because the secondary firewall has no state table entry for the session, it treats the incoming UDP media packets as unsolicited external traffic and drops them.

3. CUBE Interface Media Binding

On a Cisco IOS XE router running CUBE, voice media originates from the router itself when bridging two call legs. If CUBE is not explicitly configured to bind its media source to the appropriate egress interface, it selects the IP address of the closest interface according to the local routing table—often an internal management interface or loopback that is unreachable from the external service provider network.

• Remediation: Explicitly configure media binding under voice-service voip or within individual dial-peers:

  voice service voip
   sip
    bind control source-interface GigabitEthernet0/0/1
    bind media source-interface GigabitEthernet0/0/1
  

Key One-Way Audio Diagnostic Commands

• show voice call status: Displays all active DSP and voice gateway call legs.
• show voip rtp connections: Lists the active RTP streams with local and remote IP addresses and UDP ports, which quickly shows which interface CUBE bound the media to.
• show call active voice brief: Shows each call leg's codec, transmit and receive packet counts, and lost, early, and late packet counters.
• debug ccsip messages: Prints full SIP message exchanges to inspect the SDP c= and m= lines for negotiated IP addresses and port numbers.

2. Choppy, Distorted & Robotic Audio Diagnosis

When callers complain of choppy speech, missing syllables, or robotic distortion, the root cause is almost always intermittent packet drop or severe jitter at the network layer.

1. Priority Queue Policer Drops

As explored in Section 11.3, Low Latency Queuing implements a strict policer on the voice priority queue. When an enterprise provisions a WAN interface for 10 concurrent calls (allocating $\approx 880 \text{ kbps}$), but 14 calls occur simultaneously during peak hours, total voice traffic reaches $\approx 1,220 \text{ kbps}$. During periods of WAN link congestion, the LLQ policer silently drops the excess 340 kbps of voice packets. Because the drops are distributed across all active calls, every caller experiences choppy, clipping audio.

• Verification: Run show policy-map interface <int> and look at the b/w exceed drops counter in the priority class.

2. Playout Buffer Overrun / Underrun

If delay variation across the WAN exceeds the depth of the receiver's playout buffer (e.g., jitter surges to 60 ms on a 40 ms buffer), packets arriving late are discarded as useless. The audio decoder detects missing frames and attempts to synthesize audio using PLC. When missing frames exceed PLC capabilities, the phone renders silence or synthetic metallic clicks.

3. Physical Layer Duplex Mismatch

A classic and insidious cause of audio degradation is an Ethernet duplex mismatch. If an access switchport is set to Auto-Negotiation while an attached router or voice gateway is hardcoded to Full Duplex (or vice versa), the auto-negotiating end defaults to Half Duplex.

• At low traffic volumes, calls sound normal.
• When data transfers occur simultaneously, the half-duplex side detects collisions, resulting in Late Collisions, CRC and frame errors, and heavy packet drops that destroy voice quality.
• Verification: Run show interface <int> and inspect late collisions, CRC, and input errors.

3. Video Quality Impairments: Tiling, Freezing & Lip-Sync

Video streams require orders of magnitude more bandwidth than voice and are uniquely vulnerable to differential frame corruption.

Macroblocking & Video Tiling

Video codecs transmit full reference images (I-frames) followed by differential motion updates (P-frames and B-frames). If a router drops packets belonging to a P-frame, the decoder cannot calculate the mathematical difference for subsequent motion blocks. The screen displays large, pixelated square blocks (macroblocking / tiling) where moving objects appear distorted or smeared across the static background.

+-------------------------------------------------------------------------+
|                    VIDEO KEYFRAME CORRUPTION & RECOVERY                 |
|                                                                         |
|   Transmitter                            Receiver (Decoder)             |
|   ===========                            ==================             |
|   I-Frame (Keyframe) -------------------> Decodes cleanly               |
|   P-Frame 1 ----------------------------> Decodes motion cleanly        |
|   P-Frame 2 (PACKET DROPPED) - - - - - -> Frame corrupt / Macroblocking |
|   P-Frame 3 ----------------------------> Cannot decode! Screen freezes |
|                                                                         |
|   [RTCP Full Intra Request (FIR)] <------ Receiver requests new keyframe|
|                                                                         |
|   New I-Frame (Huge Burst) -------------> Decodes; Video clears         |
+-------------------------------------------------------------------------+

Video Freezing & RTCP Full Intra Request (FIR)

When packet loss corrupts an entire I-frame or multiple consecutive P-frames, modern video endpoints freeze the video display on the last known valid image rather than displaying scrambled artifacts. To recover, the receiving endpoint transmits an out-of-band RTCP Full Intra Request (FIR, RFC 5104) or Picture Loss Indication (PLI, RFC 4585) back to the transmitter.

• Upon receiving the FIR/PLI, the transmitting camera or video codec immediately encodes a brand-new I-frame.
• Because I-frames are massive (spanning dozens of packets), this recovery mechanism creates a sudden instantaneous traffic surge across the WAN video queue.
• If the video queue lacks sufficient burst headroom (queue limit), the new I-frame packets are themselves dropped, trapping the video call in an endless loop of freezing and keyframe requests.

Lip-Sync Desynchronization

Lip-sync skew occurs when audio and video tracks drift out of temporal synchronization. This occurs when:

• Audio and video media packets traverse different network paths or experience disparate queuing delays.
• Endpoints fail to synchronize media clocks using the 64-bit NTP wall-clock timestamp contained in RTCP Sender Reports (SR).

4. Telemetry & Diagnostic Tools

Cisco Unified Real-Time Monitoring Tool (RTMT)

RTMT runs as a client application monitoring CUCM and Expressway clusters in real time:

• Call Management Records (CMR): Written alongside Call Detail Records (CDR) at call termination. They record packets sent and received, packets lost, jitter, and latency, and Cisco phones add voice-quality metrics such as the MOS estimate (MLQK) and concealment seconds.
• Quality alerts and reports: RTMT's preconfigured ExcessiveVoiceQualityReports alert watches Quality Report Tool (QRT) submissions from users, and CMR data can be reported through CDR Analysis and Reporting (CAR) or Cloud-Connected UC analytics.

Wireshark RTP Stream Analysis

Wireshark provides definitive packet-level media inspection:

1. Capture packet trace on an endpoint switchport (via SPAN / port mirroring) or router interface (via Embedded Packet Capture - EPC).
2. Navigate to Telephony $\rightarrow$ RTP $\rightarrow$ RTP Streams.
3. Select a stream and click Stream Analysis:
   • Max Delta: The maximum time elapsed between consecutive arriving packets. Spikes indicate severe network hesitation.
   • Max Jitter: Statistical variance calculated according to RFC 3550. If jitter $> 30 \text{ ms}$, playout buffer failure is imminent.
   • Packet Loss %: Quantifies exact dropped packets based on missing RTP sequence numbers.
   • RTP Player: Reconstructs and plays the actual audio waveform to verify whether distortion occurred before or after the capture point.

5. Troubleshooting Decision Matrix

6. Annotated Diagnostic CLI Outputs

1. Diagnosing LLQ Priority Policer Drops

The following output from show policy-map interface confirms that voice media is exceeding its allocated priority bandwidth during peak hours, resulting in 42,190 dropped packets:

Router# show policy-map interface GigabitEthernet0/0/1

 GigabitEthernet0/0/1 

  Service-policy output: PM-WAN-EDGE-PARENT

    Class-map: class-default (match-any)
      1849201 packets, 189201940 bytes
      5 minute offered rate 98420000 bps, drop rate 42000 bps
      Match: any
      Queueing
        queue limit 1000 packets
        (queue depth/total drops/no-buffer drops) 0/42190/0
        (pkts output/bytes output) 1807011/184982940
        shape (average) cir 100000000, bc 400000, be 400000
        target shape_rate 100000000

      Service-policy : PM-WAN-QUEUING-CHILD

        Class-map: CM-VOICE-MEDIA (match-any)
          492010 packets, 42804870 bytes
          5 minute offered rate 1420000 bps, drop rate 42000 bps
          Match: ip dscp ef (46)
          Priority: 1000 kbps, burst bytes 25000, b/w exceed drops: 42190
          ! CRITICAL: the priority class's implicit policer is dropping voice
          ! because the offered rate (1,420 kbps) exceeds 1,000 kbps during congestion

2. Inspecting Active Media Streams on CUBE

show call active voice brief reports counters for each call leg (output abridged):

Router# show call active voice brief
 ... pid:200 Answer +14085550199 active
 dur 00:03:12 tx:9560/1529600 rx:8140/1302400
 IP 10.10.50.25:16450 SRTP: off rtt:0ms pl:0/0ms lost:1420/0/0 delay:60/40/80ms g711ulaw

The leg toward the internal endpoint shows 1,420 lost packets and far fewer packets received than sent, which points to queuing or routing loss on the campus side rather than on the carrier leg. show voip rtp connections then confirms which local address and port CUBE is using for that stream.