1.3 Hardware Switching Architecture & Packet Forwarding

1.3 Hardware Switching Architecture & Packet Forwarding

High-performance enterprise switches forward tens of millions of packets per second at wire speed. To achieve line-rate performance without introducing latency or packet jitter, modern switching platforms decouple the control plane from the data plane and utilize specialized Application-Specific Integrated Circuits (ASICs) and memory architectures.

---

1. Switching Evolution: Process Switching vs. Fast Switching vs. CEF

Understanding how packet switching mechanisms have evolved is fundamental to mastering Cisco multilayer switching platforms.

+-----------------------------------------------------------------------+
|                   EVOLUTION OF PACKET FORWARDING                      |
+-----------------------------------------------------------------------+
|                                                                       |
|  [1. PROCESS SWITCHING]       [2. FAST SWITCHING]      [3. CEF]       |
|   - Every packet to CPU       - Route-Once,            - Topology-    |
|   - Slowest (~Kpps)             Switch-Many              driven       |
|   - Full RIB/ARP lookup       - First packet to CPU    - Pre-computed |
|                               - Cache programmed         FIB & Adj    |
|                               - Cache invalidation     - 100% ASIC    |
|                                 issues                   Wire Speed   |
+-----------------------------------------------------------------------+

1. Process Switching (Software Forwarding)

• Mechanics: Every inbound packet triggers an interrupt that diverts the packet to the main Route Processor (CPU).
• Operation: For each packet, the CPU performs an unoptimized lookup in the Routing Information Base (RIB), performs an ARP lookup, recalculates the IP header checksum, rewrites the Layer 2 frame header, and computes a new Cyclic Redundancy Check (CRC).
• Performance: Extremely slow (thousands of packets per second); causes high CPU utilization under moderate traffic loads.

2. Fast Switching (Route-Once, Switch-Many)

• Mechanics: The first packet of a new flow is punted to the CPU, which performs a route and ARP lookup.
• Operation: The CPU creates an entry in a fast-switching route cache (Layer 3 prefix and Layer 2 rewrite information). All subsequent packets belonging to that identical destination flow are switched via interrupt using the fast cache without consulting the RIB.
• Limitations: Prone to cache thrashing in dynamic networks; if routes flap or ARP entries expire, the cache must be flushed, causing periodic CPU spikes.

3. Cisco Express Forwarding (CEF - Topology-Driven Forwarding)

• Mechanics: Rather than waiting for packets to arrive to populate a cache, CEF pre-computes and compiles forwarding tables proactively directly from the routing table (RIB) and ARP table.
• Operation: The data plane hardware is pre-programmed with every known route and its corresponding Layer 2 rewrite string before the first packet ever arrives.
• Performance: True line-rate switching with zero per-flow CPU intervention.

---

2. Memory Architectures: CAM vs. TCAM

Silicon memory in enterprise switches is optimized for instantaneous parallel search operations.

+-----------------------------------------------------------------------+
|                       CAM vs. TCAM ARCHITECTURE                       |
+-----------------------------------------------------------------------+
|                                                                       |
|  CAM (Content Addressable Memory)      TCAM (Ternary CAM)             |
|  - Binary Logic: 0, 1                  - Ternary Logic: 0, 1, X (Mask)|
|  - Exact Match Search                  - Longest Prefix Match (LPM)   |
|  - Used for L2 MAC Address Table       - Used for FIB, ACLs, and QoS  |
|  - Output: Port Number                 - Output: VMR (Value/Mask/Res) |
+-----------------------------------------------------------------------+

Content Addressable Memory (CAM)

• Binary Matching: Evaluates exact binary states: 0 or 1.
• Lookup Type: Exact-match lookups executed across all entries in a single clock cycle.
• Primary Application: Layer 2 MAC Address Table (Forwarding Information Base - FDB). When an incoming frame arrives with destination MAC 00a1.2b3c.4d5e, CAM checks all entries in parallel and returns the corresponding egress switch port in one clock cycle.

Ternary Content Addressable Memory (TCAM)

• Ternary Matching: Evaluates three states: 0, 1, and X ("Don't Care" / Wildcard / Mask).
• Structure: Operates using a VMR (Value, Mask, Result) architecture:
  • Value: The patterns to match (e.g., IP address 10.1.0.0, TCP port 443).
  • Mask: The bitmask indicating which bits must match exactly versus wildcard bits (255.255.0.0).
  • Result: The action executed upon matching (e.g., forward via Next-Hop pointer, permit, deny, apply QoS DSCP marking).
• Primary Applications:
  • Layer 3 Longest Prefix Match (LPM): Matches IP routes against variable-length subnet masks simultaneously in a single clock cycle.
  • Security ACLs: Evaluates complex multi-field rules (Source IP, Destination IP, Protocol, Layer 4 Port range) at wire speed.
  • Quality of Service (QoS): Traffic classification and policing.
  • Policy-Based Routing (PBR): Routing packets based on source criteria rather than destination IP alone.

Comparison: CAM vs. TCAM

3. Cisco Express Forwarding (CEF) Internal Tables & Adjacency Types

CEF structures forwarding data into two distinct, optimized internal tables:

1. Forwarding Information Base (FIB)

• Derived directly from the Layer 3 Routing Information Base (RIB).
• Organizes IP prefix entries in an inverted 256-way mtrie (multi-branch tree) structure optimized for rapid hardware TCAM searches.
• Contains every routable destination prefix, the corresponding next-hop IP address, and an index pointer referencing the appropriate entry in the Adjacency Table.
• Whenever the routing table changes (e.g., OSPF recalculates a shortest path), the control plane immediately updates the FIB.

2. Adjacency Table

• Derived directly from the ARP table and neighbor discovery databases.
• Maintains pre-formatted Layer 2 frame rewrite strings (Destination MAC, Source MAC, EtherType, 802.1Q tag) for every directly connected next-hop.
• When a packet matches a prefix in the FIB, the switch uses the pointer to grab the pre-built Layer 2 header from the Adjacency table, replaces the incoming frame header, decrements the IP TTL, updates the checksum, and transmits the frame out the physical port.

+-----------------------------------------------------------------------+
|                   CEF ADJACENCY TYPES & BEHAVIORS                     |
+-----------------------------------------------------------------------+
|                                                                       |
|  [RESOLVED]     -> Valid L2 MAC header present; wire-speed forward.   |
|  [GLEAN]        -> Directly connected subnet; MAC missing in ARP;     |
|                    punts initial packet to CPU to trigger ARP Request.|
|  [PUNT]         -> Unsupported feature/TTL=1/IP options/fragments;   |
|                    diverted to CPU Route Processor for handling.      |
|  [DROP]         -> Unroutable/unsupported; dropped in hardware.       |
|  [DISCARD/NULL0]-> Route points to Null0; discarded in hardware.      |
+-----------------------------------------------------------------------+

Specialized CEF Adjacency Types

Centralized CEF vs. Distributed CEF (dCEF)

• Centralized CEF:
  • Deployed on fixed-configuration switches (e.g., Catalyst 9200/9300).
  • The central Route Processor / Supervisor maintains the FIB and Adjacency tables and programs the local switching ASICs.
• Distributed CEF (dCEF):
  • Deployed on high-density modular chassis systems (e.g., Catalyst 9400/9600, Nexus 7000/9500).
  • The main supervisor engine compiles the master FIB and Adjacency tables and downloads identical copies to Distributed Forwarding Cards (DFCs) located on each individual line card.
  • Line cards perform complete Layer 3 forwarding and rewriting locally across the switch fabric without burdening the central supervisor engine.

---

4. TCAM Exhaustion & Switch Database Management (SDM) Templates

Physical TCAM capacity on switching ASICs is fixed and finite. If an enterprise network outgrows its allocated TCAM space (due to massive routing tables, dense ACLs, or extensive QoS policies), the switch experiences TCAM Exhaustion.

+-----------------------------------------------------------------------+
|                   TCAM EXHAUSTION FAILURE CASCADE                     |
+-----------------------------------------------------------------------+
|                                                                       |
|  1. TCAM Capacity Reaches 100% (e.g., Route Table overflow)           |
|  2. New prefixes cannot be programmed into hardware FIB               |
|  3. Switch falls back to SOFTWARE PROCESS SWITCHING (CPU)             |
|  4. Switch CPU utilization spikes to 99%-100%                         |
|  5. Massive latency spikes, buffer overruns, and packet drops occur   |
|  6. Control plane keepalives (OSPF/BGP hellos) drop -> Route Flapping |
+-----------------------------------------------------------------------+

Switch Database Management (SDM) Templates

To prevent TCAM exhaustion and tailor switch resource allocation to specific network roles, Cisco IOS / IOS-XE provides SDM Templates. SDM reallocates fixed TCAM memory blocks between Layer 2 switching, Layer 3 routing, and security/QoS tables.

! Verify current SDM template allocation
Switch# show sdm prefer
 The current template is "default" template.
 The selected template optimizes the resources in
 the switch to support balanced Layer 2 and Layer 3 functions.

 Number of unicast mac addresses:                  16K
 Number of IPv4 IGMP groups + multicast routes:    1K
 Number of IPv4 unicast routes:                    16K
    number of directly-connected IPv4 hosts:       8K
    number of indirect IPv4 prefixes:              8K
 Number of IPv4 policy based routing aces:         0.5K
 Number of IPv4 access control list aces:          1K

Common SDM Templates and Target Roles

Configuration Note: Changing an SDM template requires entering global configuration mode (sdm prefer <template-name>) followed by a manual switch reload (reload) for the ASIC memory partition to be reallocated.