4.4 Link Aggregation (LAG) and LACP Configuration

4.4 Link Aggregation (LAG) and LACP Configuration

In enterprise campus networks, interconnecting access switches to aggregation and core layers using individual physical Ethernet cables introduces severe bandwidth bottlenecks and single points of failure. If an engineer runs multiple parallel physical cables between two switches to increase capacity, standard Layer 2 loop-prevention mechanisms—such as Spanning Tree Protocol (STP)—view the parallel links as a loop and block all redundant links, leaving only a single active connection.

Link Aggregation (LAG) resolves this architectural limitation by bundling multiple physical Ethernet links into a single logical channel (often called a trunk or port-channel in legacy terminology). Spanning Tree views the aggregate bundle as a single logical port, keeping all member links actively forwarding. Understanding the operational mechanics of the Link Aggregation Control Protocol (LACP), hashing algorithms, and AOS-CX CLI configuration is essential for building resilient, high-throughput campus access architectures.

---

Link Aggregation Principles and Benefits

Bundling physical interfaces into a Link Aggregation Group delivers three primary operational advantages:

1. Increased Bandwidth and Throughput: Aggregates the physical capacity of all active member interfaces. For example, bundling four 10GbE SFP+ uplinks creates a single logical 40Gbps transmission pipe.
2. Seamless Fault Tolerance (Sub-Second Failover): If one physical cable, transceiver, or switch port in the bundle fails, traffic is immediately redistributed across the remaining active links in hardware within milliseconds. Spanning tree is not triggered, avoiding topology change notifications (TCNs) or network reconvergence delays.
3. Load Distribution: Outbound traffic flows are distributed across all operational member links using deterministic hardware hashing algorithms.

---

Static LAG vs. Dynamic LACP (IEEE 802.3ad / 802.1AX)

AOS-CX supports two methods for establishing Link Aggregation Groups: Static LAG and Dynamic LACP.

+-------------------------------------------------------------------------+
|                        STATIC LAG VS. DYNAMIC LACP                      |
|                                                                         |
|   STATIC LAG (No Protocol Negotiation)                                  |
|   Switch A [Port 1/1/47] -------------> Switch B [Port 1/1/47]          |
|   Switch A [Port 1/1/48] ----->?------> Mismatched / Bad Port (Blackhole)|
|   * No keepalive packets; cannot detect miswiring or remote faults *    |
|                                                                         |
|   DYNAMIC LACP (IEEE 802.3ad / 802.1AX LACPDUs)                         |
|   Switch A [Port 1/1/47] <== LACPDU ==> Switch B [Port 1/1/47]          |
|   Switch A [Port 1/1/48] <== LACPDU ==> Switch B [Port 1/1/48]          |
|   * Continuous negotiation; validates System ID, Port ID, and Key *     |
+-------------------------------------------------------------------------+

1. Static Link Aggregation

• Physical member ports are forced into a logical bundle without running a control-plane protocol.
• The switch assumes the remote link partner has identical configuration and physical connectivity.
• Vulnerability: If a patch cable is misconnected to an incorrect switch, if a fiber cable suffers a unidirectional fault, or if the remote partner's configuration is modified, the static LAG cannot detect the error. Traffic hashed to the broken link is silently dropped (black-holed), or a severe Layer 2 loop is formed.

2. Dynamic LACP (IEEE 802.3ad / 802.1AX)

• Member ports exchange standardized Link Aggregation Control Protocol Data Units (LACPDUs) to continuously negotiate and monitor link state.
• Parameters Validated in LACPDUs:
  • System Priority and System ID: Identifies the remote switch (System ID = Priority + Base MAC address).
  • Port Priority and Port ID: Identifies individual physical ports.
  • Operational Key: Identifies port capabilities (speed, duplex, media) to ensure only compatible links join the bundle.
  • Link State Flags: Ensures both sides agree the link is synchronized and ready for data forwarding.
• If an LACPDU mismatch occurs, or if LACPDUs stop arriving on a member link, LACP immediately removes that member port from the active forwarding bundle, preventing black holes and packet loops.

---

LACP Operational Modes and Timers

When configuring dynamic LACP, each endpoint must be configured with an operational mode: Active or Passive.

1. LACP Negotiation Modes

• Active Mode (lacp mode active): The switch port actively initiates LACP negotiation by transmitting LACPDUs at scheduled intervals, regardless of whether the remote peer sends them first.
• Passive Mode (lacp mode passive): The switch port listens for incoming LACPDUs. It will respond to LACPDUs sent by an active partner, but it will never initiate negotiation on its own.

Enterprise Best Practice: Always configure lacp mode active on both sides of an aggregation link to ensure deterministic, rapid bundle establishment.

2. LACP Timers (Rates)

LACP supports two transmission rates for heartbeat LACPDUs:

• Fast Rate (lacp rate fast): LACPDUs are transmitted every 1 second. The partner timeout is 3 seconds (3 missed LACPDUs). This provides rapid link failure detection for mission-critical core and aggregation uplinks.
• Slow / Normal Rate (lacp rate slow): LACPDUs are transmitted every 30 seconds. The partner timeout is 90 seconds. This is the AOS-CX default (no lacp rate returns to slow).

---

Hardware Load-Balancing Hash Algorithms

A critical networking concept tested on certification exams is how traffic is distributed across LAG member links. A Link Aggregation Group does NOT perform round-robin per-packet distribution.

+-------------------------------------------------------------------------+
|                    PER-FLOW LOAD BALANCING HASHING                      |
|                                                                         |
|   Flow 1 (PC A -> Server 1): Hash(SrcIP + DstIP + Port) = Link 1        |
|   [Pkt 1] [Pkt 2] [Pkt 3] ----------------------------> Physical Link 1 |
|                                                         (Preserves Order)|
|                                                                         |
|   Flow 2 (PC B -> Server 2): Hash(SrcIP + DstIP + Port) = Link 2        |
|   [Pkt A] [Pkt B] [Pkt C] ----------------------------> Physical Link 2 |
+-------------------------------------------------------------------------+

Why Per-Flow Distribution is Mandatory

If packets belonging to a single TCP stream were distributed across multiple links on a per-packet basis, slight differences in physical cable lengths, serialization delays, or queue depths would cause packets to arrive at the destination out of order. Out-of-order packet delivery forces the receiving TCP stack to generate duplicate ACKs and triggers TCP fast retransmit and window collapse, catastrophic to network performance. Therefore, LAGs distribute traffic per flow.

Hashing Header Fields

The switch hardware ASIC evaluates a mathematical hash algorithm across specific packet header fields to assign each conversation flow to a specific physical member link:

• Layer 2 Hashing: Evaluates Source MAC address and Destination MAC address (l2-src-dst).
• Layer 3 Hashing: Evaluates Source IP address and Destination IP address (l3-src-dst).
• Layer 4 Hashing: Adds Layer 4 information such as source and destination TCP/UDP ports (l4-src-dst).

On AOS-CX the hash is configured on the LAG interface with hash {l2-src-dst | l3-src-dst | l4-src-dst}, and l3-src-dst is the default. In networks with many flows between the same pair of hosts, l4-src-dst adds entropy and can spread traffic more evenly.

---

Step-by-Step AOS-CX LAG Configuration

Configuring a Link Aggregation Group in AOS-CX follows a clean, two-step process: defining the logical LAG interface, and then assigning physical member interfaces to that group.

Step 1: Create and Configure the Logical LAG Interface

In AOS-CX, the logical bundle is created using the interface lag <id> command. All Layer 2 and Layer 3 properties (such as VLAN tagging, native VLAN, descriptions, and LACP mode) are configured directly on this logical interface:

switch# configure
switch(config)# interface lag 10
switch(config-lag-if)# description "Uplink to Core VSX"
switch(config-lag-if)# no shutdown
switch(config-lag-if)# no routing
switch(config-lag-if)# vlan trunk native 999
switch(config-lag-if)# vlan trunk allowed 10,20,30,40,999
switch(config-lag-if)# lacp mode active
switch(config-lag-if)# lacp rate fast

Step 2: Assign Physical Member Interfaces to the LAG

Once the logical LAG is defined, physical front-panel interfaces are assigned to the bundle using the lag <id> command:

switch(config)# interface 1/1/47-1/1/48
switch(config-if-<1/1/47-1/1/48>)# no shutdown
switch(config-if-<1/1/47-1/1/48>)# lag 10

Important Operational Rule: Physical member interfaces automatically inherit their Layer 2 VLAN configuration, MTU, and operational parameters from the parent logical LAG interface. Physical member ports must have matching speed, duplex, and transceiver capabilities.

---

Multi-Chassis Stacking Considerations (VSF and VSX)

Traditional LAG is limited to terminating on a single physical switch chassis on each end of the connection. AOS-CX overcomes this physical constraint through network virtualization:

+-------------------------------------------------------------------------+
|                   CROSS-STACK LAG IN A VSF TOPOLOGY                     |
|                                                                         |
|   +-----------------------------------------------------------------+   |
|   |                     VSF LOGICAL CHASSIS                         |   |
|   |  +---------------------------+   +---------------------------+  |   |
|   |  | Member 1 (Conductor)      |   | Member 2 (Standby)        |  |   |
|   |  | Port 1/1/48               |   | Port 2/1/48               |  |   |
|   |  +-------------+-------------+   +-------------+-------------+  |   |
|   +----------------|-------------------------------|----------------+   |
|                    \                               /                    |
|                     \                             /                     |
|                      +====== LAG 10 (LACP) ======+                      |
|                                     |                                   |
|                         +-----------+-----------+                       |
|                         | Downstream CX 6200    |                       |
|                         | Access Switch         |                       |
|                         +-----------------------+                       |
+-------------------------------------------------------------------------+

• VSF Cross-Stack LAG: In a VSF stack, physical member ports of a single logical LAG can be distributed across different physical switches in the stack (e.g., port 1/1/48 on Member 1 and port 2/1/48 on Member 2). The downstream device connects via standard LACP, unaware that its uplinks terminate on two separate physical chassis. This provides simultaneous link-level and switch-level hardware redundancy.
• VSX-LAG (Multi-Chassis LAG): In campus aggregation and core networks using dual CX 6400, 8100, 8325, or 8360 switches, VSX-LAG pairs two independent switches across an Inter-Switch Link (ISL). Downstream switches connect via standard LACP to both VSX peers, achieving active-active multi-chassis redundancy without requiring shared control planes.

---

Verification and Troubleshooting Commands

switch# show lag 10
switch# show lacp interfaces
switch# show lacp aggregates

Abbreviated sample show lacp interfaces output:

Interface   LAG   Mode      State     Key    Priority
-----------------------------------------------------
1/1/47      10    Active    Forward   10     32768
1/1/48      10    Active    Forward   10     32768