8.3 Switching Architectures: Top-of-Rack, Middle-of-Row, and End-of-Row

8.3 Switching Architectures: Top-of-Rack, Middle-of-Row, and End-of-Row

While logical network design defines protocols, IP subnets, and routing fabrics, the physical switching architecture determines where network hardware is physically installed within the computer room whitespace and how physical cabling pathways are routed between switches and servers.

The three dominant physical switching architectures deployed in enterprise and colocation data centers are:

1. Top-of-Rack (ToR)
2. End-of-Row (EoR)
3. Middle-of-Row (MoR)

Selecting the appropriate switching topology represents a complex multi-disciplinary trade-off between switch port utilization efficiency, capital expenditure (CapEx) on network electronics versus structured cabling, operational expenditure (OpEx) for moves, adds, and changes (MACs), overhead cable tray loading, and thermal dissipation inside server cabinets.

---

Top-of-Rack (ToR) Switching Architecture

In a Top-of-Rack (ToR) switching topology (also referred to as In-Rack switching), one or two 1U/2U leaf switches are installed directly inside each individual server cabinet, typically mounted in the topmost rack units (e.g., RU 41–42) or occasionally in the vertical center of the cabinet.

+---------------------------------------------------------------------------------------------------------+
|                                TOP-OF-RACK (ToR) CABLING TOPOLOGY                                       |
+---------------------------------------------------------------------------------------------------------+

   OVERHEAD FIBER TRAY ◄═════════════════════════════════════════════════════════════════════════════
                               │ (2 to 4 Fiber Uplinks)           │ (2 to 4 Fiber Uplinks)
                               ▼                                  ▼
   CABINET 1                   ┌─────────────────────┐            ┌─────────────────────┐ CABINET 2
   +-----------------------+   │  ToR Leaf Switch 1  │            │  ToR Leaf Switch 2  │ +-----------------------+
   | [ToR Leaf Switch]     |   └──────────┬──────────┘            └──────────┬──────────┘ | [ToR Leaf Switch]     |
   |                       |              │                                  │            |                       |
   | [Server 1]            |              │ Short In-Rack                    │            | [Server 1]            |
   | [Server 2]            |              │ Copper DAC / AOC                 │            | [Server 2]            |
   | [Server 3]            |              │ Patches (≤ 3 m)                  │            | [Server 3]            |
   | [Server ...]          |              ▼                                  ▼            | [Server ...]          |
   | [Server N]            |   ┌─────────────────────┐            ┌─────────────────────┐ | [Server N]            |
   +-----------------------+   │ Server Patch Ports  │            │ Server Patch Ports  │ +-----------------------+

Physical Cabling Characteristics

• In-Rack Server Connections: Every server installed in the cabinet connects directly to the in-rack ToR leaf switch using short, pre-terminated point-to-point jumpers. These are typically passive Direct Attach Copper (DAC) twinaxial cables ($1\text{ m}$ to $3\text{ m}$ in length) or short Active Optical Cables (AOC).
• Inter-Rack Conveyance: No server copper cables ever leave the cabinet. The only cables that exit the cabinet into the overhead pathway are a small number (typically 2 to 4) of high-speed optical fiber uplinks (e.g., 100G/400G OM4 multimode or OS2 single-mode) connecting the ToR leaf switch to the central spine switches.

Advantages of ToR

1. Minimal Pathway Congestion: Because only a few optical uplinks leave each cabinet, overhead cable trays remain small (e.g., 6-inch or 12-inch wide wire mesh trays) with negligible weight loading and minimal fill ratios.
2. Modular Cabinet Deployment: Cabinets can be fully pre-cabled, integrated, and tested off-site by an integrator and rolled onto the data center floor as self-contained compute modules ("rack-and-stack").
3. Low Cabling Media Costs: Utilizing inexpensive $1\text{ m}$ to $2\text{ m}$ passive DAC copper cables for server patching eliminates the need for costly optical transceivers and structured patch panels for every server port.

Disadvantages of ToR

1. Severe Port Stranding (Inefficient Port Utilization): This is the primary economic flaw of ToR. If an enterprise installs sixteen (16) 1U servers in a 42U rack, each server requiring 2 network ports (32 ports total), and deploys two 48-port ToR switches for redundancy: $\text{Total Installed Ports} = 2 \times 48 = 96 \text{ ports}$ $\text{Total Utilized Ports} = 16 \times 2 = 32 \text{ ports}$ $\text{Stranded Ports} = 96 - 32 = 64 \text{ ports} \quad (66.7\% \text{ port waste})$ The enterprise paid capital purchase and maintenance costs for 64 switch ports that cannot be used by servers in adjacent racks without violating ToR cabling rules.
2. High Electronic Device Count & Management Overhead: A data hall with 100 server cabinets requires 200 individual ToR switches to manage, license, configure, and maintain.
3. Thermal and Power Impact: ToR switches consume 1U to 2U of valuable rack space and dissipate an additional $350\text{ W}$ to $800\text{ W}$ of heat inside the top of the server cabinet, where ambient temperatures are highest.

---

End-of-Row (EoR) Switching Architecture

In an End-of-Row (EoR) architecture, network switching is centralized. Individual server cabinets contain no active switches. Instead, large modular chassis switches (or high-density switch stacks) are installed in one or two dedicated Network Cabinets positioned at the physical end of the server row.

+---------------------------------------------------------------------------------------------------------+
|                                 END-OF-ROW (EoR) CABLING TOPOLOGY                                       |
+---------------------------------------------------------------------------------------------------------+

   MASSIVE OVERHEAD CABLE BUNDLE (Hundreds of Cat 6A / Fiber Drops) ══════════════════════════════════╗
   ▲                            ▲                            ▲                                        ║
   │ Horizontal Drops           │ Horizontal Drops           │ Horizontal Drops                       ║
   │ (15 to 30 m)               │ (30 to 60 m)               │ (60 to 90 m)                           ║
   │                            │                            │                                        ▼
+--+--------------------+    +--+--------------------+    +--+--------------------+       +-----------------------+
| Server Cabinet 1      |    | Server Cabinet 2      |    | Server Cabinet N      |       | EoR Network Cabinet   |
| (Patch Panels Only)   |    | (Patch Panels Only)   |    | (Patch Panels Only)   |       | • Modular Chassis Sw. |
| [Server 1]            |    | [Server 1]            |    | [Server 1]            |       | • High-Density Patch  |
| [Server 2]            |    | [Server 2]            |    | [Server 2]            |       |   Panels (48-port)    |
| [Server ...]          |    | [Server ...]          |    | [Server ...]          |       | • Centralized Uplinks |
| [Server N]            |    | [Server N]            |    | [Server N]            |       | • Dedicated Power/Cool|
+-----------------------+    +-----------------------+    +-----------------------+       +-----------------------+

Physical Cabling Characteristics

• Structured Horizontal Cabling: Every server cabinet is equipped with a passive copper (Category 6A/8) or optical fiber patch panel.
• Permanent horizontal cable runs extend from every server cabinet across overhead cable trays (or under raised access floors) to central patch panels in the EoR cabinet.
• Servers connect to their local cabinet patch panel via short $1\text{ m}$ patch cords, and technicians cross-connect ports in the EoR cabinet to the chassis switch.

Advantages of EoR

1. High Port Utilization Efficiency: Because all switch ports reside in a central pool, ports can be dynamically allocated to any server in any cabinet across the row. Port utilization regularly exceeds 85% to 90%, virtually eliminating stranded port waste.
2. Centralized Device Management: Instead of managing dozens of individual switches, network administrators manage a single pair of redundant modular chassis switches per row, drastically reducing software licensing, firmware updates, and monitoring points.
3. Optimized Server Cabinet Space & Cooling: Server cabinets dedicate 100% of their rack units to compute and storage equipment without competing with active network hardware for power or airflow.

Disadvantages of EoR

1. Massive Cable Tray Congestion: A row of 20 server cabinets, each with 24 network drops, generates 480 individual horizontal cables converging into the EoR cabinet. Overhead cable trays must be massive (18-inch to 24-inch wide, 4-inch to 6-inch deep) and structurally reinforced to support extreme weight.
2. Distance and Attenuation Limits: In long server rows (e.g., 25 to 30 cabinets), horizontal cable runs from the furthest cabinet to the EoR cabinet frequently exceed $60\text{ m}$ to $80\text{ m}$. High-speed Category 8 copper is limited to a total channel length of $30\text{ meters}$, making it physically impossible to deploy Cat 8 in large EoR rows.
3. High Initial Cabling Labor & CapEx: Installing structured horizontal cabling across hundreds of ports requires extensive labor for pulling, bundling, terminating, and fluke-testing every permanent link.

---

Middle-of-Row (MoR) Switching Architecture

The Middle-of-Row (MoR) architecture is an engineering evolution of EoR designed specifically to mitigate cable distance limitations and pathway congestion.

In an MoR deployment, the dedicated network cabinets are placed in the exact physical center of the server row (e.g., in a 20-cabinet row, the network cabinets occupy positions 10 and 11).

+---------------------------------------------------------------------------------------------------------+
|                               MIDDLE-OF-ROW (MoR) CABLING TOPOLOGY                                      |
+---------------------------------------------------------------------------------------------------------+

       CABLE BUNDLE (LEFT ROW HALF)                     CABLE BUNDLE (RIGHT ROW HALF)
   ═══════════════════════════════════╗             ╔═══════════════════════════════════
   ▲                                  ║             ║                                  ▲
   │ Max Run: ~30 to 40 m             ▼             ▼             Max Run: ~30 to 40 m │
+--+--------------------+       +-----------------------+       +--------------------+--+
| Server Cabinet 1      | ..... | MoR Network Cabinet   | ..... | Server Cabinet 20     |
| (Left Row Terminus)   |       | • Modular Chassis Sw. |       | (Right Row Terminus)  |
+-----------------------+       | • Central Distribution|       +-----------------------+
                                +-----------------------+

The Engineering Mechanics of MoR

1. 50% Reduction in Maximum Cable Length: By placing the network switch in the center, the maximum distance to the furthest server cabinet is cut exactly in half compared to an EoR layout: $L_{\text{max, MoR}} = \frac{L_{\text{max, EoR}}}{2}$ This distance reduction allows high-performance copper media (such as Category 6A and Category 8) to easily comply with standard channel length restrictions ($30\text{ m}$ to $100\text{ m}$).
2. Bifurcated Pathway Fill: Horizontal cable pathways are split into two opposing directions (one tray serving the left half of the row, one tray serving the right half). This eliminates the massive cable convergence bottleneck that plagues EoR cabinets, reducing required tray cross-sectional area by 50%.
3. Optimal Row Symmetry: MoR provides the centralized management and high port efficiency of EoR while delivering the balanced pathway loading of distributed architectures.

---

Architectural Comparison Matrix

+---------------------------------------------------------------------------------------------------------+
|                         SWITCHING ARCHITECTURE TRADE-OFF COMPARISON MATRIX                              |
+-----------------------+-----------------------+-----------------------+---------------------------------+
| CRITERIA              | TOP-OF-RACK (ToR)     | END-OF-ROW (EoR)      | MIDDLE-OF-ROW (MoR)             |
+-----------------------+-----------------------+-----------------------+---------------------------------+
| Switch Port           | Low (50% – 70%)       | High (85% – 95%)      | High (85% – 95%)                |
| Utilization           | Severe port stranding | Minimal port waste    | Minimal port waste              |
+-----------------------+-----------------------+-----------------------+---------------------------------+
| Overhead Pathway      | Very Low              | Very High             | Moderate                        |
| Fill & Weight         | (Few fiber uplinks)   | (Massive bundle conv.)| (Bundles split into two halves) |
+-----------------------+-----------------------+-----------------------+---------------------------------+
| In-Rack Patch Media   | Point-to-Point DAC/AOC| Structured Horizontal | Structured Horizontal           |
|                       | (Short ≤ 3 m)         | Copper / Fiber Drops  | Copper / Fiber Drops            |
+-----------------------+-----------------------+-----------------------+---------------------------------+
| Maximum Cable Length  | Very Short (1 - 3 m)  | Long (50 - 90+ m)     | Medium (25 - 45 m)              |
| to Server             | Internal to cabinet   | Approaches limits     | Well within copper channel limits
+-----------------------+-----------------------+-----------------------+---------------------------------+
| Switch Hardware CapEx | High (Many physical   | Moderate (Fewer high- | Moderate (Fewer high-           |
|                       | individual switches)  | density chassis)      | density chassis)                |
+-----------------------+-----------------------+-----------------------+---------------------------------+
| Structured Cable CapEx| Very Low              | High                  | High                            |
| & Installation Labor  | (In-rack jumpers)     | (Massive pull/term)   | (Massive pull/term)             |
+-----------------------+-----------------------+-----------------------+---------------------------------+
| Management Points     | High (Hundreds of     | Low (Central pair of  | Low (Central pair of            |
| (Device Count)        | switches to maintain) | chassis per row)      | chassis per row)                |
+-----------------------+-----------------------+-----------------------+---------------------------------+
| Server Cabinet Thermal| Added heat load       | Zero switch heat      | Zero switch heat                |
| & Space Impact        | (Consumes 1-2 RU)     | (100% rack for compute| (100% rack for compute)         |
+-----------------------+-----------------------+-----------------------+---------------------------------+

---

Physical Transmission Media in Switching Architectures

Connecting servers to switches requires selecting appropriate physical transmission media:

+---------------------------------------------------------------------------------------------------------+
|                                 TRANSMISSION MEDIA CHARACTERISTICS                                      |
+-------------------+-------------------+--------------------+--------------------+-----------------------+
| MEDIA TYPE        | MAXIMUM REACH     | LATENCY            | POWER DISSIPATION  | RELATIVE COST         |
+-------------------+-------------------+--------------------+--------------------+-----------------------+
| Passive Direct    | 1 m to 3 m        | Ultra-Low          | Negligible         | Lowest                |
| Attach Copper(DAC)| (~5 m with FEC)   | (< 0.1 µs)         | (< 0.1 W per end)  | (~US$20 - $50)        |
+-------------------+-------------------+--------------------+--------------------+-----------------------+
| Active Optical    | 5 m to 100 m      | Low                | Moderate           | Moderate              |
| Cable (AOC)       |                   | (0.1 - 0.3 µs)     | (1.0 - 1.5 W / end)| (~US$80 - $180)       |
+-------------------+-------------------+--------------------+--------------------+-----------------------+
| Structured Optical| 100 m to 150 m    | Low                | Moderate to High   | Highest (Transceivers |
| Transceivers (SR4)| (OM4 multimode)   | (0.1 - 0.3 µs)     | (1.5 - 2.5 W / end)| + Cassettes + Fiber)  |
+-------------------+-------------------+--------------------+--------------------+-----------------------+
| Balanced Copper   | 30 m (Cat 8)      | Higher (DSP PHY)   | High               | Low media cost,       |
| Twisted-Pair (UTP)| 100 m (Cat 6A)    | (1.5 - 2.5 µs)     | (2.5 - 4.0 W / port| high termination labor|
+-------------------+-------------------+--------------------+--------------------+-----------------------+

Direct Attach Copper (DAC) vs. Active Optical Cables (AOC)

• Passive DAC Cables: Consist of shielded twinaxial copper cable factory-terminated with transceiver shells (e.g., SFP28, QSFP28). They contain no active optical-to-electrical conversion silicon. Because signals travel purely electrically, DAC consumes virtually zero power and introduces sub-microsecond latency. However, high-frequency signal attenuation restricts 25G/100G passive DACs to about $3\text{ meters}$ (up to roughly $5\text{ meters}$ with forward error correction). They are suited to ToR in-rack and adjacent-rack connections.
• Active Optical Cables (AOC): Consist of multi-strand fiber optic cable permanently bonded to optical transceiver ends containing active VCSEL lasers and photodetectors. AOCs are lightweight, have a bend radius under $30\text{ mm}$, are completely immune to electromagnetic interference (EMI), and achieve reaches up to $100\text{ meters}$. They are widely deployed for MoR, EoR, and inter-rack spine connections.

---

Engineering Calculations: Cable Tray Fill Ratios

When designing structured cabling pathways for EoR and MoR rows, the DCDC must calculate overhead cable tray dimensions per ANSI/TIA-569 and ANSI/BICSI 002-2024 (whose Table 14-5 limits cable stacking height in pathways), applying the NEC where the tray also carries cables the code regulates (Article 392).

Cable Tray Fill Criteria

Because round cables cannot pack solidly, a tray at a 50% calculated fill already looks completely full. Keeping the initial fill low (25%) is required to:

1. Prevent crushing and compressive deformation of lower cable layers, which degrades Category 6A Alien Crosstalk (ANEXT) performance.
2. Allow cooling air circulation around cable bundles to prevent heat buildup.
3. Accommodate future cable installations during tenant expansion.

$\text{Design Fill Ratio} = \frac{\sum A_{\text{cables}}}{A_{\text{tray, internal}}} \le 0.25 \text{ (initial)}, \quad \le 0.50 \text{ (maximum)}$

Where: $A_{\text{tray, internal}} = W_{\text{tray}} \times D_{\text{tray}}$ $\sum A_{\text{cables}} = N_{\text{cables}} \times \frac{\pi \times d_{\text{cable}}^2}{4} \quad \text{(or using standard square approximation } d^2 \text{)}$

+---------------------------------------------------------------------------------------------------------+
|                               SAMPLE TRAY SIZING CALCULATION (EoR ROW)                                  |
+---------------------------------------------------------------------------------------------------------+

   Parameters:
   • Number of server cabinets in row: 18 cabinets
   • Structured horizontal drops per cabinet: 24 Category 6A UTP cables
   • Total cables entering EoR tray: 18 x 24 = 432 cables
   • Outside diameter of Cat 6A cable (d): 0.30 inches (7.62 mm)

   Step 1: Calculate Total Cable Cross-Sectional Area (using standard d^2 envelope):
   A_cable = (0.30 in)^2 = 0.090 sq in per cable
   Total Cable Area = 432 cables x 0.090 sq in = 38.88 sq in

   Step 2: Apply the TIA-569 25% Initial Design Fill:
   Required Tray Area = Total Cable Area / 0.25
   Required Tray Area = 38.88 sq in / 0.25 = 155.52 sq in

   Step 3: Select Standard Tray Dimensions (cable depth limited to 6 in):
   • Option A: 6-inch deep tray -> Required Width = 155.52 / 6 = 25.92 in -> Specify a 30-inch wide tray
     (actual initial fill = 38.88 / 180 = 21.6%).
   • Option B: 4-inch deep trays -> Required Width = 155.52 / 4 = 38.9 in -> exceeds a single 36-inch tray,
     so specify two 24-inch wide trays (fill = 38.88 / 192 = 20.3%).
   • Check: at the 50% TIA-569 maximum, the 30-inch x 6-inch tray could hold about 1,000 such cables.

Pathway Structural Weight Loading

In addition to volume, the structural engineer must calculate total linear weight loading. High-performance Category 6A cable weighs approximately $40\text{ lbs per 1,000 ft}$ ($0.040\text{ lbs/ft}$). $\text{Weight of 432 cables} = 432 \times 0.040\text{ lbs/ft} = 17.28\text{ lbs per linear foot} \quad (25.7\text{ kg/m})$ Adding the dead weight of a heavy steel wire basket tray ($4.5\text{ lbs/ft}$), the total suspended overhead load is $21.78\text{ lbs/ft}$. Ceiling trapeze supports and threaded unistrut rods must be structurally anchored to building structural steel to prevent catastrophic ceiling pathway collapse.

---

Practical Design Scenario: Converting an Enterprise Row from EoR to ToR

Project Profile

An enterprise data center has ten rows, each containing 20 server cabinets. The existing design utilizes an End-of-Row (EoR) architecture installed eight years ago with Category 6 UTP cabling terminated into modular chassis switches.

The enterprise is executing a major technology refresh: deploying 2U dual-socket cloud compute nodes requiring dual 25GbE network connections per server (16 servers per rack = 32 ports per rack).

Engineering Dilemma

• The existing Category 6 horizontal cabling cannot support 25 Gbps transmission (Category 6 is limited to 1 Gbps at 100 m, or marginal 10G at 37 m).
• The network team evaluates pulling 640 new Category 8 or fiber cables through the existing overhead trays to retain the EoR architecture. However, inspection reveals the existing 12-inch wire mesh trays are already at 75% fill capacity and cannot support new bundles.

DCDC Engineering Recommendation

The consultant designs a complete migration to a Top-of-Rack (ToR) switching architecture:

1. Decommissioning: All existing horizontal copper cabling and EoR patch panels are completely removed, freeing 100% of the overhead cable tray capacity.
2. ToR Leaf Deployment: Each server cabinet is equipped with two 48-port 25GbE Leaf switches mounted in RU 41 and RU 42.
3. In-Rack Patching: All servers connect to the ToR switches using $1.5\text{ m}$ passive Direct Attach Copper (DAC) twinaxial cables. Server patch cabling is contained 100% within the rack.
4. Overhead Fiber Uplinks: Each ToR switch is connected to the central spine fabric using two 100GbE OS2 single-mode duplex links. Instead of 640 heavy copper cables traversing the row, only eighty (80) duplex fiber links (160 strands) in a few lightweight fiber cables enter the overhead pathway.
5. Outcome: Overhead weight loading drops by 90%, airflow obstruction above cabinets is eliminated, and the row achieves 25 Gbps line-rate capability at a fraction of the structured cabling labor cost.

---

Common Exam Traps & DCDC Pitfalls

• Port Stranding Math: Questions frequently ask why ToR increases capital costs for switches despite lower cabling costs. The answer is port stranding: unused ports in a cabinet cannot be easily shared with servers in adjacent cabinets without violating in-rack cabling limits.
• Cable Tray Fill Rule: ANSI/TIA-569 recommends designing cable tray for about 25% calculated fill at installation and caps tray fill at 50% (with a 150 mm / 6 in maximum cable depth). The 40% figure is the conduit fill limit for three or more cables. Answers recommending 60% or 75% tray fill exceed the standard.
• DAC Distance Limitations: Assuming Direct Attach Copper (DAC) can be used across an entire server row to connect servers to an EoR cabinet. Passive DAC at 25G/100G is limited to about $3\text{ meters}$ (roughly 5 m with FEC); it cannot span the 20 to 50 meters required for EoR.
• Category 8 Distance Ceiling: Confusing Category 6A distance limits ($100\text{ m}$) with Category 8 limits. Category 8 balanced twisted-pair copper is restricted to a $30\text{-meter 2-connector channel}$ ($24\text{ m}$ horizontal cable plus $6\text{ m}$ total patch cords), making it suitable primarily for ToR, MoR, or very short EoR rows.
• Confusing MoR with EoR Benefits: Assuming MoR requires more cable than EoR. MoR reduces maximum cable length by 50% and splits cable tray volume into two directions, preventing extreme tray fill.