6.4 Environmental Criteria and Close-Coupled/Liquid Cooling: ASHRAE TC 9.9, In-Row, RDHx, and Liquid/Immersion

6.4 Environmental Criteria and Close-Coupled/Liquid Cooling: ASHRAE TC 9.9, In-Row, RDHx, and Liquid/Immersion

For over two decades, mission-critical thermal management was governed by conventional raised-floor perimeter cooling designed to satisfy conservative environmental ranges. However, two revolutionary transformations have permanently altered the data center thermal landscape:

1. The ASHRAE TC 9.9 Environmental Guidelines: The shift from arbitrary room temperatures to psychrometrically rigorous operating envelopes measured directly at the intake face of IT equipment, proving that servers operate reliably at higher temperatures and broader humidity bands.
2. The High-Density Computing Revolution: The exponential growth of Artificial Intelligence (AI), Machine Learning (ML), and High-Performance Computing (HPC). Thermal dissipation has surged past the thermodynamic limits of air cooling ($30\text{ to }40+\text{ kW}$ per rack), mandating close-coupled cooling, Direct-to-Chip (DTC) liquid loops, and liquid immersion.

---

ASHRAE TC 9.9 Thermal Guidelines for Data Processing Environments

Formulated by the American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE) Technical Committee 9.9, these guidelines represent the international consensus between IT hardware manufacturers (OEMs) and facility engineering consultants.

The Recommended vs. Allowable Envelopes

ASHRAE distinguishes sharply between Recommended operating boundaries (which optimize long-term component reliability) and Allowable operating boundaries (which permit temporary operational excursions to maximize economizer hours).

+---------------------------------------------------------------------------------------------------------+
|                                 ASHRAE TC 9.9 THERMAL ENVELOPES (2021)                                  |
+---------------------+-------------------------------+---------------------------------------------------+
| Parameter           | Recommended Envelope          | Allowable Class A1 (Strict Enterprise)            |
+---------------------+-------------------------------+---------------------------------------------------+
| Dry-Bulb Temp       | 18°C to 27°C (64.4°F to 80.6°F)| 15°C to 32°C (59.0°F to 89.6°F)                   |
| Humidity Limits     | -9°C DP to 15°C DP; max 70% RH| -12°C DP & 8% RH to 17°C DP & 80% RH              |
| Measurement Point   | Air intake face of IT chassis | Air intake face of IT chassis                     |
| Max Temp Change     | 5°C in an hour with tape;     | 5°C in an hour with tape;                         |
|                     | 20°C in an hour otherwise     | 20°C in an hour otherwise                         |
| Elevation Derating  | Derate 1°C per 300m above 900m| Derate 1°C per 300m above 900m                    |
+---------------------+-------------------------------+---------------------------------------------------+

The 70% RH upper limit applies where gaseous contamination is low (copper and silver reactivity within G1); ASHRAE lowers it to 50% RH in higher-contamination environments. Earlier editions used 60% RH.

Extended Allowable Classes (A2, A3, A4):

• Class A2 (Volume Servers / Storage): Dry-bulb 10°C to 35°C (50°F to 95°F), -12°C DP to 21°C DP, 8% to 80% RH.
• Class A3 / A4 (Ruggedized Cloud / Edge Compute): Dry-bulb up to 40°C (Class A3) or 45°C (Class A4) (104°F to 113°F).

Why Dew Point Over Relative Humidity?

ASHRAE TC 9.9 fundamentally replaced Relative Humidity (RH) with Dew Point Temperature ($T_{dp}$) as the primary moisture metric.

• Relative humidity changes constantly with air temperature: air at 15°C and 60% RH has the identical absolute moisture content as air heated to 25°C with 32% RH.
• Using RH triggers erratic and wasteful mechanical cycling: a CRAH unit sees "low RH" in warm exhaust and fires electric steam humidifiers, while another CRAH sees "high RH" in cold supply and condenses water.
• Controlling to Dew Point (-9°C to 15°C) establishes absolute moisture stability, eliminating simultaneous humidification and dehumidification.

Moisture Risk Boundaries:

1. Low Moisture Limit (Dew Point <-9°C / 15.8°F): Excessively dry air increases the risk of Electrostatic Discharge (ESD), potentially damaging delicate CMOS semiconductors during physical handling. (Mitigated by grounded static-dissipative flooring and anti-static wrist straps).
2. High Moisture Limit (Dew Point >15°C / 59°F or RH above the 70% limit): Excess moisture combined with hygroscopic dust and airborne salts promotes Conductive Anodic Filament (CAF) growth in printed circuit boards, accelerates copper and silver (creep) corrosion, and causes hygroscopic dust failures.

The X-Factor Reliability Concept

Operating IT hardware inside the expanded Allowable envelope dramatically expands annual free-cooling economizer hours. However, elevated temperatures accelerate semiconductor degradation (Arrhenius reaction rate).

ASHRAE quantifies this trade-off using the X-Factor—a mathematical failure rate multiplier normalized to a baseline of 20°C (68°F):

$\text{Component Failure Rate} = \text{Baseline Failure Rate} \times X$

+---------------------------------------------------------------------------------------------------------+
|                                   ASHRAE X-FACTOR RELIABILITY IMPACT                                    |
+----------------------------+-----------------------+----------------------------------------------------+
| Continuous Operating Temp  | Failure Rate (X-Factor)| Reliability Impact                                 |
+----------------------------+-----------------------+----------------------------------------------------+
| 20°C (68°F)                | 1.00 (Baseline)       | Standard OEM baseline reliability                  |
| 25°C (77°F)                | 1.24                  | 24% increase in annual server hardware failures    |
| 30°C (86°F)                | 1.42                  | 42% increase in annual server hardware failures    |
| 35°C (95°F)                | 1.55                  | 55% increase in annual server hardware failures    |
+----------------------------+-----------------------+----------------------------------------------------+

---

Close-Coupled Cooling Architectures

Traditional perimeter CRAH units must push air across long distances ($15\text{ to }30\text{ m}$ / 50 to 100 ft) under a raised floor. When rack densities exceed 10 kW to 15 kW per cabinet, the volumetric airflow required ($>1,500\text{ CFM}$ per rack) creates extreme underfloor static pressure drops and turbulent hot spots.

Close-coupled cooling solves this by positioning the heat exchanger directly adjacent to the server cabinets, drastically shortening the aerodynamic path to $1\text{ to }2\text{ m}$ or less.

1. In-Row Cooling Units

• Architecture: Compact CRAH/CRAC units matching the depth and height of standard server cabinets, installed directly within the equipment row between server racks.
• Airflow Path: Draws warm air directly from the contained hot aisle, cools it across a vertical chilled-water or DX coil, and discharges conditioned air directly into the cold aisle.
• Engineering Advantages:
  • Shortened air path reduces fan static pressure losses, cutting fan power consumption by 30% to 50%.
  • Electronically commutated (EC) fans modulate speed dynamically based on temperature sensors installed at neighboring server intakes.
  • Supports densities from 15 kW to 30 kW per cabinet without requiring a raised access floor.

2. Rear-Door Heat Exchangers (RDHx)

• Architecture: Replaces standard perforated rear cabinet doors with a high-density liquid-to-air cooling coil mounted directly onto the back of the 19-inch rack.
• Passive RDHx: Utilizes internal server chassis fans to push hot exhaust through the chilled water coil. The coil absorbs the heat at the point of generation, discharging room-neutral air (20°C to 24°C) into the whitespace. Supports 20 kW to 35 kW per rack with zero additional fan power.
• Active RDHx: Incorporates integrated variable-speed EC fans mounted on the rear door to overcome coil static pressure drop, supporting extreme densities up to 50 kW to 75 kW per rack.

---

Direct-to-Chip (DTC) Liquid Cooling

Above 35 kW to 40 kW per cabinet, air convection reaches its physical thermodynamic limits: air velocity must exceed hurricane force to remove the heat, creating unbearable acoustic noise ($>85\text{ dBA}$) and consuming excessive fan energy. Direct Liquid Cooling (DLC) or Direct-to-Chip (DTC) cooling replaces air heatsinks with liquid cold plates.

+---------------------------------------------------------------------------------------------------------+
|                              DIRECT-TO-CHIP (DTC) HYDRONIC ARCHITECTURE                                 |
+---------------------------------------------------------------------------------------------------------+
|                                                                                                         |
|     [ PRIMARY FACILITY LOOP (FWS) ]                [ SECONDARY TECHNOLOGY LOOP (TCS) ]                  |
|        Treated Water from Central Plant               Isolated Deionized Water / Dielectric             |
|                                                                                                         |
|    +-----------------------+                      +-----------------------+                             |
|    | Primary Chilled Water |                      | Secondary Supply Mani |                             |
|    | Supply (45°F - 85°F)  |                      | (Warm Water 90°-115°F)|                             |
|    +-----------------------+                      +-----------------------+                             |
|                |                                              |                                         |
|                v                                              v                                         |
|        +---------------+                              +---------------+                                 |
|        | Coolant       | <--- Isolated Heat Exch ---> | Cold Plate    | (Mounted on GPU / CPU)          |
|        | Distribution  |                              | Microchannels |                                 |
|        | Unit (CDU)    |                              +---------------+                                 |
|        +---------------+                                      |                                         |
|                |                                              v                                         |
|                v                                      +---------------+                                 |
|    +-----------------------+                          | Secondary     |                                 |
|    | Primary Return Water  | <----------------------- | Return Mani   | (Hot Water 120°-140°F)          |
|    | (Warm to Plant)       |                          +---------------+                                 |
|    +-----------------------+                                                                            |
|                                                                                                         |
+---------------------------------------------------------------------------------------------------------+

1. Cold Plates and Microchannels

• Copper or aluminum cold plates fitted with micro-channel fin arrays (<100 um width) are clamped directly onto CPU and GPU packages with thermal interface material (TIM).
• Liquid coolant flows directly across the micro-channels, achieving heat transfer coefficients up to 30 times higher than air convection.
• DTC typically extracts 70% to 85% of total server heat directly to the liquid loop; the remaining 15% to 30% (from power supplies, VRMs, and storage) is dissipated to ambient air and captured via secondary In-Row or RDHx units.

2. Coolant Distribution Units (CDUs)

• The CDU is a dedicated mechanical skid that establishes a hydraulic barrier between the building's Facility Water System (FWS) and the internal server Technology Cooling System (TCS).
• Functions:
  • Plate-and-Frame Isolation: Protects delicate server cold plates from raw building water, bio-fouling, and rust.
  • Pressure Regulation: Operates the secondary TCS loop at low pressure (or negative pressure/vacuum-assisted), ensuring that in the event of an unseated coupling, air is sucked into the tube rather than liquid spraying out.
  • Water Chemistry: Maintains deionized water treated with biocides, corrosion inhibitors, and ethylene/propylene glycol.

3. ASHRAE Liquid Cooling Classes (W17 to W+)

ASHRAE TC 9.9 classifies the facility water supply temperature for liquid cooling. The 2021 classes are named for their maximum supply temperature (the older W1 to W5 names are shown in parentheses):

• W17 (W1) and W27 (W2): Supply up to 17°C or 27°C; typically requires mechanical chillers, with economizer hours at W27.
• W32 (W3): Supply up to 32°C; permits chillerless operation with dry or evaporative coolers in many climates.
• W40 and W45 (W4 covered up to 45°C): Supply up to 40°C or 45°C (113°F); enables chillerless operation in nearly all climates using outdoor dry coolers.
• W+ (W5): Supply above 45°C (113°F); a high-temperature loop suited to waste heat recovery and district heating.

---

Immersion Cooling Technologies

For extreme-density computing clusters ($100\text{ to }250+\text{ kW}$ per rack) utilized in generative AI model training and computational fluid dynamics, immersion cooling eliminates heatsinks and cold plates entirely by submerging bare electronic hardware directly into a bath of non-conductive, dielectric fluid.

+---------------------------------------------------------------------------------------------------------+
|                               SINGLE-PHASE VS. TWO-PHASE IMMERSION                                      |
+-----------------------------+------------------------------------+--------------------------------------+
| Parameter                   | Single-Phase Immersion             | Two-Phase Immersion                  |
+-----------------------------+------------------------------------+--------------------------------------+
| Fluid State                 | Remains liquid throughout cycle    | Boils from liquid to vapor and back  |
| Dielectric Fluid Chemistry  | Synthetic hydrocarbons / PAO oils  | Engineered fluorochemicals (FK/PFPE) |
| Fluid Boiling Point         | Very High (>200°C / 392°F)         | Low (~50°C / 122°F)                  |
| Fluid Circulation           | Active pumps circulate hot fluid   | Natural convective vapor buoyancy    |
| External Heat Exchanger     | Liquid-to-liquid plate exchanger   | Overhead vapor condensing coil       |
| Tank Architecture           | Open-bath horizontal or vertical   | Hermetically sealed pressure vessel  |
| Typical Density Supported   | 50 kW to 100+ kW per tank          | 100 kW to 250+ kW per tank           |
| Regulatory / EHS Concerns   | Fluid maintenance & drip trays     | PFAS environmental scrutiny & cost   |
+-----------------------------+------------------------------------+--------------------------------------+

1. Single-Phase Immersion Cooling

• Mechanism: Servers are mounted vertically in horizontal bath tanks filled with clear, odorless, non-conductive synthetic hydrocarbon or polyalphaolefin (PAO) dielectric oil.
• Heat Transfer: Fluid in contact with hot semiconductor components absorbs heat via conduction and natural convection. Low-speed pumps circulate the warm dielectric oil through an external plate-and-frame heat exchanger connected to a facility water loop, returning chilled liquid to the bottom of the tank.
• Maintenance Considerations: IT technicians pull servers vertically using overhead davit cranes. Fluid does not evaporate. Drip trays and service racks allow excess oil to drain back into the tank during component replacement.

2. Two-Phase Immersion Cooling

• Mechanism: Servers are submerged in an engineered fluorochemical liquid with a precise low boiling point (typically around 50°C / 122°F).
• Phase Change Dynamics: When GPUs and CPUs heat up under load, the fluid boils directly on the silicon die surface, vaporizing into gas. Latent heat of vaporization removes immense thermal energy with zero temperature rise.
• Condensation: The rising vapor hits a water-cooled condenser coil located in the top vapor space of the sealed tank, condenses back into liquid droplets, and rains back into the bath.
• Regulatory Outlook: Boiling heat transfer gives two-phase immersion very high heat transfer coefficients. However, regulatory pressure on per- and polyfluoroalkyl substances (PFAS), 3M's exit from PFAS manufacturing (which ended production of its Novec and Fluorinert fluids by the end of 2025), and high fluid replenishment costs make single-phase immersion and direct-to-chip cooling the dominant deployment choices.

---

Technology Comparison: Air vs. Close-Coupled vs. Liquid

---

Practical Design Scenario: Designing a Heterogeneous AI Facility

Scenario Overview

A commercial colocation operator acquires a 10 MW shell building. The tenant demand profile is split:

• Zone 1 (6 MW): Standard enterprise cloud virtualization operating at 8 kW per rack.
• Zone 2 (4 MW): Advanced generative AI GPU training clusters operating at 60 kW per rack.

Engineering Solution in the BOD:

1. Zone 1 Infrastructure: Employs centralized chilled water CRAHs paired with Hot Aisle Containment (HAC). Supply air is delivered at 72°F (22.2°C), fully conforming to the ASHRAE Recommended envelope.
2. Zone 2 Infrastructure: Air cooling cannot support 60 kW/rack. The DCDC specifies closed-loop Direct-to-Chip (DTC) liquid cooling utilizing row-based CDUs. The primary facility water loop feeds the CDUs with 85°F (29.4°C) water, allowing 100% of Zone 2 cooling to operate via outdoor evaporative fluid coolers without mechanical chillers (ASHRAE Class W32, formerly W3).
3. Residual Heat Management: Because DTC captures 75% of the GPU heat, the remaining 25% (15 kW/rack) dissipates into the whitespace air. Rather than oversizing perimeter CRAHs, passive Rear-Door Heat Exchangers (RDHx) are attached to the Zone 2 cabinets, neutralizing residual air heat at the rack before it enters the room.
4. Structural & Spatial Impact: Zone 2 raised floors are eliminated; cabinets and immersion tanks rest directly on a reinforced slab rated for $24\text{ kN/m}^2$ (500 psf) with recessed drainage trenches and optical hydrocarbon leak-detection rope.

---

Common Exam Traps & DCDC Pitfalls

• Measurement Point for TC 9.9: ASHRAE TC 9.9 compliance is measured at the intake face of IT equipment (typically top, middle, and bottom of the rack), NEVER at CRAH discharge grilles, perimeter walls, or return ceiling drops.
• Rate-of-Change Limits: ASHRAE limits temperature change to 5°C (9°F) in an hour where tape drives are present and 20°C (36°F) in an hour for other (disk-based) equipment, to limit thermal stress and condensation risk.
• FWS vs. TCS Distinction: In Direct-to-Chip systems, the Facility Water System (FWS) is the primary building loop; the Technology Cooling System (TCS) is the secondary isolated server loop managed by the CDU. Never connect raw building chilled water directly to server cold plates.
• Immersion Fluid Confusion: Single-phase immersion utilizes hydrocarbons/oils that do NOT change phase; two-phase immersion utilizes fluorochemicals that boil and condense.