6.1 NVIDIA DGX Systems & HGX Architecture

6.1 NVIDIA DGX Systems & HGX Architecture

Executive Summary: Modern foundation model training and extreme-scale AI inference require purpose-built hardware architectures that eliminate bottlenecks across compute, memory bandwidth, intra-node GPU-to-GPU communications, and network I/O. NVIDIA DGX systems serve as the industry gold standard for integrated AI appliances, combining custom carrier baseboards (HGX), multi-terabyte NVLink switching fabrics, high-core-count host processors, and high-speed network interfaces into dense, factory-integrated supercomputing nodes.

---

1. The NVIDIA DGX System Lineage

NVIDIA introduced the DGX platform to provide an integrated, turnkey hardware and software environment optimized specifically for deep learning workflows. Prior to DGX, organizations assembled multi-GPU servers using standard PCIe expansion slots, resulting in severe I/O bottlenecks, thermal throttling, and complex driver configuration challenges.

                      EVOLUTION OF NVIDIA DGX PLATFORMS

  ┌────────────────┐      ┌────────────────┐      ┌────────────────┐      ┌────────────────┐
  │    DGX-1       │      │    DGX A100    │      │    DGX H100    │      │ DGX GB200 NVL72│
  │ (Pascal/Volta) │ ───► │   (Ampere)     │ ───► │   (Hopper)     │ ───► │  (Blackwell)   │
  │ 8x V100 SXM2   │      │ 8x A100 SXM4   │      │ 8x H100 SXM5   │      │ 72x B200 GPUs  │
  │ 300 GB/s NVLink│      │ 600 GB/s NVLink│      │ 900 GB/s NVLink│      │ 1.8 TB/s NVLink│
  │ 3.2 kW Power   │      │ 6.5 kW Power   │      │ 10.2 kW Power  │      │ 120 kW Power   │
  └────────────────┘      └────────────────┘      └────────────────┘      └────────────────┘

DGX A100 (Ampere Generation)

Introduced in 2020, the DGX A100 established the multi-instance and high-bandwidth interconnect paradigms that define modern AI infrastructure:

• GPU Compute: 8x NVIDIA A100 Tensor Core GPUs in SXM4 form factor (initially 40GB HBM2, later upgraded to 80GB HBM2e).
• NVLink Interconnect: 6x second-generation NVSwitch chips providing 12 NVLink links per GPU, achieving 600 GB/s bidirectional bandwidth per GPU and 4.8 TB/s aggregate bidirectional bandwidth across the node.
• Host Processing: Dual AMD EPYC 7742 processors (64 cores / 128 threads each, 2.25 GHz base / 3.4 GHz boost, 128 total physical cores).
• System Memory: 2 TB DDR4-3200 ECC system DRAM across 32 DIMM slots.
• Networking: 8x single-port NVIDIA ConnectX-6 200 Gbps HDR InfiniBand / 200GbE VPI adapters for inter-GPU compute clustering, plus 1x dual-port ConnectX-6 VPI adapter for dedicated storage traffic, and 2x 10GbE onboard ports for out-of-band management.
• Internal Storage: 2x 1.92 TB NVMe M.2 SSDs for the operating system, plus 4x 3.84 TB NVMe U.2 data-cache drives (15.36 TB in RAID 0) for temporary scratch datasets.
• Physical & Power Profile: 6U rack-mount chassis, weighing ~271 lbs (123 kg), with a maximum continuous power consumption of 6.5 kW delivered via 6x (3+3 redundant) 3000W power supply units.

DGX H100 (Hopper Generation)

Introduced in 2022, the DGX H100 represents a massive leap in floating-point compute density, precision versatility via the Transformer Engine, and I/O throughput:

• GPU Compute: 8x NVIDIA H100 Tensor Core GPUs in SXM5 form factor, each equipped with 80GB HBM3 memory (3.35 TB/s memory bandwidth per GPU).
• NVLink Interconnect: 4x third-generation NVSwitch chips providing 18 NVLink 4 links per GPU, delivering 900 GB/s bidirectional bandwidth per GPU and 7.2 TB/s aggregate bidirectional bandwidth across the node.
• Host Processing: Dual 4th Gen Intel Xeon Platinum 8480C processors (56 cores / 112 threads each, 2.0 GHz base / 3.8 GHz turbo, 112 total physical cores).
• System Memory: 2 TB DDR5-4800 ECC system DRAM across 32 DIMM slots.
• Networking: 8x single-port NVIDIA ConnectX-7 400 Gbps NDR InfiniBand / OSFP adapters for compute fabric clustering, plus 2x dual-port 400 Gbps BlueField-3 DPUs or ConnectX-7 VPI adapters for high-speed storage access and cluster management.
• Internal Storage: 8x 3.84 TB NVMe U.2 SSDs configured in RAID 0 (30.7 TB total scratch space).
• Physical & Power Profile: 8U rack-mount chassis, weighing ~287 lbs (130 kg), with a maximum continuous power consumption of 10.2 kW delivered via 6x (3+3 redundant) 3300W titanium-efficiency power supplies.

DGX B200 & DGX GB200 NVL72 (Blackwell Generation)

The Blackwell generation bifurcates into standalone enterprise node appliances and rack-scale supercomputing platforms:

• DGX B200 Appliance: Features 8x NVIDIA B200 Tensor Core GPUs delivering 1,440 GB (1.44 TB) total HBM3e memory — 180 GB per GPU at 8.0 TB/s each, for 64 TB/s of aggregate memory bandwidth. Two fifth-generation NVSwitch chips knit the GPUs into a 14.4 TB/s aggregate NVLink domain at 1.8 TB/s bidirectional per GPU. Hosted by dual Intel Xeon Platinum 8570 processors (112 cores total), 2 TB of DDR5 (configurable to 4 TB), 2x 1.92 TB NVMe M.2 OS drives plus 8x 3.84 TB NVMe U.2 data-cache drives, and networked by 4 OSFP ports serving 8 single-port ConnectX-7 VPI adapters at up to 400 Gb/s each, plus 2 dual-port BlueField-3 DPUs. It draws up to 14.3 kW in a 10U chassis from six 3,300 W power supplies in a 5+1 redundant configuration.
• DGX GB200 NVL72 Rack-Scale System: Connects 36 NVIDIA Grace CPUs and 72 NVIDIA Blackwell GPUs into a single monolithic liquid-cooled rack. The system integrates 18 compute trays (each housing 2 Grace CPUs and 4 Blackwell GPUs) and 9 NVLink Switch trays (housing 18 fifth-generation NVSwitch chips). The entire 72-GPU fabric acts as a unified computing domain with 130 TB/s aggregate NVLink bandwidth, 13.5 TB of coherent HBM3e memory, and 30 TB of high-speed system memory, drawing up to 120 kW per rack under 100% direct-to-chip liquid cooling.

---

2. Technical Comparison Matrix: Enterprise DGX Generations

---

3. HGX Baseboard Architecture vs. PCIe Add-In Cards (AIC)

When designing AI data center infrastructure, engineers must choose between systems built on NVIDIA HGX Carrier Baseboards (utilizing SXM form factors) and standard servers utilizing PCIe Add-In Cards (AIC). Understanding the architectural differences between these form factors is essential for evaluating performance, thermal envelopes, and cluster scalability.

The HGX SXM Baseboard Architecture

The HGX Baseboard is a specialized, high-density carrier board manufactured by NVIDIA and integrated by certified Original Equipment Manufacturers (OEMs, such as Dell PowerEdge XE9680, HPE Cray XD670, Supermicro GPU SuperServers, and Lenovo ThinkSystem).

Key architectural characteristics of the HGX SXM design include:

1. SXM Mezzanine Sockets: GPUs attach via high-density surface-mount mezzanine connectors rather than traditional PCIe edge connectors. These multi-thousand-pin connectors provide dedicated signal paths for high-frequency NVLink differential pairs, high-voltage power pins, and diagnostic telemetry.
2. Integrated NVSwitch Crossbars: The HGX baseboard embeds NVSwitch silicon directly onto the PCB, routing high-speed copper traces directly between the SXM sockets and switch crossbars. This establishes an all-to-all non-blocking matrix where every GPU communicates with every other GPU at maximum line rate (e.g., 900 GB/s per GPU on H100).
3. Direct Power Delivery & Thermal Envelope: HGX baseboards integrate multi-phase Voltage Regulator Modules (VRMs) directly beneath and adjacent to the SXM sockets, delivering stable current capable of supporting 700W continuous Thermal Design Power (TDP) on H100 SXM5 and up to 1000W+ on B200 SXM6.

PCIe Add-In Card (AIC) Form Factor Limitations

Standard PCIe enterprise GPUs (such as the H100 NVL PCIe or A100 PCIe) conform to standard dual-slot, full-height, full-length (FHFL) physical form factors for installation into generic enterprise server motherboards.

Key constraints of PCIe add-in cards include:

1. Power Constraints: PCIe slots supply a maximum of 75W through the motherboard connector, with additional power provided via auxiliary 8-pin EPS or 16-pin 12V-2x6 / 12VHPWR cables. Total card power is capped between 300W and 350W–450W, resulting in lower sustained GPU core clocks, fewer active Tensor Cores, and reduced overall floating-point throughput compared to SXM variants.
2. Interconnect Bandwidth Deficit: PCIe add-in cards cannot accommodate on-board NVSwitch fabrics. GPU-to-GPU communication either relies on physical top-mounted NVLink bridge connectors (which only bridge adjacent pairs of 2 GPUs, capping bandwidth at 600 GB/s) or traverses the host PCIe Gen 5 bus (capped at 128 GB/s bidirectional). In 8-GPU PCIe systems without bridges, all inter-GPU traffic must traverse host PCIe root complexes and system DRAM, creating severe communication bottlenecks during distributed AllReduce operations.
3. Thermal Density: Because PCIe cards rely on restricted server chassis airflow channels and narrower heatsinks, continuous multi-node model training can induce thermal throttling under sustained multi-hour workloads.

+-----------------------------------------------------------------------------------------+
|                        SXM BASEBOARD vs. PCIe ADD-IN CARD (AIC)                         |
+----------------------------+-----------------------------+------------------------------+
| ARCHITECTURAL METRIC       | HGX SXM BASEBOARD (H100)    | PCIe ADD-IN CARD (H100 NVL)  |
+----------------------------+-----------------------------+------------------------------+
| Form Factor Connection     | High-density SXM5 Mezzanine | Dual-Slot FHFL PCIe Gen 5    |
| Max TDP / Power Delivery   | 700W (Continuous)           | 350W – 400W (Max)            |
| Interconnect Architecture  | 4x On-board NVSwitch Fabric | 2-Way Bridge or Host PCIe    |
| All-to-All GPU Bandwidth   | 900 GB/s (Full 8-GPU Mesh)  | 128 GB/s (over PCIe Gen 5)   |
| Scale-Up Clustering Scope  | 8 GPUs Non-Blocking         | 2 GPUs (via NVLink Bridge)   |
| Target Workloads           | Distributed LLM Training    | Inference / Light Fine-Tuning|
+----------------------------+-----------------------------+------------------------------+

---

4. Enterprise Form Factor Deployment Tiers

NVIDIA packages accelerated computing into distinct form factor tiers tailored to specific operational environments:

+-----------------------------------------------------------------------------------------+
|                        ENTERPRISE ACCELERATED COMPUTING TIERS                           |
|                                                                                         |
|  [ DGX STATION ]               [ DGX ENTERPRISE SERVER ]       [ DGX SUPERPOD ]         |
|  - Office / Lab Deployment     - Data Center 19" Rack          - Multi-Rack Supercomputer│
|  - Whisper-Quiet Refrigeration - 6.5 kW - 14.3 kW Density      - Scalable Units (32 nodes)│
|  - Standard 100-240V AC Outlet - 3-Phase 480V/415V Power       - Non-Blocking IB Fabric │
|  - 4x SXM GPUs (Workstation)   - 8x SXM GPUs (6U-10U)          - Hundreds/Thousands GPUs│
+-----------------------------------------------------------------------------------------+

1. DGX Station (Workstation / Office Tier)

• Operational Profile: Designed for research laboratories, corporate offices, and engineering workgroups that require supercomputing capabilities without access to a dedicated raised-floor data center.
• Acoustic & Power Engineering: Operates on standard 100V–240V single-phase AC commercial wall power. Incorporates closed-loop liquid-to-air refrigeration, maintaining acoustic emissions below 37 dB (whisper-quiet under full load).
• Representative System (DGX Station A100): 4x A100 SXM4 80GB GPUs (320GB total HBM2e) wired as a fully connected NVLink mesh — no NVSwitch is used at this scale — plus a single 64-core AMD EPYC 7742 CPU, 512GB system RAM, and dual 10GbE network ports.

2. DGX Enterprise Server Appliances (Data Center Tier)

• Operational Profile: Standard 19-inch rack-mount appliances (6U for DGX A100, 8U for DGX H100, 10U for DGX B200) engineered for high-density enterprise data centers with hot-aisle/cold-aisle containment and high-CFM airflow.
• Power & Networking: Powered by 3-phase 480V/415V redundant power feeds. Equipped with 8 discrete compute NICs mapping directly to internal GPUs, 2 storage NICs/DPUs, and dedicated BMC/IPMI out-of-band management.
• Target Deployment: Standalone departmental AI training clusters, large-scale enterprise model fine-tuning, and high-throughput multi-tenant inference pools.

3. DGX SuperPOD (Turnkey AI Supercomputer Tier)

• Operational Profile: Standardized, modular supercomputing clusters scaling from 32 nodes (256 GPUs) up to hundreds of nodes (thousands of GPUs).
• Integrated Infrastructure: Combines DGX servers, multi-tier non-blocking Quantum InfiniBand or Spectrum-X networking fabrics, all-flash parallel storage subsystems (Lustre/VAST/Weka), and automated cluster management software (Base Command Manager).