5.2 GPUDirect Technologies: P2P, RDMA & Storage (GDS)

5.2 GPUDirect Technologies: P2P, RDMA & Storage (GDS)

Core Concept: In enterprise AI training and high-throughput inference serving, feeding massive datasets and synchronizing multi-terabyte model weights can easily saturate host CPU cores, system DRAM buses, and operating system kernel buffers. The NVIDIA GPUDirect family of technologies—encompassing GPUDirect P2P, GPUDirect RDMA, and GPUDirect Storage (GDS)—establishes direct, hardware-accelerated Direct Memory Access (DMA) paths across GPUs, network adapters, and NVMe storage devices, completely bypassing host CPU memory and operating system overhead.

---

1. The Legacy I/O Dilemma & The Bounce Buffer Bottleneck

To appreciate the necessity of GPUDirect, one must analyze the traditional I/O path implemented in legacy operating systems and standard PCIe architectures.

┌────────────────────────────────────────────────────────────────────────┐
│                     THE LEGACY HOST-STAGED DATA PATH                   │
│                                                                        │
│     ┌────────────────────────────────────────────────────────────┐     │
│     │                   HOST CPU & SYSTEM DRAM                   │     │
│     │                                                            │     │
│     │   ┌─────────────────────┐        ┌─────────────────────┐   │     │
│     │   │ Kernel Page Cache / │◄──────►│ User-Space Buffer /  │   │     │
│     │   │ Kernel Bounce Buffer│        │ Host Staging Memory │   │     │
│     │   └──────────▲──────────┘        └──────────┬──────────┘   │     │
│     └──────────────┼──────────────────────────────┼──────────────┘     │
│                    │ Step 1: DMA to Host DRAM     │ Step 2: DMA to GPU │
│           PCIe Bus │ (CPU Context Switch)         │ (CPU Copy Overhead)│
│                    ▼                              ▼                    │
│          ┌───────────────────┐          ┌───────────────────┐          │
│          │ NVMe Storage /    │          │     GPU Memory    │          │
│          │ Network HCA (NIC) │          │       (HBM)       │          │
│          └───────────────────┘          └───────────────────┘          │
│                                                                        │
│   Disadvantages: 2x PCIe traversals, CPU cache poisoning, high latency │
└────────────────────────────────────────────────────────────────────────┘

The Anatomy of a Legacy Transfer

When an application without GPUDirect transfers data between a storage device (or network adapter) and GPU memory, the data must undergo a multi-hop copy cycle:

1. Device to Host Kernel Memory: The storage controller or network interface card (NIC) executes a DMA transfer across the PCIe bus into an OS kernel-space page cache or intermediate memory buffer located in Host CPU DRAM.
2. Kernel to User Space Copy: The CPU executes a synchronous memory copy (memcpy) to move the data from kernel space into the application's pinned user-space host buffer.
3. Host to GPU Device Memory: The CUDA driver issues a second DMA transfer across the PCIe bus (cudaMemcpyHostToDevice) to stream the data from CPU DRAM into the GPU's High-Bandwidth Memory (HBM).

Structural Penalties of Legacy I/O

• Double PCIe Bus Traversal: Data must traverse the PCIe bus twice—first from the source peripheral into host RAM, and second from host RAM into the GPU. This cuts effective PCIe throughput in half (e.g., from 63 GB/s down to ~30 GB/s on PCIe Gen 5).
• Severe CPU Core Saturation: Host CPU cores spend up to 70–90% of their compute cycles managing memory copies, handling page faults, and processing operating system context switches rather than executing application logic or data preprocessing.
• CPU Cache Poisoning: Pumping gigabytes of streaming video, image, or tensor data through host CPU L1/L2/L3 caches evicts critical application code and metadata from the cache hierarchy.
• High, Unpredictable Latency: Staging through kernel buffers introduces operating system scheduling jitter, increasing end-to-end transfer latency to 15–50+ microseconds.

---

2. GPUDirect Peer-to-Peer (P2P)

GPUDirect P2P (Peer-to-Peer) enables two GPUs within the same physical host to directly access, read, and write to each other's memory address spaces without staging data through host CPU system DRAM.

┌────────────────────────────────────────────────────────────────────────┐
│                     GPUDIRECT PEER-TO-PEER (P2P)                       │
│                                                                        │
│               ┌──────────────────────────────────────┐                 │
│               │        HOST CPU & SYSTEM DRAM        │                 │
│               │        (Completely Bypassed!)        │                 │
│               └──────────────────────────────────────┘                 │
│                                                                        │
│                    DIRECT NVLINK OR PCIE P2P PATH                      │
│             ┌──────────────────────────────────────────┐               │
│             │                                          │               │
│             ▼                                          ▼               │
│     ┌───────────────┐                          ┌───────────────┐       │
│     │     GPU 0     │◄════════════════════════►│     GPU 1     │       │
│     │ Memory (HBM)  │   Direct Memory Access   │ Memory (HBM)  │       │
│     └───────────────┘   (Sub-Microsecond Lat)  └───────────────┘       │
└────────────────────────────────────────────────────────────────────────┘

Architectural Mechanics of GPUDirect P2P

• Unified Virtual Addressing (UVA): Introduced in CUDA 4.0, UVA maps host CPU memory and all GPU device memories within a single 64-bit virtual address space. A pointer address uniquely identifies which physical device hosts the memory.
• Direct DMA Transfers: When P2P is enabled via cudaDeviceEnablePeerAccess(), GPU DMA copy engines directly issue read and write requests to the target GPU's physical Base Address Registers (BARs) across PCIe switches or NVLink connections.
• Zero-Copy Remote Memory Access: Kernel threads executing on GPU 0 can directly dereference pointers located in GPU 1's HBM. The GPU memory controller translates the access into direct NVLink or PCIe load/store transactions.

Operational Benefits

• Sub-Microsecond Latency: Eliminating CPU involvement drops intra-node GPU-to-GPU transfer latency to under 0.5–1.0 $\mu$s over PCIe, and under 100–200 ns over NVLink.
• Full Interconnect Bandwidth: Achieves 100% of the underlying link bandwidth (up to 900 GB/s on Hopper NVLink 4, or 1.8 TB/s on Blackwell NVLink 5) without suffering from CPU memory bandwidth contention.

---

3. GPUDirect RDMA (Remote Direct Memory Access)

In multi-node AI clusters, GPUs across distinct servers must exchange gradients and activations across a high-speed network fabric (InfiniBand or RoCE Ethernet). GPUDirect RDMA extends direct memory access across the network interface card (NIC / HCA).

┌────────────────────────────────────────────────────────────────────────┐
│                        GPUDIRECT RDMA DATA PATH                        │
│                                                                        │
│    [ SERVER NODE 1 ]                              [ SERVER NODE 2 ]    │
│  ┌──────────────────┐                            ┌──────────────────┐  │
│  │ Host CPU / DRAM  │ (Bypassed)      (Bypassed) │ Host CPU / DRAM  │  │
│  └──────────────────┘                            └──────────────────┘  │
│           │                                                │           │
│     ┌─────▼─────┐                                    ┌─────▼─────┐     │
│     │ GPU 0 HBM │                                    │ GPU 0 HBM │     │
│     └─────┬─────┘                                    └─────▲─────┘     │
│           │ Direct PCIe DMA                    Direct PCIe │           │
│     ┌─────▼───────────────┐                    ┌───────────┴─────┐     │
│     │ ConnectX-7 HCA (NIC)├════════════════════┤ConnectX-7 HCA(NIC│    │
│     └─────────────────────┘ InfiniBand Fabric  └─────────────────┘     │
│                             (400 Gbps NDR)                             │
└────────────────────────────────────────────────────────────────────────┘

How GPUDirect RDMA Works

1. PCIe Peer-to-Peer Addressing: The network adapter (such as an NVIDIA ConnectX-7 InfiniBand HCA or BlueField-3 DPU) maps the GPU's memory pages into its own PCIe address space using standard PCIe Base Address Register (BAR) windowing and Address Translation Services (ATS).
2. Hardware-to-Hardware Transfers: When GPU 0 on Node 1 sends a tensor to GPU 0 on Node 2:
   • The ConnectX HCA's DMA engine directly reads the tensor out of Node 1's GPU HBM across the local PCIe bus.
   • The HCA encapsulates the payload into InfiniBand / RoCE packets and transmits them across the network fabric at line rate (e.g., 400 Gbps NDR or 800 Gbps XDR).
   • The receiving ConnectX HCA on Node 2 receives the packets and writes the payload directly into Node 2's GPU HBM via PCIe DMA.
3. Complete CPU Isolation: Neither Node 1's CPU nor Node 2's CPU touches the data payload, allocates temporary buffers, or handles page table migrations.

GPUDirect RDMA vs. Traditional Network Copy

---

4. GPUDirect Storage (GDS) & The cuFile API

Modern foundation model training and multimodal retrieval-augmented generation (RAG) pipelines require reading massive datasets—comprising terabytes to petabytes of high-resolution images, video frames, audio waveforms, and tokenized text embeddings—from high-speed storage subsystems.

GPUDirect Storage (GDS) establishes a direct DMA data path between local NVMe solid-state drives (SSDs) or remote NVMe-over-Fabrics (NVMe-oF) storage arrays and GPU High-Bandwidth Memory (HBM).

┌────────────────────────────────────────────────────────────────────────┐
│                     GPUDIRECT STORAGE (GDS) ARCHITECTURE               │
│                                                                        │
│     ┌────────────────────────────────────────────────────────────┐     │
│     │                   HOST CPU & SYSTEM DRAM                   │     │
│     │    [ OS Page Cache ]   [ CPU Memory Buffer ]  (BYPASSED!)  │     │
│     └────────────────────────────────────────────────────────────┘     │
│                                                                        │
│                    DIRECT PCIE / NVLINK DMA DATA PATH                  │
│        ┌───────────────────────────────────────────────────────┐       │
│        │                                                       │       │
│        ▼                                                       ▼       │
│  ┌───────────┐            Direct DMA via cuFile          ┌───────────┐ │
│  │ NVMe SSD  │◄═════════════════════════════════════════►│  GPU HBM  │ │
│  │ Storage   │    (Up to 10x Throughput / 90% CPU Free)   │  Memory   │ │
│  └───────────┘                                           └───────────┘ │
└────────────────────────────────────────────────────────────────────────┘

The cuFile API & Software Stack

GDS replaces standard POSIX filesystem system calls (read(), write(), pread()) with NVIDIA's specialized cuFile API (part of the CUDA driver and NVIDIA Magnum IO library suite):

// Conceptual cuFile Workflow for GPUDirect Storage
#include <cuFile.h>

CUfileDescr_t descr;
CUfileHandle_t handle;

// 1. Initialize cuFile driver ecosystem
cuFileDriverOpen();

// 2. Register file descriptor with cuFile
descr.handle.fd = open("/mnt/nvme/training_data.bin", O_RDONLY | O_DIRECT);
descr.type = CU_FILE_HANDLE_TYPE_OPAQUE_FD;
cuFileHandleRegister(&handle, &descr);

// 3. Register GPU target buffer address in HBM
cuFileBufRegister(gpu_hbm_ptr, buffer_size, 0);

// 4. Perform direct DMA read from NVMe to GPU HBM (Zero CPU bounce buffer)
cuFileRead(handle, gpu_hbm_ptr, buffer_size, file_offset, 0);

// 5. Cleanup
cuFileBufDeregister(gpu_hbm_ptr);
cuFileHandleDeregister(handle);
cuFileDriverClose();

Architectural Mechanics of GDS

1. Direct NVMe DMA: When cuFileRead() is called, the NVMe controller reads data blocks directly from flash storage and pushes them across the PCIe switch fabric directly into the physical BAR address space of the target GPU's HBM.
2. Bypassing the OS Page Cache: GDS mandates direct I/O (O_DIRECT), bypassing Linux page cache buffering. This prevents cache eviction of OS binaries and eliminates CPU memory copy overhead.
3. Distributed NVMe-oF Support: GDS seamlessly extends over remote storage architectures utilizing NVMe-over-Fabrics (NVMe-oF) with RoCEv2 or InfiniBand, enabling disaggregated storage arrays (such as DDN, VAST Data, or Weka) to stream data directly into GPU clusters at hundreds of gigabytes per second.

GDS Performance Impact Matrix

---

5. NVIDIA Magnum IO: The Unified Acceleration Framework

NVIDIA Magnum IO is the overarching I/O architecture and software suite that organizes GPUDirect technologies into an optimized, enterprise-grade framework for AI infrastructure:

┌────────────────────────────────────────────────────────────────────────┐
│                      NVIDIA MAGNUM IO SOFTWARE SUITE                   │
│                                                                        │
│  ┌───────────────────┐  ┌────────────────────┐  ┌───────────────────┐  │
│  │  Magnum IO Storage│  │Magnum IO Networking│  │Magnum IO Compute  │  │
│  │  - GPUDirect GDS  │  │  - GPUDirect RDMA  │  │  - GPUDirect P2P  │  │
│  │  - cuFile API     │  │  - NCCL / SHARP    │  │  - NVLink Fabric  │  │
│  │  - NVMe-oF Drivers│  │  - DOCA Libs (DPU) │  │  - Unified Memory │  │
│  └───────────────────┘  └────────────────────┘  └───────────────────┘  │
└────────────────────────────────────────────────────────────────────────┘

• Magnum IO Storage: Encompasses GPUDirect Storage, cuFile, and parallel filesystem integrations, maximizing throughput between storage fabrics and GPU HBM.
• Magnum IO Networking: Houses GPUDirect RDMA, NVIDIA Collective Communications Library (NCCL), and Scalable Hierarchical Aggregation and Reduction Protocol (SHARP), coordinating inter-node collective transfers across InfiniBand and Ethernet fabrics.
• Magnum IO Compute / Management: Manages NVLink topology discovery, Unified Virtual Addressing (UVA), and DPU offloading via NVIDIA DOCA.