7.3 NVIDIA Container Toolkit & Kubernetes GPU Operator

7.3 NVIDIA Container Toolkit & Kubernetes GPU Operator

Executive Summary: Running accelerated AI workloads inside containerized environments requires bridging the boundary between the host Linux kernel and container namespaces. Standard Linux containers isolate user processes from physical hardware, preventing access to specialized PCIe accelerators. The NVIDIA Container Toolkit provides the low-level libraries, OCI runtime hooks, and Container Device Interface (CDI) specifications required to expose GPUs to containers safely. To manage these components at scale across hundreds of Kubernetes nodes, NVIDIA developed the NVIDIA GPU Operator and NVIDIA Network Operator—declarative Kubernetes operators that automate the entire lifecycle of drivers, container runtimes, device plugins, telemetry exporters, and high-speed InfiniBand networking.

---

1. The Container GPU Acceleration Architecture

In standard Linux containerization, containers share the host operating system kernel while isolating user-space filesystems, process trees, and network namespaces. Because GPU computing requires direct communication with kernel-level driver modules (nvidia.ko, nvidia-uvm.ko), containers cannot access GPU accelerators by default.

               NVIDIA CONTAINER ACCELERATION STACK

  ┌────────────────────────────────────────────────────────────────────────┐
  │               CONTAINER APPLICATION LAYER (PyTorch, TensorFlow)        │
  │      CUDA Runtime (cudart) │ cuDNN │ TensorRT │ User Application       │
  ├────────────────────────────────────────────────────────────────────────┤
  │               CONTAINER ISOLATION BOUNDARY (OCI Container)             │
  │   Injected Device Nodes: /dev/nvidia0..N, /dev/nvidia-uvm, /dev/nvidiactl│
  │   Injected User-Space Driver Libraries: libcuda.so.1, libnvidia-ml.so.1 │
  ├────────────────────────────────────────────────────────────────────────┤
  │          NVIDIA CONTAINER TOOLKIT (libnvidia-container / nvidia-ctk)   │
  ├────────────────────────────────────────────────────────────────────────┤
  │      CONTAINER RUNTIME ENGINE (containerd / CRI-O / Docker Engine)     │
  ├────────────────────────────────────────────────────────────────────────┤
  │         HOST LINUX KERNEL SPACE (NVIDIA Display Driver & UVM)          │
  │      nvidia.ko │ nvidia-uvm.ko │ nvidia-modeset.ko │ nvidia-peermem.ko  │
  ├────────────────────────────────────────────────────────────────────────┤
  │                  PHYSICAL ACCELERATOR HARDWARE                         │
  │      NVIDIA H100 / A100 / B200 SXM GPUs & NVSwitch Fabric              │
  └────────────────────────────────────────────────────────────────────────┘

The NVIDIA Design Philosophy

NVIDIA's container architecture enforces a strict separation between the User Space and the Kernel Space:

• Host Kernel Space: Only the base NVIDIA kernel modules (nvidia.ko, unified memory driver nvidia-uvm.ko, and modeset drivers) are loaded into the host kernel.
• Container Image Space: Container images (such as NGC PyTorch containers) package application code, deep learning frameworks, CUDA runtime libraries, and CUDA-X acceleration libraries (cuBLAS, cuDNN, NCCL). Container images never include kernel drivers.
• Dynamic Driver Injection: When a container launches, the NVIDIA Container Toolkit dynamically mounts the matching host user-space driver libraries (libcuda.so, libnvidia-ml.so) and physical device character nodes (/dev/nvidia*) directly into the container namespace.

---

2. NVIDIA Container Toolkit Components & CDI

The NVIDIA Container Toolkit comprises modular utilities that integrate seamlessly with Open Container Initiative (OCI) compliant runtime engines (such as containerd, CRI-O, and Docker).

                  CONTAINER TOOLKIT COMPONENT HIERARCHY

  ┌────────────────────────────────────────────────────────────────────────┐
  │           nvidia-ctk (NVIDIA Container Toolkit CLI Utility)            │
  │    - Configures container runtimes     - Generates CDI specifications  │
  │    - Validates device nodes & modes    - Manages runtime hooks         │
  └───────────────────────────────────┬────────────────────────────────────┘
                                      │ Generates / Invokes
  ┌───────────────────────────────────▼────────────────────────────────────┐
  │       nvidia-container-runtime (OCI Runtime Shim Wrapper)              │
  │    - Intercepts OCI container lifecycle (prestart / create hooks)     │
  │    - Delegates execution to standard low-level runtimes (runc / crun) │
  └───────────────────────────────────┬────────────────────────────────────┘
                                      │ Executes
  ┌───────────────────────────────────▼────────────────────────────────────┐
  │         libnvidia-container (Low-Level C Library & CLI)                │
  │    - Configures Linux cgroups (device whitelisting)                    │
  │    - Mounts driver libraries, device nodes, and IPC sockets            │
  └────────────────────────────────────────────────────────────────────────┘

Core Toolkit Utilities

1. libnvidia-container: A dedicated C library and CLI binary (nvidia-container-cli) that provides a well-defined API for injecting NVIDIA GPUs into Linux containers. It directly configures Linux device control groups (cgroups) and securely binds host libraries into container mount namespaces.
2. nvidia-container-runtime: An OCI-compliant runtime shim that wraps standard low-level runtimes (such as runc). It inspects container specifications for NVIDIA-specific environment variables (e.g., NVIDIA_VISIBLE_DEVICES=all) and modifies the container configuration before passing execution to runc.
3. nvidia-ctk (Container Toolkit CLI): The modern unified command-line tool used to configure container runtimes, inspect devices, and manage Container Device Interface (CDI) configurations.

The Transition to Container Device Interface (CDI)

Historically, GPU injection relied on custom OCI prestart hooks that required runtime-specific hacks. The cloud-native ecosystem has standardized on the Container Device Interface (CDI)—an open specification for container runtimes to interact with third-party devices.

Using nvidia-ctk, administrators generate a declarative CDI specification file (located at /etc/cdi/nvidia.yaml or /var/run/cdi/nvidia.yaml):

# Generate the CDI specification for all detected NVIDIA GPUs
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

With CDI configured, container engines inject GPUs natively using standard device flags without requiring custom runtime wrappers:

# Running a container with CDI device injection in Podman/Docker
podman run --rm --device nvidia.com/gpu=all nvcr.io/nvidia/cuda:12.6.0-base-ubuntu24.04 nvidia-smi

---

3. NVIDIA GPU Operator Architecture

In enterprise Kubernetes clusters spanning hundreds of accelerated worker nodes, manually installing kernel headers, matching NVIDIA drivers, configuring container runtimes, deploying device plugins, and updating monitoring daemons across heterogeneous nodes creates immense operational overhead.

The NVIDIA GPU Operator automates the complete lifecycle management of all software components required to provision and operate GPU nodes in Kubernetes.

                      NVIDIA GPU OPERATOR ARCHITECTURE

  ┌────────────────────────────────────────────────────────────────────────┐
  │                   Kubernetes ClusterPolicy CRD                         │
  │    Declarative configuration governing all node GPU components         │
  └───────────────────────────────────┬────────────────────────────────────┘
                                      │ Deploys & Reconciles
  ┌───────────────────────────────────▼────────────────────────────────────┐
  │                   NVIDIA GPU Operator Controller Daemon                │
  └───────┬──────────────┬──────────────┬──────────────┬─────────────┬─────┘
          │              │              │              │             │
  ┌───────▼──────┐┌──────▼──────┐┌──────▼──────┐┌──────▼──────┐┌────▼─────┐
  │ Node Feature ││ NVIDIA      ││ Container   ││ K8s Device  ││ DCGM     │
  │ Discovery    ││ Driver      ││ Toolkit     ││ Plugin & GFD││ Exporter │
  │ (NFD) Daemon ││ Container   ││ DaemonSet   ││ DaemonSets  ││ (Metrics)│
  └──────────────┘└─────────────┘└─────────────┘└─────────────┘└──────────┘

The ClusterPolicy Custom Resource

The entire GPU Operator stack is governed declaratively through a single Kubernetes Custom Resource Definition (CRD) called ClusterPolicy. Administrators specify desired versions, driver configurations, and feature toggles in the ClusterPolicy manifest; the operator controller continuously reconciles cluster state to match.

Managed Component Stack

1. Node Feature Discovery (NFD): Automatically inspects worker node hardware via PCI bus queries and labels nodes equipped with NVIDIA accelerators (e.g., feature.node.kubernetes.io/pci-10de.present=true).
2. GPU Feature Discovery (GFD): Queries GPU hardware properties and generates granular Kubernetes node labels detailing GPU architecture, product model, memory capacity, compute capability, and active MIG geometry (e.g., nvidia.com/gpu.product=NVIDIA-H100-80GB-HBM3, nvidia.com/gpu.count=8).
3. NVIDIA Driver Container: Compiles or loads pre-compiled NVIDIA kernel modules (nvidia.ko, nvidia-uvm.ko) directly on containerized nodes without modifying the host OS root filesystem. If a worker node's Linux kernel updates, the driver container automatically rebuilds the matching driver module upon reboot.
4. NVIDIA Container Toolkit DaemonSet: Installs the container toolkit binaries on the node and automatically configures the local container runtime (containerd or CRI-O) to support GPU acceleration.
5. NVIDIA Kubernetes Device Plugin: Implements the Kubernetes Device Plugin API, discovering available GPUs and advertising them as allocatable cluster capacity (nvidia.com/gpu: 8). When a pod requests a GPU, the device plugin assigns the physical device and manages cgroup access.
6. NVIDIA DCGM Exporter: Runs the Data Center GPU Manager engine (nv-hostengine) and exports hundreds of granular GPU telemetry metrics (utilization, SM activity, memory usage, NVLink throughput, temperature, power, XID errors) into Prometheus format on port 9400.
7. NVIDIA MIG Manager: Dynamically reconfigures GPU hardware into declarative Multi-Instance GPU (MIG) slices on supported Ampere/Hopper/Blackwell GPUs without requiring manual node cordoning or reboots.
8. KubeVirt GPU Add-on: Manages PCI pass-through and vGPU assignment for virtual machines running alongside containers in OpenShift or KubeVirt environments.

---

4. NVIDIA Network Operator & High-Speed Secondary Networking

In distributed multi-node AI training, inter-GPU communication across nodes requires direct access to high-speed InfiniBand or RoCE network interfaces via GPUDirect RDMA. Standard Kubernetes networking plugins (such as Flannel or Calico) provide single-interface overlay networks (VXLAN/Geneve) designed for pod-to-pod web traffic, introducing severe CPU overhead and packet encapsulation latency.

The NVIDIA Network Operator automates the deployment of high-performance networking stacks, enabling pods to attach directly to high-speed secondary InfiniBand/RoCE fabrics.

                 NVIDIA NETWORK OPERATOR COMPONENT STACK

  ┌────────────────────────────────────────────────────────────────────────┐
  │               POD APPLICATION (Distributed PyTorch / NCCL)             │
  │   Primary Interface (eth0 - K8s API) │ Secondary Interfaces (net1..net8│
  │   via Calico/Cilium CNI              │ InfiniBand / RoCE GPUDirect RDMA│
  ├──────────────────────────────────────┴─────────────────────────────────┤
  │                  MULTUS CNI (Multi-Network Meta-Plugin)                │
  ├────────────────────────────────────────────────────────────────────────┤
  │   SR-IOV Network Device Plugin   │    RDMA Shared Device Plugin        │
  ├──────────────────────────────────┴─────────────────────────────────────┤
  │           MOFED / DOCA DRIVER CONTAINER (OFED Host Kernel Drivers)     │
  ├────────────────────────────────────────────────────────────────────────┤
  │           PHYSICAL NETWORK HARDWARE (ConnectX-7 / BlueField-3 DPUs)    │
  └────────────────────────────────────────────────────────────────────────┘

Core Network Operator Components

1. MOFED / DOCA Driver Container: Declaratively compiles and loads host Mellanox OpenFabrics Enterprise Distribution (OFED) drivers and kernel modules on worker nodes equipped with NVIDIA ConnectX-6/7 NICs or BlueField-2/3 DPUs.
2. Multus CNI Plugin: A container network interface (CNI) meta-plugin that acts as an interface aggregator. It allows a single Kubernetes pod to have multiple network interfaces simultaneously: eth0 for cluster control plane communication, and net1 through net8 for high-throughput, low-latency inter-GPU compute clustering.
3. SR-IOV & RDMA Shared Device Plugins: Discovers physical InfiniBand/Ethernet PCIe adapters and advertises them as allocatable Kubernetes resources (e.g., nvidia.com/rdma_shared_device_a). Passes hardware Virtual Functions (VFs) or RDMA character devices (/dev/infiniband/uverbs*) directly into pod namespaces.
4. Secondary IPAM Plugins (Whereabouts / MACVLAN): Manages dynamic cluster-wide IP address allocation across the secondary InfiniBand/RoCE network fabric without requiring external DHCP servers.