9.1 GPU Telemetry, nvidia-smi & DCGM Architecture

9.1 GPU Telemetry, nvidia-smi & DCGM Architecture

Core Objective: Operating enterprise AI infrastructure requires continuous, low-overhead observability into hardware health, computational efficiency, power consumption, and interconnect integrity. System administrators must master both immediate command-line diagnostic tools (nvidia-smi) and enterprise-scale telemetry engines (NVIDIA Data Center GPU Manager / DCGM) to monitor multi-node accelerated clusters, distinguish between superficial utilization and actual compute saturation, and rapidly diagnose low-level GPU hardware faults.

---

1. nvidia-smi & NVIDIA Management Library (NVML)

The NVIDIA System Management Interface (nvidia-smi) is a command-line utility built directly on top of the NVIDIA Management Library (NVML)—a C-based programmatic API for monitoring and managing various states of NVIDIA GPU devices. While nvidia-smi is ubiquitous for interactive administration, understanding its underlying operational flags, query capabilities, and configuration parameters is critical for enterprise AI operations.

+-----------------------------------------------------------------------------+
|                        NVIDIA TELEMETRY ARCHITECTURE                        |
|                                                                             |
|   [ User / CLI ]          [ Cluster Tools / Python ]      [ Prometheus ]    |
|         │                             │                          │          |
|   `nvidia-smi`                  `pydcgm` API               DCGM Exporter    |
|         │                             │                          │          |
|         ▼                             ▼                          ▼          |
|   ┌────────────┐             ┌──────────────────┐       ┌────────────────┐  |
|   │ NVML (C)   │             │ `nv-hostengine`  │       │  Port :9400    │  |
|   └─────┬──────┘             │  (DCGM Daemon)   │       │  /metrics      │  |
|         │                    └────────┬─────────┘       └────────────────┘  |
|         │                             │                                     |
|         └──────────────┬──────────────┘                                     |
|                        ▼                                                    |
|              ┌──────────────────┐                                           |
|              │  NVIDIA Kernel   │ (nvidia.ko, nvidia-uvm.ko)                |
|              │     Drivers      │                                           |
|              └────────┬─────────┘                                           |
|                        ▼                                                    |
|   ┌─────────────────────────────────────────────────────────────────────┐   |
|   │ NVIDIA Enterprise GPUs (Hopper H100 / Blackwell B200 / SXM5 / NVLink)│   |
|   └─────────────────────────────────────────────────────────────────────┘   |
+-----------------------------------------------------------------------------+

Persistence Mode (nvidia-smi -pm)

Under standard Linux operating system behavior, the NVIDIA kernel driver module (nvidia.ko) is dynamically loaded into memory when an application initializes a CUDA context and is completely unloaded when the last CUDA process terminates.

• The Problem: In fast-paced AI training pipelines, batch inference servers, or Slurm job queues with short tasks, repeatedly loading and unloading the kernel driver introduces 1 to 2 seconds of latency per invocation. Furthermore, it triggers repeated PCIe bus re-enumeration, resets hardware state registers, and can cause intermittent device initialization failures.
• The Solution: Enabling Persistence Mode ensures that the NVIDIA kernel driver remains permanently resident in system memory, even when no client applications or CUDA processes are active.

  # Enable persistence mode across all installed GPUs
  sudo nvidia-smi -pm 1
  
  # Verify persistence mode status
  nvidia-smi --query-gpu=gpu_name,persistence_mode --format=csv
  

• Production Best Practice: Rather than running manual CLI commands, production deployments configure the nvidia-persistenced background daemon via systemd (systemctl enable --now nvidia-persistenced.service), ensuring zero driver tear-down across cluster reboots.

Advanced Querying & Scripting Options

Interactive nvidia-smi output provides a human-readable visual summary, but automated orchestration scripts require deterministic, structured metrics. Using the --query-gpu flag combined with --format=csv allows administrators to extract precise telemetry parameters without parsing text tables:

# Query core telemetry parameters in CSV format without headers or unit strings
nvidia-smi --query-gpu=timestamp,index,name,utilization.gpu,utilization.memory,\
memory.total,memory.used,memory.free,temperature.gpu,power.draw,power.limit,\
clocks.current.graphics,clocks.current.memory,pstate \
--format=csv,noheader,nounits

GPU Compute Modes (nvidia-smi -c)

NVIDIA GPUs support configurable Compute Modes that determine how many host processes and user threads can concurrently establish CUDA execution contexts on a specific physical device:

1. Default Mode (0 / DEFAULT): Multiple host processes and multi-threaded applications can concurrently allocate memory and launch CUDA kernels on the same GPU. The hardware time-slices execution among active processes unless Multi-Instance GPU (MIG) or Multi-Process Service (MPS) is configured.
2. Exclusive Process Mode (3 / EXCLUSIVE_PROCESS): Restricts the GPU so that only one single host process can initialize a CUDA context at any given time. However, that single process may spawn multiple concurrent threads. If a second process attempts to allocate CUDA resources on the device, the driver immediately throws an all CUDA-capable devices are busy or unavailable error. This mode is widely used in high-performance computing (HPC) and Slurm batch clusters to prevent unintentional job interference on shared nodes.
3. Prohibited Mode (2 / PROHIBITED): Completely blocks any process from initializing a CUDA context on the GPU. Useful for isolating faulty or thermally compromised GPUs pending administrative maintenance.

# Set all GPUs to Exclusive Process mode
sudo nvidia-smi -c EXCLUSIVE_PROCESS

# Set GPU index 0 back to Default mode
sudo nvidia-smi -i 0 -c DEFAULT

Hardware Control: Power Management & Clock Locking

nvidia-smi allows administrators to enforce hardware-level constraints to prevent thermal runaway, equalize performance across cluster nodes, or stabilize execution during benchmarking:

• Power Limiting (-pl): Enforces a maximum wattage threshold within the device's hardware-defined minimum and maximum power envelope. On an NVIDIA H100 SXM5 with a 700W default TDP, administrators can restrict power draw during high-density facility constraints:

  # Set maximum power limit to 500 Watts on GPU 0
  sudo nvidia-smi -i 0 -pl 500
  

• Graphics & Memory Clock Locking (-lgc, -lmc): Under dynamic load, modern GPUs automatically adjust core frequencies using GPU Boost algorithms, fluctuating based on instantaneous temperature and power headroom. While beneficial for peak throughput, frequency fluctuations introduce runtime non-determinism during latency-sensitive inference benchmarking. Administrators can lock clocks to fixed values:

  # Lock graphics core clock between 1200 MHz (min) and 1800 MHz (max)
  sudo nvidia-smi -i 0 -lgc 1200,1800
  
  # Lock graphics clock to a static 1500 MHz frequency
  sudo nvidia-smi -i 0 -lgc 1500
  
  # Reset graphics clock locking back to dynamic factory defaults
  sudo nvidia-smi -i 0 -rgc
  

---

2. NVIDIA Data Center GPU Manager (DCGM) Architecture

While nvidia-smi is effective for single-server queries, relying on it for enterprise cluster monitoring presents severe architectural limitations:

• Process Execution Overhead: Executing nvidia-smi forks a new Linux process, parses driver state tables, and performs synchronous, blocking NVML API calls. Running high-frequency cron scripts or polling loops consumes significant host CPU cycles and can induce driver lock contention.
• Lack of Micro-Architectural Depth: nvidia-smi cannot report detailed Streaming Multiprocessor (SM) sub-pipeline metrics, Tensor Core active cycles, fine-grained NVLink replay errors, or PCIe transaction throughput.

To overcome these limitations, NVIDIA developed the Data Center GPU Manager (DCGM). DCGM is an enterprise-grade suite of tools, background daemons, and APIs designed specifically for continuous, low-overhead (<1% system overhead) telemetry, comprehensive diagnostic testing, health monitoring, and automated policy enforcement across large-scale GPU supercomputing clusters.

                    ┌─────────────────────────────────────────────────┐
                    │                DCGM ARCHITECTURE                │
                    └─────────────────────────────────────────────────┘

        [ Cluster Orchestrators ]    [ Monitoring Agent ]    [ User / Admin ]
          (Kubernetes / Slurm)         (DCGM Exporter)            (CLI)
                   │                          │                     │
                   │ (C / Python API)         │ (HTTP :9400)        │ (`dcgmi`)
                   ▼                          ▼                     ▼
        ┌─────────────────────────────────────────────────────────────┐
        │                      CLIENT INTERFACES                      │
        └──────────────────────────────┬──────────────────────────────┘
                                       │ TCP Port 5555 / IPC Socket
                                       ▼
        ┌─────────────────────────────────────────────────────────────┐
        │            `nv-hostengine` (DCGM Daemon Process)            │
        │  ┌──────────────────┐  ┌──────────────────┐  ┌────────────┐  │
        │  │ Telemetry Engine │  │ Health Monitor   │  │ Diagnostics│  │
        │  │ (Async Collector)│  │ (Watchdog Engine)│  │ (L1 - L4)  │  │
        │  └────────┬─────────┘  └────────┬─────────┘  └─────┬──────┘  │
        └───────────┼─────────────────────┼──────────────────┼────────┘
                    │                     │                  │
                    └─────────────────────┼──────────────────┘
                                          ▼
                        ┌───────────────────────────────────┐
                        │   NVIDIA Management Library (NVML)│
                        └─────────────────┬─────────────────┘
                                          ▼
                        ┌───────────────────────────────────┐
                        │       NVIDIA Kernel Driver        │
                        └─────────────────┬─────────────────┘
                                          ▼
                        ┌───────────────────────────────────┐
                        │ Physical GPUs / NVSwitch Hardware │
                        └───────────────────────────────────┘

Core Daemon: nv-hostengine

At the center of DCGM is nv-hostengine, an asynchronous background service running on the host system (systemctl start nvidia-dcgm). nv-hostengine acts as the centralized data collector and hardware abstraction layer:

• It runs persistent, low-overhead background polling threads that sample internal GPU performance counters, thermal sensors, and error registers at microsecond intervals.
• It buffers telemetry in memory, allowing external clients to query historical and instantaneous metrics without triggering synchronous driver calls.

Deployment Modes: Standalone vs. Embedded

DCGM can be deployed in two architectural operational modes:

DCGM Command-Line Interface (dcgmi)

The dcgmi utility provides administrative access to nv-hostengine for configuration and immediate status checks:

• dcgmi discovery -l: Discovers and enumerates all physical GPUs, NVSwitches, and MIG entities managed by DCGM.
• dcgmi group -c <group_name>: Creates logical groupings of GPUs (e.g., grouping GPUs assigned to a specific tenant or job).
• dcgmi fieldgroup -c <fg_name> -f <field_ids>: Defines customized telemetry field groups for optimized polling.
• dcgmi stats --enable / --show: Collects and displays fine-grained compute, memory, and PCIe statistics over a workload's lifecycle.
• dcgmi health -c / -g: Configures and evaluates automated background health monitoring systems.
• dcgmi diag -r <level>: Executes hardware diagnostic verification suites (Levels 1–4).

---

3. Core Telemetry Metrics: Utilization vs. True Profiling

A critical competency for AI infrastructure engineers is understanding the difference between high-level activity metrics and deep silicon-level hardware saturation.

+-----------------------------------------------------------------------------+
|                   GPU UTILIZATION vs. HARDWARE SATURATION                   |
+----------------------------+------------------------------------------------+
| HIGH-LEVEL METRIC          | SILICON-LEVEL PROFILING METRIC                 |
+----------------------------+------------------------------------------------+
| `nvidia-smi` GPU Util (%)  | DCGM Streaming Multiprocessor (SM) Occupancy   |
| - Measures temporal kernel | - Measures actual active warps relative to     |
|   activity on GPU device   |   architectural maximum warp capacity          |
| - 1 thread active = 100%   | - True indicator of compute saturation         |
+----------------------------+------------------------------------------------+
| Framebuffer Memory Used    | Memory Bandwidth Utilization (DRAM Active)     |
| - Total GiB allocated      | - Measures % of memory controller cycles spent |
| - Static buffer footprint  |   reading/writing to HBM (e.g., 3.35 TB/s)     |
+----------------------------+------------------------------------------------+
| Unspecified Compute Pipe   | Tensor Core Active Pipe Ratio (FP16/BF16/FP8)  |
| - Cannot distinguish FP32  | - Measures % of cycles Tensor Core Matrix      |
|   scalar from Tensor Core  |   Multiply Accumulate (MMA) units execute math |
+----------------------------+------------------------------------------------+

1. GPU Utilization (%) vs. SM Occupancy

• Superficial GPU Utilization (utilization.gpu): Represents the percentage of time over a sample period (typically 1 second) during which at least one CUDA kernel was executing on the GPU. If an unoptimized algorithm executes a single thread on a single Streaming Multiprocessor (SM) of an H100 GPU (leaving the other 131 SMs completely idle), nvidia-smi will report 100% GPU Utilization, presenting a false illusion of complete resource saturation.
• DCGM SM Active (DCGM_FI_PROF_SM_ACTIVE): The fraction of time that at least one warp is active on each SM, averaged across all SMs on the chip.
• DCGM SM Occupancy (DCGM_FI_PROF_SM_OCCUPANCY): The ratio of active warps executing on the SMs to the theoretical maximum number of warps that could physically reside on those SMs simultaneously. True high-performance deep learning training workloads typically achieve >70-80% SM Occupancy.

2. Tensor Core Utilization (DCGM_FI_PROF_PIPE_TENSOR_ACTIVE)

Measures the fraction of cycles that the GPU's specialized Tensor Cores / Matrix Multiply-Accumulate (MMA) pipelines are actively executing instructions. In deep learning training and inference, if utilization.gpu is 100% but DCGM_FI_PROF_PIPE_TENSOR_ACTIVE is near 0%, it indicates that the application is erroneously executing scalar FP32/INT32 operations on standard CUDA cores rather than leveraging mixed-precision (FP16, BF16, FP8) Tensor Core acceleration.

3. Memory Subsystem & Bandwidth Metrics

• DRAM Active (DCGM_FI_PROF_DRAM_ACTIVE): The percentage of time the High Bandwidth Memory (HBM3/HBM3e) controllers are actively reading or writing data. For memory-bound operations (such as LLM autoregressive token decoding or large recommendation model embedding lookups), this metric identifies memory bus bottlenecks.
• PCIe / NVLink Throughput Counters:
  • DCGM_FI_PROF_PCIE_TX_BYTES / RX_BYTES: Real-time byte transfer rates across the host PCIe root complex.
  • DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL: Real-time aggregate bidirectional bandwidth across all NVLink ports. Essential for detecting imbalanced inter-GPU communication during distributed Data Parallel (DDP) or Tensor Parallel (TP) All-Reduce operations.

---

4. Hardware Diagnostic Codes: NVIDIA XID Errors

When a hardware anomaly, illegal driver operation, or severe memory corruption occurs within an NVIDIA GPU subsystem, the NVIDIA kernel driver generates a specific XID Error Code and writes it directly to the Linux kernel ring buffer (dmesg / /var/log/messages).

XID errors represent the primary low-level diagnostic signal used by infrastructure engineers and automated cluster watchdog agents to isolate failing hardware.

+-----------------------------------------------------------------------------------------+
|                         CRITICAL NVIDIA XID ERROR CODES                                 |
+------+-------------------------+----------------------------------+---------------------+
| XID  | ERROR CLASSIFICATION    | ROOT CAUSE & DESCRIPTION         | REMEDIATION ACTION  |
+------+-------------------------+----------------------------------+---------------------+
| 31   | GPU Memory Page Fault   | Illegal memory access by CUDA    | Software fix: debug |
|      | (MMU Fault)             | kernel (out-of-bounds / nullptr) | application code    |
+------+-------------------------+----------------------------------+---------------------+
| 43   | GPU Stopped Processing  | Driver timeout / hardware engine | Restart application |
|      | (Engine Hang)           | hang / firmware state stall      | or reset GPU driver |
+------+-------------------------+----------------------------------+---------------------+
| 45   | Preemptive Reset Fired  | Driver forced engine recovery to | Inspect logs for    |
|      |                         | clear locked hardware channel    | thermal/driver bugs |
+------+-------------------------+----------------------------------+---------------------+
| 62   | Internal Microcontroller| GPU System Processor (GSP) or    | Cold reboot / check |
|      | / Firmware Halt         | internal firmware crash          | VBIOS firmware      |
+------+-------------------------+----------------------------------+---------------------+
| 79   | GPU Fallen Off the Bus  | Fatal PCIe link drop / power rail| Immediate node drain|
|      | (Fatal Hardware Loss)   | failure / unrecoverable hardware | & physical hardware/|
|      |                         | fault (device unreadable)        | PCIe RMA replacement|
+------+-------------------------+----------------------------------+---------------------+

Detailed Analysis of Critical XID Codes

• XID 31 (GPU Memory Page Fault): The GPU Memory Management Unit (MMU) detected an illegal address access from a running CUDA thread (e.g., accessing an unallocated device pointer, buffer overflow, or reading freed memory). This is almost exclusively an application software bug, not a hardware failure.
• XID 43 (GPU Stopped Processing): A hardware engine on the GPU failed to complete an assigned task within the driver timeout period, causing the driver to declare an engine hang. Can be caused by infinite loops in CUDA kernels, power throttling, or corrupted command buffers.
• XID 62 (Internal Microcontroller / GSP Error): Modern GPUs (Turing, Ampere, Hopper, Blackwell) offload driver and power tasks to an embedded RISC-V GPU System Processor (GSP). An XID 62 indicates that the GSP firmware has halted unexpectedly, requiring a host reboot or driver reinitialization.
• XID 79 (GPU Fallen Off the Bus): The most severe hardware fault encountered in data center operations. The host PCIe root complex has completely lost communication with the GPU board (e.g., PCIe link training failure, power delivery voltage drop, or catastrophic ASIC failure). The GPU disappears from lspci and nvidia-smi. The host node must be immediately cordoned, drained, and scheduled for physical hardware inspection or RMA.