9.2 DCGM Health Checks, Diagnostics & Prometheus Monitoring

9.2 DCGM Health Checks, Diagnostics & Prometheus Monitoring

Operational Blueprint: In large-scale accelerated computing clusters comprising hundreds or thousands of GPUs, silent hardware degradation, intermittent NVLink CRC errors, and uncorrectable memory faults directly threaten training job completion and inference reliability. Implementing an automated, proactive monitoring framework requires mastering DCGM diagnostic tiers, real-time health watchdogs, and cloud-native Prometheus/Grafana alerting pipelines with automated node remediation.

---

1. DCGM Diagnostic Levels (dcgmi diag)

NVIDIA DCGM includes an integrated, multi-tiered diagnostic testing suite designed to validate hardware functionality, stress computational and memory subsystems, and detect latent component failures. Diagnostics can be invoked via the CLI (dcgmi diag -r <level>), programmatic APIs, or automated node provisioning scripts.

+-----------------------------------------------------------------------------------------+
|                              DCGM DIAGNOSTIC EXECUTION TIERS                            |
+-------+-------------------+---------------+---------------------------------------------+
| LEVEL | TIER NAME         | TYPICAL DURATION | TESTS PERFORMED & PRIMARY USE CASE       |
+-------+-------------------+---------------+---------------------------------------------+
| 1     | Short / Sanity    | < 30 Seconds  | - Driver initialization & NVML validation   |
|       |                   |               | - PCIe link generation & width check (x16)  |
|       |                   |               | - NVLink link detection & basic topology    |
|       |                   |               | * Use Case: Pre-job validation / Slurm prolog|
+-------+-------------------+---------------+---------------------------------------------+
| 2     | Medium / Deploy   | ~1 – 2 Minutes| - All Level 1 tests                         |
|       |                   |               | - Targeted GPU memory allocation & read/write|
|       |                   |               | - Memory bandwidth sanity verification      |
|       |                   |               | * Use Case: Post-driver install / Node boot |
+-------+-------------------+---------------+---------------------------------------------+
| 3     | Long / Stress     | ~10 – 15 Min  | - All Level 2 tests                         |
|       |                   |               | - High-intensity GEMM compute stress test   |
|       |                   |               | - Full CUDA memory test (DRAM pattern test) |
|       |                   |               | - Full-bandwidth NVLink peer stress test    |
|       |                   |               | * Use Case: Node commissioning & burn-in    |
+-------+-------------------+---------------+---------------------------------------------+
| 4     | Extended Burn-In  | > 30 Minutes  | - Deep memory stress under thermal saturation|
|       |                   | (Customizable)| - Prolonged power cycling & pulse testing   |
|       |                   |               | * Use Case: Deep RCA / Flaky node RMA check |
+-------+-------------------+---------------+---------------------------------------------+

Running DCGM Diagnostics

Administrators can execute diagnostics against specific physical GPUs or entire system groups:

# Run a Level 1 quick sanity check on all GPUs
dcgmi diag -r 1

# Run a Level 3 comprehensive stress test on GPU group 'training_nodes'
dcgmi diag -g <group_id> -r 3

# Run a Level 3 test with specific custom parameters (e.g., targeted stress time)
dcgmi diag -r 3 --parameter "gemm.test_duration=300;memory.test_duration=300"

Operational Integration: Slurm & Kubernetes Epilog/Prolog

• Pre-Job Check (Prolog): Slurm clusters execute dcgmi diag -r 1 during the job prolog script. If a GPU fails link verification or driver checks, the job is prevented from launching, and the scheduler flags the node for administrative inspection without failing the user's workload.
• Post-Job Verification (Epilog): After a multi-day distributed training run finishes, the epilog script checks for residual ECC errors or memory leaks before returning the node to the available scheduling pool.

---

2. DCGM Health Monitoring & Watchdogs

Unlike active diagnostic tests (which require GPUs to be idle and take exclusive control of compute engines), the DCGM Health Monitor runs continuously in the background during live production workloads with near-zero performance impact.

+-----------------------------------------------------------------------------+
|                        DCGM HEALTH MONITORING SUBSYSTEM                     |
|                                                                             |
|   ┌───────────────────┐   ┌───────────────────┐   ┌───────────────────┐     |
|   │    PCIe HEALTH    │   │   NVLINK HEALTH   │   │   MEMORY HEALTH   │     |
|   │ - Link Width deg. │   │ - Link Down state │   │ - SBE Thresholds  │     |
|   │ - Link Speed deg. │   │ - CRC Error rate  │   │ - DBE Fatal Error │     |
|   │ - Replay Rollback │   │ - Recovery Failure│   │ - Dynamic Page Ret│     |
|   └───────────────────┘   └───────────────────┘   └───────────────────┘     |
|   ┌───────────────────┐   ┌───────────────────┐   ┌───────────────────┐     |
|   │  THERMAL HEALTH   │   │   POWER HEALTH    │   │  HARDWARE STATUS  │     |
|   │ - Thermal Throttl.│   │ - Voltage Drop    │   │ - Inforom Corrupt │     |
|   │ - Critical Trip   │   │ - Power Cap Viol. │   │ - MCU / GSP State │     |
|   └───────────────────┘   └───────────────────┘   └───────────────────┘     |
+-----------------------------------------------------------------------------+

Enabling and Configuring Health Watches

Health monitoring watches are configured per GPU group using bitmasks or letter identifiers:

# Enable all health monitoring watches on GPU group 0
dcgmi health -g 0 -s pmtnda

# Query the instantaneous health status of GPU group 0
dcgmi health -g 0 -c

Where -s flags correspond to:

• p: PCIe Watchdog (monitors bus link degradation, e.g., link dropping from 16x PCIe Gen 5 down to 8x or Gen 1 speeds).
• m: Memory Watchdog (monitors uncorrectable DBEs and excessive SBE accumulation).
• t: Thermal Watchdog (monitors thermal slowdown and shutdown threshold violations).
• n: NVLink Watchdog (monitors physical link failure, recovery retries, and high CRC error rates).
• d: Driver / GSP Watchdog (monitors driver responsiveness and kernel exception states).
• a: All Watches Enabled.

Memory Reliability: Single-Bit (SBE) vs. Double-Bit (DBE) ECC Errors

Modern enterprise GPUs (Hopper H100, Blackwell B200) utilize Error-Correcting Code (ECC) protection across High Bandwidth Memory (HBM) and SRAM caches:

---

3. Production Monitoring Stack: DCGM Exporter, Prometheus & Grafana

In modern cloud-native AI platforms, DCGM is integrated into the standardized Prometheus observability ecosystem via the NVIDIA DCGM Exporter.

+-----------------------------------------------------------------------------------------+
|                        CLOUD-NATIVE AI MONITORING PIPELINE                              |
|                                                                                         |
|  ┌──────────────────┐      ┌─────────────────┐      ┌────────────┐      ┌────────────┐  |
|  │ NVIDIA H100 Node │      │  DCGM Exporter  │      │ Prometheus │      │  Grafana   │  |
|  │  nv-hostengine   │ ───► │   (DaemonSet)   │ ───► │   Server   │ ───► │ Dashboards │  |
|  │  (NVML Sampling) │      │  Port :9400/met │      │ (Scraper)  │      │ & Displays │  |
|  └──────────────────┘      └─────────────────┘      └─────┬──────┘      └────────────┘  |
|                                                           │                             |
|                                                           ▼                             |
|                                                    ┌──────────────┐                     |
|                                                    │ Alertmanager │                     |
|                                                    └──────┬───────┘                     |
|                                                           │ Webhook                     |
|                                                           ▼                             |
|                                                    ┌──────────────┐                     |
|                                                    │ Node Drainer │                     |
|                                                    │  Controller  │                     |
|                                                    └──────────────┘                     |
+-----------------------------------------------------------------------------------------+

DCGM Exporter Architecture

• Deployed as a Kubernetes DaemonSet (part of the NVIDIA GPU Operator).
• Periodically queries nv-hostengine using low-overhead APIs and formats raw metrics into OpenMetrics / Prometheus text format exposed over HTTP on port 9400 (/metrics).
• Configured via a customizable CSV field mapping file (/etc/dcgm-exporter/default-counters.csv), allowing administrators to select exact telemetry IDs to expose without overloading Prometheus time-series storage.
• Automatically enriches metrics with Kubernetes container, pod, namespace, and GPU instance metadata by correlating device UUIDs with container runtime cgroups.

Key Prometheus DCGM Metrics

---

4. Kubernetes Integration: Automated Node Cordoning & Draining

When managing large clusters, manual intervention upon hardware failure is too slow to prevent cascading job failures in distributed training runs. Enterprise AI platforms implement Automated Node Remediation:

                       AUTOMATED REMEDIATION WORKFLOW

   ┌────────────────────────────────────────────────────────────────┐
   │ 1. HARDWARE ANOMALY DETECTED                                   │
   │    DCGM Exporter captures fatal event:                         │
   │    - Uncorrectable Double-Bit Error (DBE > 0)                  │
   │    - GPU Fallen off the Bus (XID 79)                           │
   │    - NVLink Port Failure                                       │
   └───────────────────────────────┬────────────────────────────────┘
                                   │
                                   ▼
   ┌────────────────────────────────────────────────────────────────┐
   │ 2. PROMETHEUS ALERT FIRED                                      │
   │    Alertmanager triggers automated webhook to Kubernetes       │
   │    Remediation Controller.                                     │
   └───────────────────────────────┬────────────────────────────────┘
                                   │
                                   ▼
   ┌────────────────────────────────────────────────────────────────┐
   │ 3. NODE TAINT & CORDON                                         │
   │    Node is immediately cordoned (`kubectl cordon <node>`) and   │
   │    tainted with `nvidia.com/gpu.unhealthy=true:NoSchedule` to   │
   │    block any incoming workloads.                               │
   └───────────────────────────────┬────────────────────────────────┘
                                   │
                                   ▼
   ┌────────────────────────────────────────────────────────────────┐
   │ 4. GRACEFUL POD DRAIN                                          │
   │    Active pods are evicted (`kubectl drain <node> --ignore-    │
   │    daemonsets`). Workload scheduler re-schedules training jobs │
   │    from their latest saved checkpoint on a healthy node.       │
   └───────────────────────────────┬────────────────────────────────┘
                                   │
                                   ▼
   ┌────────────────────────────────────────────────────────────────┐
   │ 5. ISOLATED DIAGNOSTIC EXECUTION                               │
   │    Node runs `dcgmi diag -r 3` in isolation. If test fails,     │
   │    an automated RMA ticket is dispatched to NVIDIA Enterprise. │
   └────────────────────────────────────────────────────────────────┘