7.2 NVIDIA Run:ai Architecture & Dynamic Orchestration

7.2 NVIDIA Run:ai Architecture & Dynamic Orchestration

Executive Summary: In traditional enterprise AI deployments, GPU clusters suffer from the "30% Utilization Trap": data scientists manually reserve entire 8-GPU nodes for exploratory Jupyter notebooks, leave them idle overnight, and create siloed departmental clusters. NVIDIA Run:ai is an enterprise AI workload orchestration and dynamic resource management platform designed to eliminate these inefficiencies. By introducing a centralized control plane, dynamic fractional GPU slicing, and an intelligent batch scheduler that supports guaranteed and over-quota allocations with automated preemption, Run:ai transforms static GPU clusters into a centralized, highly utilized, elastic AI supercomputing pool.

---

1. The Enterprise AI Utilization Challenge

Modern enterprise AI teams face significant operational friction when sharing multi-million-dollar accelerated hardware:

• Static Resource Allocations: Researchers allocate full GPUs for exploratory data analysis, code debugging, and preprocessing—workloads that often utilize less than 10% of the GPU's compute capability and memory bandwidth.
• Siloed Hardware Infrastructure: Departmental clusters (e.g., Computer Vision, NLP, Autonomous Driving) operate independently. While the Vision team's cluster sits 80% idle over a weekend, the NLP team's large training jobs wait in days-long backlogs.
• Lack of Preemption and Checkpointing: Standard Kubernetes environments cannot safely terminate a low-priority interactive notebook to yield compute resources to an urgent, distributed foundation model training job.

                      THE GPU UTILIZATION REVOLUTION

  TRADITIONAL SILOED ALLOCATION (~30% Saturation):   RUN:AI DYNAMIC POOLING (>85% Saturation):
  ┌────────────────────────────────────────┐         ┌────────────────────────────────────────┐
  │ Node 1: Reserved for Team A (Idle)     │         │ Node 1: Team B (Guaranteed Training)   │
  │ Node 2: Jupyter Notebook (10% Util)    │  ────►  │ Node 2: Team A (Over-Quota Dynamic)    │
  │ Node 3: Queued Job Stalled (No Quota)  │         │ Node 3: 4x Fractional Workspaces       │
  │ Node 4: Team C Cluster (Offline)       │         │ Node 4: Batch Auto-Resume Checkpoint   │
  └────────────────────────────────────────┘         └────────────────────────────────────────┘

---

2. NVIDIA Run:ai Platform Architecture

The Run:ai platform architecture consists of three integrated planes designed to provide enterprise governance without sacrificing researcher velocity.

                     NVIDIA RUN:AI THREE-TIER ARCHITECTURE

  ┌────────────────────────────────────────────────────────────────────────┐
  │                    RUN:AI CONTROL PLANE (Central / SaaS)               │
  │    - Multitenancy & RBAC Engine       - Departments, Projects & Quotas │
  │    - SSO & Directory Integration      - Cluster Telemetry & Analytics  │
  └───────────────────────────────────▲────────────────────────────────────┘
                                      │ TLS Secure Sync
  ┌───────────────────────────────────▼────────────────────────────────────┐
  │                   RUN:AI CLUSTER DAEMON (In-Cluster Agent)             │
  │    - Syncs Quota Policies & Roles     - Streams Pod & Node Metrics     │
  └───────────────────────────────────▲────────────────────────────────────┘
                                      │ Local K8s API
  ┌───────────────────────────────────▼────────────────────────────────────┐
  │             RUN:AI WORKLOAD ORCHESTRATOR (Batch Scheduler Engine)      │
  │    - All-or-Nothing Gang Scheduling   - Guaranteed vs Over-Quota Logic │
  │    - Dynamic Fractional GPU Manager   - Automated Preemption Engine    │
  └────────────────────────────────────────────────────────────────────────┘

The Three Architectural Planes

1. Control Plane (Management & Policy Layer):
   • Can be deployed as a fully managed SaaS platform or hosted on-premises in air-gapped data centers.
   • Manages organizational hierarchy: Departments (divisions/business units) contain Projects (teams/initiatives).
   • Administers role-based access control (RBAC), single sign-on (SSO via SAML 2.0 / OIDC), storage credentials, and aggregated cluster-wide analytics.
2. Cluster Daemon (In-Cluster Synchronization Agent):
   • Lightweight Kubernetes operator deployed directly inside each managed AI cluster.
   • Continuously synchronizes project definitions, access tokens, and scheduling rules between the Control Plane and the local Kubernetes API server.
   • Collects fine-grained hardware metrics and workload states, streaming telemetry back to the centralized dashboard.
3. Workload Orchestrator (The Run:ai Scheduler):
   • An advanced batch scheduling engine deployed as a Kubernetes custom scheduler.
   • Replaces the default kube-scheduler for AI workloads, executing gang scheduling, topology-aware node selection, dynamic fractional GPU slicing, and priority preemption.

Developer Workflows & Self-Service Experience

Run:ai provides a unified interface across CLI, Web UI, and REST APIs, abstracting underlying Kubernetes complexities for researchers:

• Interactive Workspaces: Spin up JupyterLab, VS Code, or custom Docker environments with exact GPU fractions and pre-configured persistent volume claims (PVCs).
• Unattended Training Jobs: Submit multi-node distributed training jobs with automatic fault recovery and elastic restart.
• Inference Deployments: Deploy autoscaling inference microservices (Triton, vLLM, NIM) that scale down to zero when idle.

# Submitting a distributed training job using the Run:ai CLI
runai submit llama-train \
    --project nlp-research \
    --image nvcr.io/nvidia/pytorch:24.08-py3 \
    --gpu 8 \
    --node-pools h100-cluster \
    --command -- torchrun train.py --epochs 50

---

3. Dynamic Fractional GPU Sharing

In many AI workflows (such as exploratory data analysis, code debugging, and light inference), a workload requires only a fraction of an enterprise GPU's compute power and memory. While hardware Multi-Instance GPU (MIG) provides strict physical isolation, it requires rigid administrative pre-configuration.

Run:ai Dynamic Fractional GPU provides a dynamic, software-defined sharing layer that allows multiple containers to share a single physical GPU elastically.

                   DYNAMIC FRACTIONAL GPU vs. HARDWARE MIG

  DYNAMIC FRACTIONAL GPU (Software Layer):    HARDWARE MIG (Silicon Partitioning):
  ┌───────────────────────────────────────┐   ┌───────────────────────────────────────┐
  │       NVIDIA H100 (80GB HBM3)         │   │       NVIDIA H100 (80GB HBM3)         │
  ├───────────┬───────────┬───────────────┤   ├───────────┬───────────┬───────────────┤
  │ Pod A     │ Pod B     │ Pod C         │   │ Instance 1│ Instance 2│ Instance 3    │
  │ 0.2 GPU   │ 0.3 GPU   │ 0.5 GPU       │   │ 2g.20gb   │ 2g.20gb   │ 3g.40gb       │
  │ (16GB)    │ (24GB)    │ (40GB)        │   │ (Hardware │ (Hardware │ (Hardware     │
  │ Dynamic   │ Dynamic   │ Dynamic       │   │ Isolated) │ Isolated) │ Isolated)     │
  └───────────┴───────────┴───────────────┘   └───────────┴───────────┴───────────────┘
  * Any arbitrary fraction (0.1, 0.25, 0.5)   * Fixed hardware profile geometries
  * No node reboot / reconfiguration required * Requires administrative reconfiguration
  * Memory dynamically allocated & capped    * Full physical SM & Cache fault isolation

Fractional GPU Allocation Comparison Matrix

How Run:ai Enforces Fractional Limits

Run:ai injects a lightweight user-space interception library into the container via standard Linux environment variables (LD_PRELOAD). This interceptor intercepts CUDA driver and runtime API calls (such as cudaMalloc and cuMemAlloc), allowing the application to see only its allocated fraction of memory. If a container attempts to exceed its designated memory limit, the allocator returns an out-of-memory error to that specific container, completely protecting co-located workloads from memory corruption or crashes.

---

4. Guaranteed vs. Over-Quota Workloads & Dynamic Preemption

Run:ai resolves the tension between fair resource distribution and maximum hardware utilization through its Guaranteed vs. Over-Quota scheduling model.

                       RUN:AI WORKLOAD PRIORITY TIERS

  ┌────────────────────────────────────────────────────────────────────────┐
  │   TIER 1: GUARANTEED WORKLOADS (Non-Preemptible, High Priority)        │
  │   - Workloads running within the Project's allocated nominal quota     │
  │   - Guaranteed execution; cannot be preempted by other teams           │
  └────────────────────────────────────────────────────────────────────────┘
                                      ▲
                                      │ Can Preempt Over-Quota Work
  ┌───────────────────────────────────┴────────────────────────────────────┐
  │   TIER 2: OVER-QUOTA WORKLOADS (Opportunistic, Preemptible)            │
  │   - Workloads running beyond the Project's allocated nominal quota     │
  │   - Consumes idle cluster capacity; immediately yielded when needed    │
  └────────────────────────────────────────────────────────────────────────┘

Resource Allocation Mechanics

1. Guaranteed Quota (Nominal Allocation): The baseline number of GPUs committed to a specific project (e.g., 16 GPUs for Team Alpha). When Team Alpha submits jobs within this 16-GPU threshold, the jobs run under Guaranteed status and are strictly non-preemptible.
2. Over-Quota Allocation (Opportunistic Bursting): If Team Beta has 32 guaranteed GPUs but is only utilizing 8, there are 24 idle GPUs in the cluster. Team Alpha can submit additional jobs that burst beyond their 16-GPU quota, consuming Team Beta's idle capacity. These bursting jobs run in the Over-Quota tier.
3. Automated Fair-Share Rebalancing:
   • When Team Beta returns and submits a job requiring their guaranteed 24 GPUs, the Run:ai Workload Orchestrator identifies which project is furthest above its assigned quota.
   • The scheduler selects Team Alpha's lowest-priority over-quota workloads and initiates automated preemption.

Graceful Checkpoint-Resume Flow

Preemption must not result in corrupted training progress. Run:ai orchestrates a multi-step graceful shutdown:

  1. PREEMPTION SIGNAL  ──►  2. CHECKPOINT FLUSH  ──►  3. CONTAINER STOP  ──►  4. AUTOMATIC RESUME
     Scheduler sends         Training code catches    Container stops        Job requeued with
     SIGTERM to container    signal & saves state     gracefully after       original priority;
     (Grace: 60-120s)        to shared NFS/GDS        flushing weights       resumes at checkpoint

1. Signal Notification: The orchestrator dispatches a SIGTERM signal to the container and begins a configurable grace period (e.g., 60 to 180 seconds).
2. State Serialization: The PyTorch/TensorFlow training script intercepts SIGTERM, flushes the current model weights, optimizer states, and epoch counters to persistent shared storage (NFS, Lustre, or GPUDirect Storage), and exits cleanly.
3. Eviction & Resource Yield: The node is freed within seconds, allowing Team Beta's guaranteed workload to begin immediately.
4. Automated Requeueing: The preempted job returns to the pending queue with elevated scheduling priority. As soon as GPU capacity becomes available anywhere in the cluster, Run:ai automatically respawns the job, which loads its latest checkpoint and resumes execution seamlessly.