7.1 AI Workload Scheduling: Slurm & Kubernetes

7.1 AI Workload Scheduling: Slurm & Kubernetes

Executive Summary: Deep learning training workloads differ fundamentally from traditional enterprise web microservices. A distributed large language model (LLM) training job is not a collection of independent, loosely coupled processes; it is a tightly synchronized, all-or-nothing distributed application where hundreds or thousands of GPUs must execute simultaneously across high-speed NVLink and InfiniBand fabrics. Managing these compute-intensive workloads requires sophisticated workload managers. Organizations generally deploy one of two primary architectural paradigms: traditional HPC batch schedulers (Slurm) or cloud-native container orchestrators (Kubernetes) augmented with batch scheduling extensions such as Volcano and Kueue.

---

1. The Scheduling Paradigm Shift: HPC vs. Cloud-Native

To understand modern AI infrastructure operations, infrastructure engineers must contrast the design principles of High-Performance Computing (HPC) batch schedulers with cloud-native container orchestration platforms.

                      AI WORKLOAD SCHEDULING PARADIGMS

  ┌────────────────────────────────────────┐  ┌────────────────────────────────────────┐
  │     HIGH-PERFORMANCE COMPUTING (HPC)   │  │        CLOUD-NATIVE AI PLATFORM        │
  │             (e.g., Slurm)              │  │          (e.g., Kubernetes)            │
  ├────────────────────────────────────────┤  ├────────────────────────────────────────┤
  │ • Static Batch Queueing                │  │ • Dynamic Microservices & Jobs         │
  │ • Rigid All-or-Nothing Gang Allocation │  │ • Pod-by-Pod Sequential Scheduling    │
  │ • Direct Hardware / Bare-Metal Access  │  │ • Containerized Abstraction Layers     │
  │ • Native Topology & NUMA Awareness     │  │ • Requires Add-on Batch Plugins (Kueue)│
  │ • Long-Running Multi-Day Training Runs │  │ • Elastic Inference & Rapid Prototyping│
  │ • POSIX Shared Filesystem Integration  │  │ • Declarative YAML / GitOps Control    │
  └────────────────────────────────────────┘  └────────────────────────────────────────┘

Architectural Comparison Matrix

---

2. Slurm Workload Manager Architecture & Mechanics

Slurm (Simple Linux Utility for Resource Management) is the dominant open-source workload manager across supercomputing centers and dedicated AI supercomputers (including many large-scale NVIDIA DGX SuperPOD deployments).

                         SLURM DAEMON ARCHITECTURE

  ┌────────────────────────────────────────────────────────────────────────┐
  │                        slurmctld (Central Controller)                  │
  │    - Job Queues & Reservations       - Fair-Share Priority Engine      │
  │    - Node State & Health Monitoring  - Backfill Scheduling Algorithm   │
  └────────────────────▲───────────────────────────────▲───────────────────┘
                       │                               │
        RPC Management │                               │ Accounting Records
                       │                               │
  ┌────────────────────▼──────────────┐  ┌─────────────▼───────────────────┐
  │        slurmd (Compute Nodes)     │  │      slurmdbd (Database Daemon) │
  │  - Spawns tasks via slurmd/stepd  │  │  - Historical resource tracking │
  │  - Monitors local CPU/GPU/GRES    │  │  - Multi-cluster user accounts  │
  │  - Enforces cgroups resource limits│  │  - Fair-Share usage decay data  │
  └───────────────────────────────────┘  └─────────────────────────────────┘

Core Slurm Daemons & Components

1. slurmctld (Central Management Daemon): The brain of the Slurm cluster. It maintains cluster state, monitors compute node availability, processes job submissions, computes job priorities using multi-factor algorithms, and executes the scheduling engine.
2. slurmd (Compute Node Daemon): Runs on every compute host. It monitors local physical resources, receives job allocation instructions from slurmctld, spawns per-job container/step daemons (slurmstepd), attaches hardware devices via Linux cgroups, and reports node status back to the controller.
3. slurmdbd (Database Daemon): Secure interface to a central relational database (typically MySQL/MariaDB). It stores user accounts, project hierarchies, Quality of Service (QoS) definitions, and historical job execution logs used for billing and Fair-Share calculations.
4. Client CLI Utilities:
   • sbatch: Submits asynchronous batch job scripts to the scheduling queue.
   • srun: Launches interactive jobs or parallel task steps in real time.
   • salloc: Allocates compute resources dynamically for interactive exploration.
   • squeue: Queries the state of active, pending, and running jobs in the queues.
   • sinfo: Reports compute node and partition availability.
   • scontrol: Administrative tool to view and modify cluster configuration, nodes, and running jobs.

Generic Resource Scheduling (GRES) for GPUs

Slurm manages accelerators through the Generic Resource (GRES) framework. In slurm.conf and gres.conf, system administrators explicitly define the physical topology of GPUs, mapping them to specific PCIe buses, NUMA memory domains, and NVLink fabrics.

# Example /etc/slurm/gres.conf on a DGX H100 node:
NodeName=dgx-h100-[01-32] Name=gpu Type=h100 File=/dev/nvidia[0-7] Cores=0-111

When a researcher submits a job requesting GPUs, Slurm leverages GRES to isolate the specific GPU device files inside the container or process namespace via Linux cgroups:

#!/bin/bash
#SBATCH --job-name=llama3-70b-finetune
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --gres=gpu:h100:8
#SBATCH --cpus-per-task=14
#SBATCH --partition=deeplearning
#SBATCH --qos=high_priority
#SBATCH --time=48:00:00
#SBATCH --output=logs/%x_%j.out

# Launch 32 PyTorch distributed processes across 4 DGX H100 nodes
srun --export=ALL torchrun \
    --nproc_per_node=8 \
    --nnodes=4 \
    --rdzv_id=$SLURM_JOB_ID \
    --rdzv_backend=c10d \
    --rdzv_endpoint=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n 1):29500 \
    train.py --config config_70b.yaml

Advanced Slurm Scheduling Mechanics

• Multi-Factor Priority Algorithm: Slurm calculates the priority of queued jobs using a weighted formula combining multiple independent factors: $\text{Job Priority} = W_A \cdot \text{Age} + W_{FS} \cdot \text{Fair-Share} + W_{JS} \cdot \text{Job Size} + W_P \cdot \text{Partition} + W_Q \cdot \text{QoS}$
• Fair-Share Multi-Tenant Algorithm: Tracks historical GPU consumption per user, research group, and department. As an organization consumes compute hours, its Fair-Share score decreases, elevating the priority of underserved teams. A half-life decay parameter gradually discounts past usage over time.
• Backfill Scheduling: In traditional First-In, First-Out (FIFO) scheduling, a massive 64-node job waiting for resources would block all incoming work, leaving idle nodes underutilized. Slurm's backfill scheduler plans a future reservation for the large job, then searches the pending queue to identify smaller, shorter jobs that can execute on idle nodes and finish before the large job's reservation window begins, maximizing cluster utilization without delaying high-priority work.
• Partitions & Quality of Service (QoS): Partitions act as virtual queues with specific node subsets and access rules. QoS overlays define priority boosts, maximum wall-clock limits, GPU concurrency caps, and preemption rules (e.g., allowing a production training job to preempt an exploratory interactive run).

---

3. Kubernetes Native Scheduling Limitations for Deep Learning

The native Kubernetes scheduler (kube-scheduler) was architected around the requirements of web microservices: stateless, independently scalable containers with asynchronous lifecycles. When applied directly to distributed AI training, kube-scheduler introduces fundamental operational limitations.

      THE DISTRIBUTED DEADLOCK SCENARIO (LACK OF GANG SCHEDULING)

  Cluster Capacity: 2 Nodes (16 Total GPUs)
  Job 1 requires 16 GPUs (2 Nodes)  |  Job 2 requires 16 GPUs (2 Nodes)

  ┌───────────────────────────────────┐  ┌───────────────────────────────────┐
  │          COMPUTE NODE 01          │  │          COMPUTE NODE 02          │
  │         (8x H100 SXM5 GPUs)       │  │         (8x H100 SXM5 GPUs)       │
  ├───────────────────────────────────┤  ├───────────────────────────────────┤
  │  [Allocated to Job 1 - Pod 0]     │  │  [Allocated to Job 2 - Pod 0]     │
  │  Status: WAITING FOR PEER PODS    │  │  Status: WAITING FOR PEER PODS    │
  │  (Consumes 8 GPUs, Idles)         │  │  (Consumes 8 GPUs, Idles)         │
  └───────────────────────────────────┘  └───────────────────────────────────┘
    ▲                                      ▲
    └────────────── MUTUAL RESOURCE DEADLOCK ──────────────┘
    - Job 1 holds Node 1 and blocks waiting for Node 2.
    - Job 2 holds Node 2 and blocks waiting for Node 1.
    - Result: 16 enterprise GPUs sit 100% idle indefinitely.

Core Failure Modes of Native kube-scheduler in AI Clusters

1. Pod-by-Pod Scheduling & Distributed Deadlocks:
   • kube-scheduler evaluates pods individually and sequentially without holistic awareness of multi-pod distributed jobs.
   • In an 8-node PyTorchJob, if only 4 nodes are available, native Kubernetes schedules 4 pods and leaves the remaining 4 pending. Because distributed PyTorch utilizes synchronous collective initialization (such as torch.distributed.init_process_group with NCCL), the running pods stall indefinitely waiting for the missing ranks to join the communication barrier. If another distributed job acquires the remaining nodes, both jobs enter a mutual resource deadlock, consuming expensive GPU hours while delivering zero compute progress.
2. Lack of Hardware & Interconnect Topology Awareness:
   • In modern multi-GPU nodes, GPUs are arranged across specific PCIe root complexes, CPU sockets, and NVLink switch crossbars. Connecting two GPUs across distinct NUMA domains or misaligned PCIe switches degrades inter-GPU memory bandwidth significantly.
   • Native Kubernetes treats nvidia.com/gpu: 2 as fungible scalar integers. It may assign GPU 0 and GPU 7 to the same container—forcing communication across separate CPU sockets and host PCIe buses rather than adjacent NVLink pairs.
3. GPU Bin-Packing vs. Spreading Deficiencies:
   • Native scoring algorithms (NodeResourcesFit) can inadvertently scatter interactive single-GPU pods across every node in a cluster, fragmenting 8-GPU nodes so that no single node retains 8 contiguous GPUs for large-scale training jobs.

---

4. Cloud-Native Batch Schedulers: Volcano & Kueue

To overcome native Kubernetes limitations, the cloud-native ecosystem developed purpose-built batch scheduling frameworks that bring supercomputing-grade orchestration into standard Kubernetes clusters.

                         CLOUD-NATIVE BATCH SCHEDULERS

  ┌────────────────────────────────────────┐  ┌────────────────────────────────────────┐
  │             VOLCANO (CNCF)             │  │         KUEUE (K8s SIG-SCHEDULING)     │
  ├────────────────────────────────────────┤  ├────────────────────────────────────────┤
  │ • Full custom secondary scheduler      │  │ • Non-intrusive Job Queue Controller   │
  │ • PodGroup Custom Resource Definition  │  │ • ClusterQueue & LocalQueue CRDs       │
  │ • Native Gang Scheduling engine        │  │ • Workload Admission & Cohort Borrowing│
  │ • Custom Plugins (NUMA, SSH, MPI)      │  │ • Integrates with PyTorchJob, Ray, Jobs│
  │ • Replaces kube-scheduler for batch    │  │ • Works with native & custom schedulers│
  └────────────────────────────────────────┘  └────────────────────────────────────────┘

Volcano Architecture & Capabilities

Volcano is a Cloud Native Computing Foundation (CNCF) batch scheduling engine engineered specifically for high-performance AI, deep learning, and big data workloads:

• PodGroup CRD & Gang Scheduling: Volcano introduces the PodGroup Custom Resource Definition with a minMember parameter. Volcano will not bind any pod in a job to a node until sufficient cluster resources exist to bind all minMember pods simultaneously. If resources are insufficient, the entire job remains pending in the queue, completely eliminating distributed deadlocks.
• Queue Management & Dynamic Weights: Supports multi-tenant hierarchical queues with assigned weights, minimum resource guarantees, and maximum capacity caps.
• Task-Topology & NUMA Awareness: Evaluates network distance and NUMA topology when placing multi-task jobs, placing cooperating ranks on the closest possible physical nodes.

Kueue Architecture & Capabilities

Kueue is a Kubernetes-native, lightweight batch queuing controller developed by Kubernetes Special Interest Group (SIG) Scheduling. Unlike Volcano, which replaces the core scheduler, Kueue operates as an admission and job lifecycle controller that governs when jobs are admitted to the cluster:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: enterprise-ai-queue
spec:
  namespaceSelector: {}
  cohort: research-division
  resourceGroups:
    - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
      flavors:
        - name: h100-sxm5-flavor
          resources:
            - name: "nvidia.com/gpu"
              nominalQuota: 64
              borrowingLimit: 32

• Core Constructs:
  • ResourceFlavor: Associates physical node labels and taints with specific hardware types (e.g., H100 SXM5 vs. A100 PCIe).
  • ClusterQueue: Cluster-scoped resource pool defining nominal quotas, borrowing limits, and preemption policies.
  • LocalQueue: Namespaced queue submitted to by data scientists, mapping directly to an upstream ClusterQueue.
  • Workload: Represents the resource request of a batch application (e.g., PyTorchJob, RayCluster, JobSet, or standard Batch Job).
• Cohort Sharing & Resource Borrowing: Multiple ClusterQueues can join a shared Cohort. When Department A is not using its nominal quota of 64 GPUs, Department B can dynamically borrow up to its borrowingLimit. When Department A submits new work, Kueue automatically preempts Department B's borrowed workloads to reclaim the guaranteed capacity.