8.3 Multi-Tenant Workload Isolation & Resource Slicing

8.3 Multi-Tenant Workload Isolation & Resource Slicing

Architectural Mandate: As AI models proliferate across enterprise business units, infrastructure engineers must maximize hardware utilization across expensive accelerator fleets while enforcing strict security, performance, and fault isolation boundaries. Selecting the appropriate GPU partitioning mechanism requires balancing multi-tenant density, administrative complexity, and Quality of Service (QoS) guarantees.

---

1. Comprehensive Comparison Matrix of GPU Sharing Mechanisms

Enterprise AI infrastructure supports five distinct methodologies for allocating and sharing GPU compute and memory resources:

               SPECTRUM OF GPU MULTI-TENANCY & SHARING MECHANISMS

  [Software Multiplexing] ────────────────────────────────────────► [Hardware Silicon Isolation]
  
    K8s Time-Slicing       CUDA MPS            NVIDIA vGPU        Multi-Instance GPU    Bare-Metal
     (Device Plugin)     (Client/Server)     (Hypervisor VM)           (MIG)          Dedicated GPU
  ┌───────────────────┐┌─────────────────┐┌───────────────────┐┌───────────────────┐┌───────────────┐
  │* Oversubscription ││* Shared Context ││* VM Isolation     ││* Silicon Slicing  ││* 100% Compute │
  │* Zero Memory QoS  ││* Active Threads ││* Time-Slice or MIG││* Dedicated SM/HBM ││* 100% Memory  │
  │* High Crash Risk  ││* No Fault Isolat││* Enterprise Mgmt  ││* Full Fault Isolat││* Full Scale-Up│
  └───────────────────┘└─────────────────┘└───────────────────┘└───────────────────┘└───────────────┘

Detailed Technical Comparison Matrix

---

2. Security Boundaries, Fault Isolation & Memory Protection

Understanding the failure modes and security boundaries of each sharing mechanism is critical for designing multi-tenant enterprise platforms:

+---------------------------------------------------------------------------------------------------+
|                         FAULT ISOLATION IN MULTI-TENANT SHARING                                   |
|                                                                                                   |
|  SCENARIO: Tenant A executes an invalid pointer dereference or triggers an Out-Of-Memory (OOM)    |
|                                                                                                   |
|  A. IN SOFTWARE SHARING (MPS / K8s Time-Slicing):                                                 |
|     ┌──────────────┐      ┌──────────────┐                                                        |
|     │   TENANT A   │      │   TENANT B   │                                                        |
|     │ (OOM Crash!) │      │  (Inference) │                                                        |
|     └──────┬───────┘      └──────┬───────┘                                                        |
|            │                     │                                                                |
|            ▼                     ▼                                                                |
|     ══════════════════════════════════════════                                                    |
|     SHARED GPU CONTEXT / DRIVER MEMORY ENGINE  ──► CUDA EXCEPTION THROWS (GPU RESET)              |
|     ══════════════════════════════════════════                                                    |
|                                                * Tenant B's active inference job CRASHES          |
|                                                * Entire GPU drops offline for recovery            |
|                                                                                                   |
|  B. IN HARDWARE SILICON SHARING (MIG):                                                            |
|     ┌──────────────┐      ┌──────────────┐                                                        |
|     │ MIG SLICE 0  │      │ MIG SLICE 1  │                                                        |
|     │ (Tenant A)   │      │ (Tenant B)   │                                                        |
|     │ (OOM Crash!) │      │ (Inference)  │                                                        |
|     └──────┬───────┘      └──────┬───────┘                                                        |
|            │ (Isolated Fault)    │                                                                |
|            ▼                     ▼                                                                |
|     ┌──────────────┐      ┌──────────────┐                                                        |
|     │ MMU / Log 0  │      │ MMU / Log 1  │                                                        |
|     ├──────────────┤      ├──────────────┤                                                        |
|     │ 10GB HBM (0) │      │ 20GB HBM (1) │                                                        |
|     └──────────────┘      └──────────────┘                                                        |
|     * Slice 0 context resets safely            * Slice 1 experiences ZERO interruption            |
|     * Hardware MMU traps invalid address       * 100% Memory & Compute SLA maintained             |
+---------------------------------------------------------------------------------------------------+

1. Fault Isolation in Multi-Instance GPU (MIG)

• Independent Hardware Contexts: Each MIG instance operates with dedicated memory management units, page directory pointers, interrupt lines, and hardware error logging registers.
• Error Containment: If a tenant container encounters a segmentation fault, out-of-memory condition, or illegal memory access, the GPU hardware traps the fault strictly within that instance. The host driver resets the affected MIG instance without issuing a global device reset, allowing neighboring instances to continue running with zero downtime or latency degradation.

2. Failure Propagation in CUDA Multi-Process Service (MPS)

• Shared Memory Address Space: In CUDA MPS, multiple client processes connect to a centralized MPS control daemon, sharing the same underlying CUDA driver context on the GPU.
• Uncontained Failures: While MPS allows fine-grained SM allocation (CUDA_MPS_ACTIVE_THREAD_PERCENTAGE), an unhandled fatal exception in a single client process can crash the shared MPS server daemon, corrupting the execution state for all co-located clients.

3. Memory Protection in Kubernetes Time-Slicing

• No Memory Quota Enforcement: Kubernetes time-slicing advertises multiple virtual GPUs from a single physical device. However, all containers access the same physical global memory. If Pod A experiences a memory leak, it can allocate 100% of the GPU's VRAM, causing Pod B to immediately fail with a CUDA out-of-memory error.

---

3. Operational Administration & CLI Workflows (nvidia-smi mig)

System administrators configure, inspect, and manage the lifecycle of MIG instances using the nvidia-smi command-line utility.

Step 1: Query Current MIG State & Enable MIG Mode

By default, enterprise GPUs boot with MIG mode disabled. Enabling MIG mode requires administrative root privileges and requires that no active compute processes are running on the target GPU:

# Check MIG mode status on all GPUs
nvidia-smi --query-gpu=index,name,mig.mode.current --format=csv

# Enable MIG mode on GPU 0 (requires root)
sudo nvidia-smi -i 0 -mig 1

Operational Tip: If active processes or display servers are attached to the GPU, nvidia-smi -mig 1 will fail. On modern Ampere/Hopper systems with dynamic driver reload, a reboot or GPU reset (sudo nvidia-smi --gpu-reset -i 0) applies the setting.

Step 2: List Available GPU Instance Profiles

To determine the valid profile configurations and their associated Profile IDs for the target GPU:

# List all supported GPU Instance Profiles on GPU 0
nvidia-smi mig -lgip -i 0

Example Output on NVIDIA A100-SXM4-80GB:

+--------------------------------------------------------------------------+
| GPU instance profiles:                                                   |
| GPU   ID    Name             Shared Memory (MB)  Total Memory (MB)  CE  ME|
|==========================================================================|
|   0    0    7g.80gb                       81920              81250   7   7|
|   0    5    4g.40gb                       40960              40625   4   4|
|   0    9    3g.40gb                       40960              40625   3   3|
|   0   14    2g.20gb                       20480              20312   2   2|
|   0   19    1g.10gb                       10240              10156   1   1|
+--------------------------------------------------------------------------+

Step 3: Create GPU Instances (GI) and Compute Instances (CI)

Administrators can create GPU Instances using Profile IDs or profile names. Passing the -C flag automatically provisions matching Compute Instances within each created GI:

# Create a 3g.40gb instance (Profile ID 9) and automatically create CI (-C)
sudo nvidia-smi mig -cgi 9 -C -i 0

# Create two 2g.20gb instances (Profile ID 14) and one 1g.10gb instance (ID 19)
sudo nvidia-smi mig -cgi 14,14,19 -C -i 0

Step 4: List and Inspect Provisioned Instances

To view active MIG instances and their unique device UUIDs:

# List created GPU instances on all GPUs
nvidia-smi mig -lgi

# Standard nvidia-smi display now shows partitioned MIG instances
nvidia-smi

Step 5: Target a Specific MIG Instance in Applications / Docker

To expose a specific MIG instance to an application, export its unique MIG device UUID via CUDA_VISIBLE_DEVICES:

# Find the UUID from nvidia-smi -L
# Example: MIG-GPU-9b5f1234-abcd-ef01-2345-6789abcdef01/1/0

# Expose the instance to a local application
export CUDA_VISIBLE_DEVICES=MIG-GPU-9b5f1234-abcd-ef01-2345-6789abcdef01/1/0
python3 serve_model.py --model llama-3-8b-instruct

# Run a Docker container mapped strictly to the MIG instance
docker run --gpus '"device=MIG-GPU-9b5f1234-abcd-ef01-2345-6789abcdef01/1/0"' \
  -v /models:/models nvcr.io/nvidia/tritonserver:24.06-py3 tritonserver --model-repository=/models

Step 6: Teardown and Disabling MIG Mode

To destroy instances and return the GPU to monolithic mode:

# Destroy all Compute Instances (CI) on GPU 0
sudo nvidia-smi mig -dci -i 0

# Destroy all GPU Instances (GI) on GPU 0
sudo nvidia-smi mig -dgi -i 0

# Disable MIG mode on GPU 0
sudo nvidia-smi -i 0 -mig 0

---

4. Automated Dynamic Slicing with NVIDIA MIG Manager in Kubernetes

Manually executing nvidia-smi mig commands across hundreds of Kubernetes cluster nodes is operationally impractical. The NVIDIA MIG Manager component (integrated into the NVIDIA GPU Operator) provides automated, declarative, dynamic MIG lifecycle management.

                    KUBERNETES DYNAMIC MIG ORCHESTRATION

  ┌────────────────────────────────────────────────────────────────────────┐
  │     1. Declarative Node Label Applied (e.g. mig.config=all-1g.10gb)     │
  └───────────────────────────────────┬────────────────────────────────────┘
                                      │ Node Label Detected
                                      ▼
  ┌────────────────────────────────────────────────────────────────────────┐
  │   2. NVIDIA MIG Manager (mig-parted daemon on host)                    │
  │      - Automatically cordons & drains active pods from node            │
  │      - Invokes NVML API to teardown existing MIG instances             │
  │      - Re-partitions GPU to match new ConfigMap specification          │
  └───────────────────────────────────┬────────────────────────────────────┘
                                      │ Reconfiguration Complete
                                      ▼
  ┌────────────────────────────────────────────────────────────────────────┐
  │   3. NVIDIA Kubernetes Device Plugin & GPU Feature Discovery (GFD)     │
  │      - Discovers new MIG UUIDs                                         │
  │      - Advertises Extended Resources to Kubelet:                       │
  │        nvidia.com/mig-1g.10gb: 7                                       │
  └───────────────────────────────────┬────────────────────────────────────┘
                                      │ Uncordon Node
                                      ▼
  ┌────────────────────────────────────────────────────────────────────────┐
  │   4. Kube-Scheduler Dispatches Pending AI Pods to Specific MIG Slices  │
  └────────────────────────────────────────────────────────────────────────┘

Declarative MIG Configuration via ConfigMap

The MIG Manager uses a Kubernetes ConfigMap defining standardized MIG geometry templates (e.g., all-1g.10gb, all-2g.20gb, all-3g.40gb, all-balanced):

apiVersion: v1
kind: ConfigMap
metadata:
  name: default-mig-parted-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    mig-configs:
      all-1g.10gb:
        - devices: [all]
          mig-enabled: true
          mig-devices:
            "1g.10gb": 7
      all-2g.20gb:
        - devices: [all]
          mig-enabled: true
          mig-devices:
            "2g.20gb": 3
            "1g.10gb": 1
      all-3g.40gb:
        - devices: [all]
          mig-enabled: true
          mig-devices:
            "3g.40gb": 2
            "1g.10gb": 1

Applying Dynamic Configurations to Cluster Nodes

To reconfigure an entire 8-GPU node to host 56x 1g.10gb inference microservices, an administrator simply applies a label to the Kubernetes node:

# Trigger dynamic re-partitioning across all GPUs on worker node 'gpu-node-01'
kubectl label node gpu-node-01 nvidia.com/mig.config=all-1g.10gb --overwrite

Scheduling Pods onto MIG Slices via Extended Resources

When MIG mode is active, the NVIDIA Kubernetes Device Plugin registers MIG slices as extended resources with the Kubelet. Application developers request MIG slices directly in their Pod deployment manifests:

apiVersion: v1
kind: Pod
metadata:
  name: embedding-service
  namespace: ai-inference
spec:
  containers:
  - name: bge-embedding-worker
    image: nvcr.io/nvidia/pytorch:24.06-py3
    command: ["python3", "serve_embeddings.py"]
    resources:
      limits:
        nvidia.com/mig-1g.10gb: 1   # Requests exactly one 1g.10gb hardware slice
      requests:
        nvidia.com/mig-1g.10gb: 1
        cpu: "2"
        memory: "8Gi"