3.5 Software Components Across the AI Development & Deployment Lifecycle
Key Takeaways
- Blueprint objective 1.7 asks you to describe the software components tied to the life cycle of AI development and deployment, so the exam expects you to place a named tool in the correct lifecycle stage.
- The canonical stages are data preparation, model development and training, experiment orchestration, optimisation, deployment and serving, and monitoring with feedback.
- RAPIDS, DALI, and NeMo Curator accelerate the data stage; NGC framework containers, NeMo, and TAO Toolkit cover development and transfer learning.
- TensorRT and TensorRT-LLM are optimisation-stage tools that compile a trained model into an inference engine — they never train, and confusing the two stages is a common exam trap.
- Triton and NIM serve models, while DCGM plus Prometheus and Grafana close the loop with operational telemetry that feeds retraining decisions.
3.5 Software Components Across the AI Development & Deployment Lifecycle
Blueprint anchor: Domain 1 objective 1.7 — "Describe the software components related to the life cycle of AI development and deployment." The exam does not ask you to operate these tools; it asks you to place a named component in the right stage and to recognise which stage a described activity belongs to.
1. The Six-Stage Lifecycle
An enterprise AI system is never a single artefact. It is a loop that runs continuously, and each turn of the loop touches a different part of the software stack.
| Stage | What Happens | NVIDIA / Ecosystem Components | Artefact Produced |
|---|---|---|---|
| 1. Data preparation | Ingest, clean, deduplicate, label, transform, augment | RAPIDS (cuDF, cuML), Spark RAPIDS accelerator, DALI (data loading/augmentation), NeMo Curator (LLM corpus curation) | Curated, versioned dataset |
| 2. Model development & training | Architecture selection, pretraining, fine-tuning, alignment | NGC framework containers (PyTorch, TensorFlow, JAX), NeMo, Megatron-Core, TAO Toolkit (transfer learning on pretrained vision/speech models) | Trained checkpoint |
| 3. Experiment orchestration | Scheduling, quotas, distributed job launch, experiment tracking, checkpoint management | Base Command Manager, Run:ai, Slurm, Kubernetes with the GPU Operator, MLflow/Weights & Biases | Reproducible run history |
| 4. Optimisation | Quantise, prune, distil, fuse kernels, compile to an inference engine | TensorRT, TensorRT-LLM, TensorRT Model Optimizer, ONNX export | Optimised inference engine/plan |
| 5. Deployment & serving | Host the model, batch requests, scale, expose an API | Triton Inference Server, NIM, NVIDIA AI Enterprise, Fleet Command (edge) | Running inference endpoint |
| 6. Monitoring & feedback | Track hardware health, latency, throughput, drift; decide when to retrain | DCGM + DCGM Exporter, Prometheus, Grafana, Triton metrics endpoint, NeMo Guardrails for output policy | Telemetry, alerts, retraining trigger |
The loop closes because stage 6 produces the evidence that sends you back to stage 1: drifting accuracy, new failure classes, or fresh production data worth labelling.
2. Stage Boundaries the Exam Actually Tests
Training frameworks versus inference compilers
This is the single most commonly confused boundary. PyTorch, TensorFlow, and NeMo train. TensorRT and TensorRT-LLM do not train — they consume an already-trained model and compile it into a hardware-specific engine through layer fusion, kernel auto-tuning, precision calibration, and memory planning. If a stem describes "reducing latency of an existing model without changing its accuracy target," the answer lives in stage 4, not stage 2.
Optimisation versus serving
TensorRT produces an engine; Triton runs it. Triton is framework-agnostic — it has backends for TensorRT, ONNX Runtime, PyTorch, Python, and more — so "we need to serve models from three different frameworks behind one endpoint" is a Triton answer, while "we need this one model to run as fast as possible on H100" is a TensorRT answer.
Transfer learning versus pretraining
TAO Toolkit exists so that teams can adapt NVIDIA pretrained vision and speech models to their own classes with modest data and no deep-learning research effort. Pretraining a foundation model from scratch is a NeMo/Megatron-Core activity at cluster scale. A stem mentioning "a few thousand labelled images and no ML research team" is pointing at transfer learning.
Registry versus runtime
The NGC catalogue is where versioned containers and pretrained models come from; it is not a runtime. Pinning an NGC container tag across a cluster is how you keep CUDA, driver, and framework versions consistent — a configuration-management answer, not a performance answer.
3. Why Containers Anchor the Whole Lifecycle
Every stage above ships as a container in practice, and that is deliberate:
- Version coherence. A deep-learning stack couples the GPU driver, CUDA runtime, cuDNN, NCCL, the framework, and the model code. NGC images are built and tested as a matched set, so a job that ran in March runs identically in September.
- Portability across stages and sites. The same image can run on a developer workstation, a Slurm batch node, a Kubernetes pod, and a cloud instance.
- Reproducibility. An experiment is identified by image tag plus dataset version plus hyperparameters, which is what makes stage 3's run history meaningful.
- Scheduling granularity. Because the unit of work is a container, orchestration (stage 3) and serving (stage 5) can use the same primitives — the NVIDIA Container Toolkit exposes GPUs to the container runtime, and the GPU Operator automates that plumbing across a Kubernetes cluster.
Exam cue: if a scenario mentions inconsistent results or dependency conflicts between nodes, the expected answer names standardised NGC containers, not a hardware change.
A team has a fully trained model and needs to cut inference latency on H100 GPUs without retraining it. Which lifecycle stage and NVIDIA component apply?
An enterprise has a few thousand labelled images of its own product defects and no deep-learning research staff, but wants a production-grade vision model. Which approach and NVIDIA component fit best?
Jobs that succeed on one cluster node fail on another because of mismatched CUDA, cuDNN, and framework versions. Which lifecycle practice does the NCA-AIIO material point to as the fix?