1.2 AI, Machine Learning & Deep Learning Workloads

1.2 AI, Machine Learning & Deep Learning Workloads

Core Concept: Understanding the architectural taxonomy of modern artificial intelligence and the mathematical mechanics of neural networks is fundamental for designing and operating GPU-accelerated computing infrastructure. Different model families—such as convolutional vision networks, autoregressive large language models, and deep recommendation systems—exhibit vastly different compute, memory bandwidth, and interconnect scaling requirements.

---

1. The AI Taxonomy: AI vs. ML vs. DL vs. GenAI

Modern artificial intelligence is structured as a series of nested subfields, where each subsequent tier introduces greater computational complexity, specialized data structures, and distinct hardware acceleration requirements.

┌────────────────────────────────────────────────────────────────────────┐
│                     ARTIFICIAL INTELLIGENCE (AI)                       │
│  Rule-based systems, expert systems, symbolic logic, search heuristics  │
│  ┌──────────────────────────────────────────────────────────────────┐  │
│  │                     MACHINE LEARNING (ML)                        │  │
│  │  Statistical algorithms, feature engineering, regression, SVMs    │  │
│  │  ┌────────────────────────────────────────────────────────────┐  │  │
│  │  │                     DEEP LEARNING (DL)                     │  │  │
│  │  │  Multi-layer neural networks, backpropagation, GEMM cores  │  │  │
│  │  │  ┌──────────────────────────────────────────────────────┐  │  │  │
│  │  │  │                  GENERATIVE AI (GenAI)               │  │  │  │
│  │  │  │  Transformers, LLMs, Diffusion Models, Autoregressive │  │  │  │
│  │  │  └──────────────────────────────────────────────────────┘  │  │  │
│  │  └────────────────────────────────────────────────────────────┘  │  │
│  └──────────────────────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────────────────────┘

Detailed Taxonomy Comparison

---

2. Neural Network Computational Mechanics

Deep neural networks process information through multidimensional tensor transformations across layers of artificial neurons. Understanding the cyclical execution flow is crucial for diagnosing GPU utilization and distributed cluster bottlenecks.

   ┌───────────────────────────────────────────────────────────────────┐
   │                       1. FORWARD PASS                             │
   │   Input X ───► [ Z = W · X + b ] ───► [ A = σ(Z) ] ───► Output ŷ   │
   └─────────────────────────────────┬─────────────────────────────────┘
                                     │
                                     ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │                     2. LOSS CALCULATION                           │
   │   Compute Error via Loss Function: L = Loss(ŷ, y_true)            │
   └─────────────────────────────────┬─────────────────────────────────┘
                                     │
                                     ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │                     3. BACKWARD PASS (BACKPROP)                   │
   │   Compute Gradients via Chain Rule: ∂L/∂W = (∂L/∂ŷ)·(∂ŷ/∂Z)·(∂Z/∂W)│
   └─────────────────────────────────┬─────────────────────────────────┘
                                     │
                                     ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │                     4. OPTIMIZER WEIGHT UPDATE                    │
   │   Update Parameters: W_new = W_old - η · Optimizer(∂L/∂W)         │
   │   (SGD, Momentum, Adam, AdamW Tracking First & Second Moments)    │
   └───────────────────────────────────────────────────────────────────┘

Step 1: Forward Pass (Inference & Activation Computation)

In the forward pass, input data $X$ (represented as a batch of multidimensional tensors) propagates through consecutive network layers. Each linear layer performs a General Matrix Multiply (GEMM) followed by a bias addition: $Z = W \cdot X + b$ Where $W$ represents the learnable weight matrix, $X$ is the input tensor, and $b$ is the bias vector. The linear output $Z$ is then passed through an activation function $\sigma(Z)$ to produce activation tensor $A$.

Step 2: Loss Function & Objective Evaluation

The model's predicted output $\hat{y}$ is compared against the ground-truth target $y$ using a domain-specific mathematical loss function:

• Cross-Entropy Loss: Used for multi-class classification and autoregressive token generation in LLMs: $L_{CE} = -\sum_{i} y_i \log(\hat{y}_i)$
• Mean Squared Error (MSE): Used for continuous regression tasks: $L_{MSE} = \frac{1}{N} \sum_{i=1}^N (y_i - \hat{y}_i)^2$

Step 3: Backward Pass & Gradient Computation (Backpropagation)

Backpropagation utilizes the calculus chain rule to compute the partial derivative of the scalar loss $L$ with respect to every learnable parameter (weights $W$ and biases $b$) across all layers in reverse order: $\frac{\partial L}{\partial W} = \frac{\partial L}{\partial A} \cdot \frac{\partial A}{\partial Z} \cdot \frac{\partial Z}{\partial W}$ These partial derivatives, known as gradients, indicate the direction and magnitude of parameter adjustment required to minimize total loss. The backward pass requires approximately 2× the floating-point operations (FLOPs) of the forward pass.

Step 4: Optimizer Parameter Updates

The calculated gradients are ingested by an optimization algorithm to update the model parameters. Different optimizers exhibit varying memory overheads:

• Stochastic Gradient Descent (SGD): Directly updates weights along the negative gradient vector: $W \leftarrow W - \eta \nabla_W L$, where $\eta$ is the learning rate. Low memory overhead (0 extra bytes per parameter).
• SGD with Momentum: Tracks an exponential moving average of past gradients to accelerate through flat loss surfaces. Requires 4 additional bytes per parameter (FP32 momentum vector).
• Adam (Adaptive Moment Estimation): Maintains running estimates of both the uncentered first moment (mean gradient $m_t$) and second moment (uncentered variance $v_t$) for every parameter: $m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t, \quad v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2$ Requires 8 additional bytes per parameter (two FP32 state vectors).
• AdamW (Decoupled Weight Decay): The industry standard for training Transformers and LLMs. Decouples $L_2$ weight decay regularization from gradient updates, preventing weight decay from being distorted by the adaptive learning rate.

---

3. Evolution of Activation Functions

Activation functions introduce non-linear mapping capabilities, enabling deep networks to approximate complex mathematical functions beyond simple linear hyperplanes.

---

4. Workload Computational Profiles & System Demands

Different enterprise AI workloads place distinct stresses on GPU compute, memory subsystem capacity, and distributed networking.

+-----------------------------------------------------------------------------+
|                        WORKLOAD COMPUTATIONAL PROFILES                      |
|                                                                             |
|  [ COMPUTER VISION ]            [ LARGE LANGUAGE MODELS ]   [ RECOMMENDER ] │
|  - 2D/3D Convolutions           - Multi-Head Self-Attention - Massive Sparse│
|  - Dense GEMM Operations        - Matrix Multiplication     - Embedding Tbls│
|  - High Spatial Locality        - KV Cache Dynamics         - Low Compute / │
|  - High Arithmetic Intensity    - All-Reduce / TP Comm      - High DRAM Band│
|  ==> COMPUTE-BOUND              ==> COMPUTE & MEMORY BOUND  ==> MEMORY-BOUND│
+-----------------------------------------------------------------------------+

1. Computer Vision (CV): Convolutional Networks & Vision Transformers

• Mechanics: Classical CV relies on 2D/3D convolutional kernels sliding across pixel grids, implemented via im2col tensor flattening mapped to Tensor Core GEMM operations. Vision Transformers (ViTs) divide images into $16 \times 16$ pixel patches treated as sequential tokens.
• Computational Profile: High arithmetic intensity (FLOPs per byte). High spatial data reuse in GPU SRAM/L2 cache. Generally compute-bound during both training and batch inference.

2. Natural Language Processing & Large Language Models (NLP / LLMs)

• Mechanics: Built upon the Transformer architecture utilizing Multi-Head Self-Attention (MHSA): $\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$ Where Query ($Q$), Key ($K$), and Value ($V$) projections generate attention score matrices with $O(N^2)$ computational and memory complexity relative to sequence length $N$.
• Computational Profile: Highly bifurcated profile. Training and prompt prefill are compute-bound dense GEMMs requiring massive Tensor Core throughput and inter-GPU communication (Tensor Parallel All-Reduce). Autoregressive token decoding is memory-bandwidth bound, constrained by streaming model weights and the Key-Value (KV) cache from HBM for every single generated token.

3. Deep Learning Recommendation Models (DLRM)

• Mechanics: DLRMs (e.g., Meta DLRM, NVIDIA Merlin) process two distinct data streams: dense numerical features (processed via compute-heavy Bottom MLPs) and sparse categorical features (user IDs, search terms, item IDs).
• Computational Profile: Sparse features are indexed into massive Embedding Tables spanning hundreds of gigabytes to terabytes of memory. Table lookups perform memory gathers (sparse memory lookups) with very low arithmetic intensity. Scaling DLRMs requires massive GPU HBM capacity and high All-to-All network interconnects to distribute embedding tables across cluster nodes.