4.2 Large Language Models, Tokenization & Vector Embeddings

4.2 Large Language Models, Tokenization & Vector Embeddings

Large Language Models (LLMs) represent the core technological pillar of modern natural language processing and generative AI. While human users perceive LLMs as conversational entities possessing conceptual understanding, computationally these models are sophisticated mathematical engines that evaluate statistical dependencies across numerical sequences. To bridge the gap between human language and floating-point matrix arithmetic, LLMs rely on a multi-stage pipeline: segmenting raw character strings into discrete tokens via tokenization, projecting those tokens into continuous geometric vectors via vector embeddings, and evaluating semantic relationships across a bounded context window. For the OCI AI Foundations exam, candidates must understand how tokenization operates, how embedding spaces encode semantic meaning, and how memory and computational constraints shape LLM inference.

---

The Anatomy of an LLM: Autoregressive Next-Token Prediction

At a fundamental level, an autoregressive language model is trained to solve a single statistical objective: next-token prediction. Given an arbitrary prefix sequence of tokens $(w_1, w_2, \dots, w_{t-1})$, the model calculates a conditional probability distribution over an entire fixed vocabulary $V$ to select the most probable subsequent token $w_t$:

$P(w_1, w_2, \dots, w_T) = \prod_{t=1}^T P(w_t \mid w_1, w_2, \dots, w_{t-1})$

The Generation Loop

[Input Prompt Tokens] ──> [Transformer Forward Pass] ──> [Unnormalized Logits Vector] ──> [Softmax] ──> [Sampling Strategy]
         ▲                                                                                                    │
         └────────────────────────── Append Selected Token w_t ◄──────────────────────────────────────────────┘

1. Ingestion: The model ingests the initial prompt sequence.
2. Logit Evaluation: The final linear projection layer outputs a vector of raw, unnormalized real values (called logits) containing one floating-point score for every token in the model's vocabulary (typically 32,000 to 128,000 distinct tokens).
3. Probability Normalization: The logits are passed through a Softmax function, converting them into a valid probability distribution summing to 1.0: $P(w_t = v_i \mid w_{<t}) = \frac{e^{z_i / T}}{\sum_{j=1}^{|V|} e^{z_j / T}}$ where $z_i$ represents the logit for vocabulary token $v_i$, and $T$ represents the temperature hyperparameter.
4. Sampling: A decoding algorithm (such as greedy argmax, Top-P, or Top-K) selects the next token $w_t$.
5. Autoregressive Feedback: The chosen token $w_t$ is appended to the input prompt, and the updated sequence is fed back into the model to predict token $w_{t+1}$. This sequential loop continues until the model produces a special End-of-Sequence (EOS) stop token or reaches the maximum context length.

---

Tokenization: Bridging Raw Text and Numerical IDs

Neural networks cannot process raw characters or text strings directly; they require numerical tensor inputs. Tokenization is the deterministic pre-processing step that divides raw text into discrete linguistic units called tokens, and maps each token to a unique integer index in a predefined vocabulary table.

Why Character-Level and Word-Level Tokenization Fail

Historically, natural language systems experimented with two naive tokenization approaches, both of which suffer from severe architectural limitations:

1. Word-Level Tokenization: Each unique word is assigned an ID. While intuitive, human vocabularies are vast and constantly evolving. Word-level tokenization produces massive vocabularies (millions of words), inflating embedding layer parameters. More critically, it cannot handle unseen words, misspellings, or compound terms, triggering catastrophic Out-of-Vocabulary (OOV) errors where unknown words collapse into a generic <UNK> token.
2. Character-Level Tokenization: Each individual letter, digit, and punctuation mark is an individual token. While character vocabularies are tiny (a few hundred characters) and eliminate OOV errors, character-level sequences are excessively long. Because Transformer self-attention scales quadratically with sequence length ($O(N^2)$), processing individual characters imposes crippling computational latency and dilutes semantic relationships across long distances.

Subword Tokenization: The Modern Standard

Modern LLMs utilize subword tokenization, which dynamically decomposes common words into full-word tokens and rare, complex, or inflected words into smaller meaningful subword chunks (morphemes, prefixes, suffixes, or character n-grams). Common algorithms include:

• Byte-Pair Encoding (BPE): Begins with a base vocabulary of individual characters or bytes and iteratively merges the most frequently co-occurring adjacent character pairs across the training corpus into new subwords until reaching a predefined vocabulary limit (used in GPT-4, Llama, and RoBERTa).
• WordPiece: Similar to BPE, but instead of merging based solely on frequency, it scores merges based on maximizing the likelihood of the training data according to a statistical language model (used in BERT).
• SentencePiece: A language-independent subword tokenizer that treats the input as a raw stream of unicode bytes, including whitespace characters (represented as special characters like _). SentencePiece avoids language-specific rule-based pre-segmenters, making it ideal for multilingual models (used in T5, Llama, and Gemini).

Raw Text:       "unbelievable performance"
Word-Level:     ["unbelievable", "performance"]
Character-Level: ['u', 'n', 'b', 'e', 'l', 'i', 'e', 'v', 'a', 'b', 'l', 'e', ' ', 'p', 'e', ...]
Subword (BPE):  ["un", "believ", "able", " performance"]
Token IDs:      [2849, 19483, 471, 3291]

The Token-to-Word Rule of Thumb

On the OCI AI Foundations exam, candidates must understand how token counts relate to real-world word volumes. Because subwords capture common words as single tokens while fragmenting rare or specialized words into 2 to 3 subwords, the standard industry metric for English text is:

$\mathbf{1{,}000 \text{ Tokens}} \approx \mathbf{750 \text{ English Words}} \quad (\approx 0.75 \text{ words per token})$ $\mathbf{1 \text{ English Word}} \approx \mathbf{1.33 \text{ Tokens}}$

This rule of thumb is critical for enterprise cost estimation, API pricing calculations (where cloud providers charge per 1,000 or 1,000,000 tokens), and context window allocation in services like the OCI Generative AI Service.

---

Vector Embeddings and Semantic Geometry

Once raw text is converted into a sequence of integer token IDs, each token ID is mapped to a continuous mathematical representation via an Embedding Lookup Layer.

From Discrete IDs to Dense Vectors

In early natural language processing, words were represented as One-Hot Vectors—sparse vectors of length $|V|$ containing a single $1.0$ at the word's index and $0.0$ everywhere else. One-hot vectors suffer from severe computational limitations: they are excessively high-dimensional, highly sparse, and mathematically orthogonal. The dot product between any two distinct one-hot vectors is zero, meaning that "physician" and "doctor" are treated as mathematically equidistant to "submarine."

Vector Embeddings resolve this by projecting discrete token IDs into a dense, continuous, high-dimensional vector space $\mathbb{R}^d$ (where dimension $d$ typically ranges from 768 to 4,096 dimensions):

$\text{Token ID } (4821) \xrightarrow{\text{Embedding Matrix } W_e} \mathbf{e} \in \mathbb{R}^d = [0.042, -0.819, 0.231, \dots, -0.105]$

Sparse One-Hot (Dimension = 50,000):   [0, 0, 0, 0, ..., 1, ..., 0, 0]  (No semantic geometry)
Dense Embedding (Dimension = 1,024):  [0.21, -0.54, 0.88, 0.12, ...]  (Rich semantic geometry)

Semantic Geometry and Vector Arithmetic

In a well-trained embedding space, geometric distance and directional orientation reflect semantic meaning. Words that appear in similar linguistic contexts or share conceptual attributes are positioned close to one another in vector space. Furthermore, linear spatial directions encode abstract relational concepts, enabling semantic vector arithmetic:

$\vec{v}_{\text{King}} - \vec{v}_{\text{Man}} + \vec{v}_{\text{Woman}} \approx \vec{v}_{\text{Queen}}$ $\vec{v}_{\text{Paris}} - \vec{v}_{\text{France}} + \vec{v}_{\text{Japan}} \approx \vec{v}_{\text{Tokyo}}$

Beyond single tokens, specialized Embedding Models (such as Cohere Embed available in OCI Generative AI) ingest entire sentences, paragraphs, or documents and project them into a single fixed-size dense vector. These document embeddings capture the holistic semantic essence of the text, enabling semantic search, recommendation systems, clustering, and Retrieval-Augmented Generation (RAG).

---

Measuring Semantic Proximity: Cosine Similarity

To compare how conceptually similar two embedding vectors are, systems evaluate their geometric alignment in high-dimensional space.

1. Cosine Similarity

Cosine similarity evaluates the cosine of the angle $\theta$ between two vectors $\mathbf{u}$ and $\mathbf{v}$. It measures directional orientation rather than absolute vector magnitude:

$\text{Cosine Similarity}(\mathbf{u}, \mathbf{v}) = \cos(\theta) = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \|\mathbf{v}\|} = \frac{\sum_{i=1}^d u_i v_i}{\sqrt{\sum_{i=1}^d u_i^2} \sqrt{\sum_{i=1}^d v_i^2}}$

• Output Range: $[-1.0, 1.0]$. A score of $+1.0$ indicates identical directional orientation (maximum semantic similarity); $0.0$ indicates orthogonality (no semantic relationship); $-1.0$ indicates diametrically opposite orientation.
• Why Cosine Similarity is Favored: When processing documents of varying lengths, the magnitude (length) of an embedding vector can fluctuate. Cosine similarity normalizes for magnitude, ensuring that a short sentence and a detailed paragraph discussing the identical topic achieve a near-perfect similarity score.

2. Euclidean Distance ($L_2$ Distance)

Euclidean distance calculates the straight-line physical distance between vector endpoints:

$d(\mathbf{u}, \mathbf{v}) = \sqrt{\sum_{i=1}^d (u_i - v_i)^2}$

Unlike cosine similarity, Euclidean distance is sensitive to vector magnitude. If vectors are unit-normalized ($|\mathbf{u}| = 1$), Euclidean distance and cosine similarity are monotonically related.

3. Dot Product (Inner Product)

The raw dot product $\mathbf{u} \cdot \mathbf{v} = \sum_{i=1}^d u_i v_i$ combines angular alignment with vector magnitudes. When embeddings are normalized during inference, the dot product equals the cosine similarity while requiring fewer computational operations on GPU hardware.

---

Context Windows and Computational Complexity

The context window defines the maximum sequence length (measured in total tokens) that an LLM can process in a single inference pass. This budget encompasses both the input prompt tokens and the generated output completion tokens.

$\text{Total Context Budget} = \text{Prompt Tokens} + \text{Max Generation Tokens}$

If an enterprise user submits a 10,000-token prompt to an LLM with a 16,384-token context window, the model can generate at most $16,384 - 10,000 = 6,384$ completion tokens before truncating or failing.

The Quadratic Attention Bottleneck: $O(N^2)$

Why cannot models support infinite context windows? In a standard Transformer architecture, the self-attention mechanism compares every token in the sequence against every other token. For an input sequence of length $N$, computing pairwise attention scores requires an $N \times N$ attention matrix:

$\text{Computational Complexity} = O(N^2) \quad | \quad \text{Memory Footprint} = O(N^2)$

Doubling the context window from 4,000 tokens to 8,000 tokens increases the computational operations and memory consumption of the attention layer by a factor of $4$ ($2^2 = 4$). Expanding to 128,000 tokens demands specialized hardware optimizations (such as FlashAttention, sparse attention, and grouped-query attention).

Key-Value (KV) Caching in Autoregressive Inference

During naive autoregressive generation, generating each new token would require re-running the entire Transformer forward pass across all previous tokens, resulting in redundant matrix calculations. To eliminate this bottleneck, modern inference engines utilize Key-Value (KV) Caching:

• As previous tokens pass through the self-attention layers, their calculated Key ($K$) and Value ($V$) projection tensors are stored in high-speed GPU Video RAM (VRAM).
• When predicting the next token $w_t$, the model only computes the Query ($Q$) for the single incoming token, retrieving past $K$ and $V$ vectors directly from the cache.
• This reduces the computational complexity of generating each new token from $O(N^2)$ down to $O(N)$, dramatically accelerating generation speed at the expense of high GPU memory consumption.