2.2 Large Language Models & How They Work

2.2 Large Language Models and How They Work

Core principle: An LLM estimates distributions over tokens from its parameters and current context. The mechanism can produce remarkably useful language and code, but probability, fluency, and a low decoding temperature are not evidence that a claim is true.

---

From Text to Tokens

A tokenizer converts text into tokens and token IDs from a model-specific vocabulary. Tokens may be whole words, word pieces, punctuation, whitespace, bytes, or other units. The split and IDs differ across models and languages. “One token is four characters” is at most a rough English budgeting heuristic, not a rule. Code, identifiers, numbers, and languages can tokenize very differently.

Each token ID maps to a learned vector. Positional information helps the model distinguish order. Transformer layers repeatedly combine these representations using attention and feed-forward computations. The internal token representations used for next-token prediction should not be confused with a separate embedding model used to represent whole passages for search. Both involve vectors, but they have different training objectives and uses.

Transformer Attention

Self-attention lets a position weight information from other permitted positions. Multiple heads can learn different patterns, but an attention weight is not a human explanation or a guarantee that the model used the right evidence. Causal language models mask future positions during training and generation so a token cannot depend on later output tokens.

Transformers improved parallelization during training because computations for input positions can be performed in parallel within a layer. Generation by a standard autoregressive decoder is different: it selects one token, appends it, and then computes the next. Systems can optimize or batch this work, but the logical dependency remains sequential.

Autoregressive Generation

For each output step, the model produces scores, often called logits, across its vocabulary. Decoding converts those scores into a token choice. The chosen token becomes part of the next context, so an early mistake can influence later text. A simplified path is:

1. tokenize the current context;
2. compute contextual representations;
3. produce next-token scores;
4. apply the product's decoding settings;
5. choose a token and repeat until a stop condition.

This objective rewards plausible continuation, not fact-checking. Training patterns may contain errors; context may be missing, stale, or adversarial; and the model can combine fragments into a claim that no source supports.

Decoding Controls

Temperature typically rescales logits before sampling. Lower values concentrate probability on higher-scoring tokens; higher values generally increase diversity. Top-p limits sampling to a set whose cumulative probability reaches a threshold. Top-k limits the candidate count. Exact semantics and allowed ranges are product-specific.

A low temperature often improves consistency, but it does not make the underlying highest-probability answer correct. Temperature zero may use greedy decoding or another provider-defined behavior, and repeated requests may still vary because of nondeterministic infrastructure, model updates, seeds, tools, routing, or hidden context. For an exact task, use schemas, compilers, tests, deterministic calculators, and validators.

Context Windows

A context window limits the material a model can use for a request. Products differ on whether they publish one combined input-output limit, separate limits, reserved capacity, or automatic history management. Do not calculate a universal remainder without the actual model contract.

More capacity is not perfect memory. Long-context performance depends on task, model, relevant-information position, distractors, and retrieval. Some research finds lower retrieval performance for facts in the middle of long inputs, but the curve and magnitude are not universal. Curate relevant passages, keep source identifiers, ask for citations, and test the actual model at realistic lengths.

Embeddings and Retrieval

A retrieval system often uses an embedding model to map passages and a query into vectors, then ranks them by a similarity measure such as cosine similarity. Similarity means geometric closeness under that model; it is not truth, entailment, or authority. Metadata filters, access controls, reranking, freshness, and source verification are still needed.

Scrum Application

An LLM can draft a Sprint Goal option, acceptance criteria, code, or a summary. The Scrum Team inspects the output using the source and evidence appropriate to the task. A model setting cannot decide value, quality, team effectiveness, or whether an Increment meets the Definition of Done. Treat outputs as candidates inside an empirical loop.