5.1 Generative AI Fundamentals
Key Takeaways
- Generative AI creates NEW content (text, images, code, audio) rather than only analyzing or classifying existing data; it is the largest AI-900 domain at 20-25%.
- Large Language Models (LLMs) are deep neural networks that predict the next token; GPT stands for Generative Pre-trained Transformer.
- The Transformer's self-attention mechanism lets the model weigh all words at once, enabling context understanding and long context windows (GPT-4o ~128K, GPT-4.1 up to ~1M tokens).
- Customization progresses from pre-training (most expensive) to fine-tuning (moderate) to prompt engineering (cheapest, no training) — the AI-900 emphasizes prompt engineering.
- Key limitations are hallucinations (confident false output), bias, knowledge cutoff, non-determinism, and prompt injection; RAG and grounding reduce hallucinations.
Quick Answer: Generative AI creates new content (text, images, code, audio) using large language models (LLMs) built on the Transformer architecture. LLMs like GPT are pre-trained on massive datasets and can generate human-like text, answer questions, summarize documents, and write code. Key limitations include hallucinations, bias, and lack of real-time knowledge. Generative AI is the largest AI-900 domain at 20-25% after the May 2025 update.
What Is Generative AI?
Generative AI is a category of AI that creates new, original content. Unlike traditional AI that analyzes or classifies existing data ("is this email spam?"), generative AI produces new outputs that did not exist before.
What Generative AI Can Create
| Content Type | Example | Model Family |
|---|---|---|
| Text | Articles, emails, summaries, code, poetry | GPT-4.1, GPT-4o |
| Images | Photographs, illustrations, design concepts | GPT-image-1 (DALL-E 3 superseded) |
| Code | Functions, classes, entire applications | GPT-4.1, GPT-4o |
| Audio | Transcription, synthesized speech | Whisper (STT), neural TTS |
| Conversations | Chatbot responses, assistant dialogues | GPT-4.1, GPT-4o |
Generative AI vs. Traditional AI
| Aspect | Traditional AI | Generative AI |
|---|---|---|
| Task | Analyze and classify existing data | Create new content |
| Output | Labels, scores, predictions | Text, images, code, audio |
| Example | "This email is spam" (classification) | "Write a professional email about..." (generation) |
| Data flow | Input -> analysis -> label | Prompt -> generation -> new content |
Large Language Models (LLMs)
Large Language Models are the neural networks that power text-based generative AI. They are "large" because they have billions of parameters — adjustable values learned during training.
How LLMs Work (Conceptual)
- Pre-training: the model is trained on massive amounts of text (books, websites, articles, code) and learns the statistical patterns of language — which words tend to follow which.
- Token prediction: at its core, an LLM predicts the next token (a word or word-part). Given "The capital of France is", it predicts "Paris" as the most likely next token.
- Context understanding: through the Transformer's self-attention mechanism, the model interprets the context of words — "bank" differs in "river bank" vs. "bank account".
- Emergent capabilities: large models exhibit abilities that were not explicitly trained, such as multi-step reasoning and following complex instructions.
Key LLM Concepts
| Concept | Definition | Example |
|---|---|---|
| Token | A unit of text (word, subword, or character) | "Hello world" ~ 2 tokens; "unbelievable" ~ 3 tokens |
| Context window | Maximum tokens processed at once | GPT-4o ~ 128K tokens; GPT-4.1 supports up to ~1,000,000 |
| Parameters | Internal values learned during training | Frontier models: hundreds of billions |
| Temperature | Controls randomness (0 = deterministic, higher = creative) | 0 for factual; 0.7 for creative writing |
| Inference | Generating output from the model | Sending a prompt and receiving a response |
The Transformer Architecture
The Transformer is the neural-network architecture that makes modern generative AI possible. It was introduced in the 2017 paper "Attention Is All You Need."
Why Transformers Changed Everything
Before Transformers, language models processed text sequentially (one word at a time). Transformers introduced self-attention, letting the model consider ALL words in the input at once:
| Old Approach (RNNs) | Transformer Approach |
|---|---|
| Process words one at a time (sequential) | Process all words simultaneously (parallel) |
| Struggle with long-distance relationships | Capture relationships across the entire text |
| Slow to train | Fast to train (parallelizable on GPUs) |
| Limited context | Very long context windows |
Self-Attention Explained (Conceptual)
Self-attention lets each word "attend to" (weigh the relevance of) every other word. In "The cat sat on the mat because it was tired," self-attention determines that "it" refers to "cat," not "mat," by scoring the relevance of every word. This is how the model understands context and references.
Pre-Training, Fine-Tuning, and Prompting
| Approach | Cost | What It Does |
|---|---|---|
| Pre-training | Highest | Train a foundation model on massive general data (thousands of GPUs for weeks) |
| Fine-tuning | Moderate | Adapt a pre-trained model to a domain with a smaller labeled dataset |
| Prompt engineering | Lowest | Use the model as-is by crafting effective prompts — no training needed |
On the Exam: Know the progression — pre-training (expensive, builds the foundation) -> fine-tuning (moderate, adapts a domain) -> prompt engineering (cheap, customizes via instructions). The AI-900 focuses on prompt engineering because it requires no ML expertise.
Limitations of Generative AI
1. Hallucinations
The model generates information that sounds plausible but is factually wrong. It does not "know" facts — it predicts statistically likely text. For example, "Einstein won the Nobel Prize in 1922 for relativity" sounds right but is wrong: he won the 1921 prize, awarded for the photoelectric effect.
2. Bias
Models inherit biases in their training data and can perpetuate stereotypes or underrepresentation.
3. Knowledge Cutoff
Pre-trained models do not know about events after their training data was collected — a key reason grounding and RAG matter.
4. Non-Determinism
The same prompt can yield different responses depending on temperature and sampling, so outputs are not guaranteed to repeat.
5. Prompt Injection
Malicious users can craft inputs that trick the model into ignoring its instructions or producing harmful content.
On the Exam: Hallucination is the most commonly tested limitation. Know that generative AI can produce convincing false information, and that RAG (Retrieval-Augmented Generation) and grounding are the key techniques to reduce it.
What is the key difference between generative AI and traditional AI?
What is a "hallucination" in the context of generative AI?
What does "GPT" stand for in models like GPT-4o and GPT-4.1?
Which feature of the Transformer architecture allows it to understand context by considering the relevance of all words in the input simultaneously?
Which approach to customizing generative AI is the fastest and least expensive?