5.4 Retrieval-Augmented Generation (RAG)

Quick Answer: RAG enhances generative AI by retrieving relevant documents from external sources and including them as context in the prompt. This grounds the model's response in factual data, reducing hallucinations and enabling answers about proprietary or recent information. Azure AI Search is the primary Azure service for implementing RAG.

What Is RAG?

Retrieval-Augmented Generation (RAG) is an architecture pattern that combines information retrieval with generative AI to produce more accurate, grounded, and up-to-date responses.

The Problem RAG Solves

Generative models have two fundamental limits:

1. Knowledge cutoff — they do not know about events after their training data was collected.
2. No proprietary data — they have no access to your organization's private documents, databases, or systems.

RAG solves both by retrieving relevant information and injecting it into the prompt before the model generates a response.

RAG vs. Standard Generation

How RAG Works

The Three-Step RAG Process

Step 1: Retrieve

• The user asks a question.
• The system searches a knowledge base (Azure AI Search) for relevant documents.
• Retrieval can use keyword search, vector search, or hybrid search.
• The top-K most relevant passages are selected.

Step 2: Augment

• Retrieved passages are added to the prompt as context.
• The prompt now includes: system message + retrieved context + user question.
• This "augments" the model's knowledge with specific, relevant data.

Step 3: Generate

• The model produces a response grounded in the retrieved context.
• The answer is based on the provided documents, not just pre-trained memory.
• This significantly reduces hallucinations.

RAG Example

Without RAG: "What is our company's parental-leave policy?" -> the model has no access to your policy and may either decline or hallucinate one.

With RAG: (1) Retrieve the policy from HR documents, (2) Augment the prompt with that document, (3) Generate: "According to your policy, employees receive 16 weeks of paid parental leave for both birth and adoptive parents."

Embeddings and Vector Search

What Are Embeddings?

Embeddings are numerical vector representations of text that capture its semantic meaning. Similar meanings produce similar vectors, which enables semantic search.

Traditional keyword search matches exact words: "reset password" only finds documents with those exact words. Semantic search with embeddings matches meaning: "reset password" also finds "change credentials," "update login," or "recover account access," even without the literal words.

Types of Search in Azure AI Search

Azure AI Search for RAG

Azure AI Search (formerly Azure Cognitive Search) is the primary service for building RAG solutions on Azure:

RAG Architecture on Azure (conceptual flow)

User question -> Azure AI Search retrieves relevant documents -> prompt is built (system message + retrieved docs + question) -> Azure OpenAI generates a grounded response -> grounded answer is returned to the user.

A practical detail worth knowing: source documents are usually split into smaller chunks before indexing, and each chunk is converted into an embedding. At query time the system embeds the user's question, finds the closest chunks, and passes only those into the prompt. This keeps the prompt within the model's context window and ensures the most relevant passages — not entire documents — are used as grounding.

On the Exam: RAG uses a retrieval step (Azure AI Search) to find relevant documents, then includes them in the prompt for the generative model (Azure OpenAI). This grounds responses and reduces hallucinations. You do NOT need to know how to implement RAG — just the concept and why it matters.

RAG vs. Fine-Tuning

On the Exam: If a question asks about answering from company documents or providing up-to-date information, RAG is usually correct. Fine-tuning is for changing the model's overall behavior or style, not for injecting specific, changing knowledge.