4.2 Retrieval-Augmented Generation (RAG)

How RAG is used for AI solutions

Retrieval-augmented generation (RAG) is a three-step pattern. When a user asks a question, the application retrieves relevant chunks (passages) from an enterprise store, augments the prompt with those chunks, and then generates an answer with the chunks in context. Microsoft Foundry documents this retrieve-then-augment-then-generate flow. The model is still a generator. The difference is that it is generating while looking at your passages, not only at what it memorized in pretraining.

A closed model — pretrained weights only — cannot reliably answer "What is our current parental-leave maximum?" That sentence was never in the public training mix, or it is older than your last policy change. RAG does not rewrite the foundation model. It hands the model the current paragraph from the HR handbook at inference time. That is why fabrications drop: the model has a specific text to stay close to, and you can demand citations to that text.

Fabrications do not go to zero. Microsoft's RAG limitations are blunt. If retrieval returns irrelevant or incomplete passages, the model can still produce incomplete or inaccurate answers despite grounding. Leaders should hear that sentence before they fund a RAG program. Grounding is a quality multiplier for good retrieval. It is not an insurance policy for a messy corpus.

Azure AI Search as the common retrieval layer

Azure AI Search is Microsoft's fully managed retrieval service for classic search and for RAG. You will meet it again in the Foundry Tools chapter. For this skill, remember the leader view: Search hosts indexes — and, in agentic patterns, knowledge sources and knowledge bases — that store chunked enterprise content for keyword, vector, hybrid, and multimodal queries.

Classic search is an index-first, single-request pattern: one query, ranked documents, no model planning during retrieval. Agentic retrieval, also called agentic RAG, uses a model to break a complex question into focused subqueries, run them in parallel, rerank, and return structured grounding data with citations. Some of that surface is generally available. Richer synthesis and related features may still be preview. AB-731 does not ask you to write the pipeline. It asks you to know that Azure AI Search is the common retrieval layer Microsoft Foundry connects to.

Foundry's RAG workflow for a custom app is: prepare and chunk private documents, create an Azure AI Search index, connect the Foundry project to that index, generate with retrieved context, then test and evaluate. Foundry IQ is the managed knowledge layer that turns enterprise content into permission-aware knowledge bases for agents in the Microsoft Foundry portal. Treat that as a product name you will map later. The business object you own now is the data estate feeding the index.

Azure AI Search can pull from supported platforms such as Azure Blob Storage, Azure Cosmos DB, Microsoft SharePoint, and Microsoft OneLake, or accept pushed JSON. It can enrich content at index time with chunking and embeddings. Hybrid search combines full-text precision with vector recall. Those capabilities matter because leaders choose whether the retrieval layer can actually find the paragraph operations will trust.

Leader view: data estate and indexes, not embeddings code

Embeddings are numeric representations used for vector similarity search. Someone on the engineering team will choose an embedding model and a chunk size. That is not the Transformation Leader's deliverable. Your deliverables are:

• Which systems of record are in scope, and which are excluded.
• Who owns quality, access, and the definition of a trusted document.
• Whether a shared index is acceptable, or whether isolation requires separate indexes — an HR handbook and a product maintenance manual are a classic split.
• How often the index must refresh so inference is not run on stale copies.
• Which document-level access control or security filters apply at retrieval time.
• How you will measure retrieval quality and groundedness before go-live.

Microsoft Foundry includes groundedness, relevance, and retrieval evaluators so teams can score whether answers stay inside retrieved context. Groundedness asks whether the response fabricates beyond the provided context. Retrieval asks whether the chunks were even relevant to the question. Leaders ask for those scores on representative questions. They do not configure the software development kit.

Fine-tuning is the wrong reflex when the need is fresh private knowledge. Microsoft Foundry guidance is to use RAG when you need answers grounded in private or frequently changing data, and to use fine-tuning when you need to change model behavior, style, or task performance. Buying a larger model also does not replace an index.

Limitations you must budget

• Bad retrieval yields bad answers. Chunking that splits a table in half, an index that omits the policy appendix, or a hybrid query that ranks the wrong product line will still produce a fluent wrong answer.
• Access control must apply to retrieved documents. Azure AI Search supports document-level access control with security filters, Microsoft Entra ID for production — API keys are for development — and patterns that inherit user permissions from SharePoint or from blob access control lists. If the index is a flattened copy without security trimming, RAG becomes an accidental data-exfiltration engine.
• Treat retrieved content as untrusted input. A document can hide a prompt-injection payload. System messages and application logic need to reduce that risk.
• Retrieval, embeddings, and extra prompt tokens add cost and latency on top of the model call.
• Hallucination despite grounding remains possible. Require citations and instructions to stick to retrieved content.

In practice: A chief financial officer funds a contract question-and-answer bot. Legal discovers the index ingested a shared folder that included a counterparty draft marked Highly Confidential, and that intern accounts could retrieve it. The RAG pattern was correct. The permission design was not. The CFO's correction is to stop go-live until security filters match the source access lists, not to buy a larger model.