2.3 Retrieval-Augmented Generation (RAG) & Grounding

2.3 Retrieval-Augmented Generation and Grounding

Core principle: RAG gives a model selected external context at request time. It can improve access to current or private sources, but retrieval and generation can both fail. A citation is useful only when it points to a relevant passage that actually supports the claim.

---

What RAG Changes

A pretrained model's parameters do not automatically contain an organization's current Product Goal, Definition of Done, architecture decisions, or approved policy. Retrieval-augmented generation (RAG) searches an external corpus for material relevant to a request and places selected passages in the model context. The base model weights remain unchanged.

A common pipeline has two parts.

Ingestion and indexing

1. admit only authorized, appropriate sources;
2. parse documents and preserve title, version, owner, effective date, access label, and location;
3. split material into coherent chunks;
4. create embeddings or another searchable representation;
5. store content and metadata in an index;
6. update or delete entries when originals change.

Request time

1. interpret or rewrite the query;
2. retrieve candidates with semantic, keyword, or hybrid search;
3. apply permission filters and perhaps rerank results;
4. assemble selected passages with trusted instructions and the user task;
5. generate a response with source locations;
6. validate the output and any proposed action.

Not every system uses a vector database, top-k retrieval, or the same sequence. RAG is an architectural family, not one mandatory vendor stack.

What RAG Is Good For

RAG is often appropriate when knowledge changes more frequently than a model can be retrained, content is organization-specific, answers should reference sources, or access rules differ by user. Examples include locating an architecture decision, comparing a Product Backlog item with the current Definition of Done, or summarizing approved customer research.

Fine-tuning serves a different purpose. It can adapt stable behavior, form, or task performance by updating model parameters. Prompting, retrieval, tools, fine-tuning, and deterministic search can be combined. RAG is not always the cheapest or best choice: a direct database query or ordinary search may be more reliable for exact facts.

Failure Modes

Re-indexing is not instantaneous unless the actual pipeline provides and verifies that service level. An updated document may not be searchable until ingestion, parsing, embedding, replication, and cache invalidation complete.

Evaluate Retrieval and Generation Separately

For retrieval, build representative questions with known relevant sources and measure whether authorized passages are found. Inspect false positives, permission behavior, freshness, and performance for different user vocabulary. For generation, check whether each claim is entailed by its cited passage, whether material contradictions are disclosed, whether unknowns remain unknown, and whether the required format is valid.

Asking a model to attach citations does not ensure those citations exist or support the prose. Follow the link and read the passage. For an exact count or state, query the authoritative system directly when possible.

Scrum Example

A Product Owner asks an assistant for themes in approved interview notes. The system retrieves consented, access-controlled passages and returns themes with source identifiers and counterexamples. The Product Owner and Scrum Team inspect the evidence and may formulate Product Backlog hypotheses. Synthetic themes do not replace discovery, and the model does not order the Product Backlog.

A Developer asks whether a change meets an architecture rule. The assistant retrieves the current decision record and cites the relevant paragraph. Developers verify the source and actual implementation. RAG supports transparency by making provenance inspectable; it does not declare the Increment Done.

Security Boundary

RAG can expose private data as effectively as direct search. Apply least privilege, tenant and document authorization, data minimization, logging appropriate to privacy, deletion, and incident handling. Retrieved passages can contain personal data or prompt injection. Limit downstream tools so a bad passage cannot trigger an unrestricted external action.