11.4 RAG and Agent Performance Evaluation

11.4 RAG and Agent Performance Evaluation

Ragas Framework Deep-Dive (Retrieval Augmented Generation Assessment)

Ragas is an open-source evaluation framework specifically architected to benchmark RAG pipelines using "LLM-as-a-judge" methodologies. Ragas evaluates four core metrics that map directly to the RAG Triad:

                  ┌───────────────────┐
                  │    User Query     │
                  └─────────┬─────────┘
                            │
           Context Precision│▲ Context Recall
                   (Retriever Evaluation)
                            │
                            ▼
                  ┌───────────────────┐
                  │ Retrieved Context │
                  └─────────┬─────────┘
                            │
               Faithfulness │  Answer Relevance
                    (Generator Evaluation)
                            │
                            ▼
                  ┌───────────────────┐
                  │ Generated Answer  │
                  └───────────────────┘

1. Faithfulness (Groundedness)

• Definition: The ratio of claims made in the generated answer that can be directly verified against the retrieved context.
• Scoring Process: The evaluator LLM parses the generated answer into discrete atomic claims. For each claim, the evaluator checks if it is logically entailed by the retrieved context chunks.
• Formula: $\text{Faithfulness} = \frac{|\text{Number of verifiable claims in answer supported by context}|}{|\text{Total claims generated in answer}|}$
• Target: A score approaching 1.0 indicates zero hallucination.

2. Answer Relevance

• Definition: Measures how pertinent the generated response is to the input query.
• Scoring Process: Ragas instructs an evaluator model to generate $N$ potential questions based exclusively on the generated answer. It then calculates the mean cosine similarity between the embeddings of the generated questions and the original user query.
• Key Characteristic: Highly penalizes evasive answers, incomplete responses, or tangential verbosity, even if the answer is factually grounded.

3. Context Precision

• Definition: Evaluates whether all ground-truth-relevant context chunks are ranked at the very top of the retrieved context list.
• Calculation: Employs Mean Reciprocal Rank (MRR) and Average Precision principles. If relevant chunks are buried at positions 4 and 5 while positions 1 and 2 contain irrelevant filler, Context Precision drops significantly.
• Impact: LLMs suffer from the "lost in the middle" phenomenon—information placed at the top or bottom of a context window is recalled far more reliably than information in the middle.

4. Context Recall

• Definition: Measures whether the retrieved context contains all the necessary information required to answer the prompt, evaluated against a reference ground truth.
• Formula: $\text{Context Recall} = \frac{|\text{Number of reference ground-truth sentences attributable to context}|}{|\text{Total sentences in reference ground truth}|}$
• Impact: Reflects the fundamental performance of the retrieval engine (embedding model + vector index).

---

Diagnostic Matrix: Pinpointing & Fixing RAG Failure Signatures

The true power of Ragas metrics lies in cross-metric diagnostic analysis. By inspecting metric combinations, developers can pinpoint the exact architectural component causing failures and apply targeted AWS remediations:

[!IMPORTANT] Exam Diagnostic Tip: On the AIP-C01 exam, if a question describes a scenario where "the retrieved passages contain the exact answer, but the model outputs unverified claims not present in the reference documents," the failure is Low Faithfulness (hallucination). The correct architectural fix on AWS is configuring Contextual Grounding Checks in Amazon Bedrock Guardrails, not altering the chunking strategy or vector database.

Native RAG and agent evaluation

Amazon Bedrock RAG evaluation can score retrieve-only behavior separately from retrieve-and-generate behavior. Current built-in concepts include context relevance and coverage for retrieval and correctness, completeness, helpfulness, coherence, faithfulness, citation quality, harmfulness, stereotyping, and refusal for generated responses. Supply ground truth where the selected metric requires it and preserve evaluator-model configuration.

Agent evaluation adds task-level evidence. Measure task completion, correct tool selection, parameter validity, tool success, unnecessary calls, latency, cost, stopping behavior, and the quality of multi-step outcomes. Amazon Bedrock agent evaluations can support managed assessment for eligible agents; custom traces and golden workflows cover application-specific tools. Do not grade an agent only on final prose—a plausible answer can hide a failed or unauthorized action path.