7.4 Contextual Grounding Checks & Hallucination Prevention

7.4 Contextual Grounding Checks & Hallucination Prevention

This independent study guide by OpenExamPrep helps candidates prepare for the AWS Certified Generative AI Developer - Professional (AIP-C01) examination. Retrieval-Augmented Generation (RAG) is the foundational architecture for enterprise AI applications, combining the vast parametric knowledge of foundation models with private, authoritative corporate data retrieved from vector databases. However, even when provided with accurate context chunks, foundation models remain susceptible to hallucinations—generating statements that sound authoritative but are factually unsupported by, or directly contradict, the retrieved source materials.

In mission-critical industries such as clinical healthcare, wealth management, and statutory legal compliance, hallucinations represent catastrophic operational and regulatory liabilities. Amazon Bedrock Guardrails addresses this vulnerability through its Contextual Grounding Policy, a built-in verification engine that mathematically evaluates model outputs for factual grounding and query relevance before returning text to end users.

---

The Dual-Score Evaluation Model

The Contextual Grounding Policy evaluates generated text across two distinct, complementary dimensions: Grounding and Relevance. Each dimension is evaluated independently and scored on a continuous scale from 0.0 (worst) to 1.0 (best).

[User Prompt] ────────────────────────────────────────┐
                                                      │
[Retrieved Context Passages] ────────┐                │
                                     ▼                ▼
[Foundation Model] ──► [Generated Response] ──► [Contextual Grounding Policy]
                                                      │
                           ┌──────────────────────────┴──────────────────────────┐
                           │                                                     │
                           ▼                                                     ▼
               [Grounding Evaluation]                                 [Relevance Evaluation]
            Is every factual claim supported                       Does the response directly answer
            by the retrieved context?                              the user's prompt?
                           │                                                     │
                     Grounding Score                                       Relevance Score
                   (e.g., 0.92 >= 0.85)                                  (e.g., 0.88 >= 0.75)
                           │                                                     │
                           └──────────────────────────┬──────────────────────────┘
                                                      │
                                           Both Scores >= Threshold?
                                            ├── YES ──► Emit Response to User
                                            └── NO  ──► Block & Emit Fallback Message

1. Grounding Score (Factual Substantiation)

• Core Definition: The Grounding Score evaluates whether the factual claims, entities, statistics, and assertions in the model-generated output are fully substantiated by the provided reference text (grounding source).
• Scoring Range: 0.00 to 1.00.
  • A score of 1.00 indicates that 100% of the assertions in the generated text are backed by direct evidence in the retrieved source passages.
  • A score near 0.00 indicates that the model has completely fabricated facts, extrapolated unsupported claims, or relied on out-of-date parametric training data that contradicts the retrieved context.
• Evaluation Mechanism: Guardrails uses a managed natural language inference (NLI) model that decomposes the generated completion into discrete atomic claims and performs cross-entropy entailment checks against the grounding source chunks.

2. Relevance Score (Query Responsiveness)

• Core Definition: The Relevance Score evaluates whether the generated output directly answers the user's specific prompt or query.
• Scoring Range: 0.00 to 1.00.
  • A score of 1.00 indicates that the response is tightly aligned, directly responsive, and focused on answering the user's question.
  • A low score indicates that the model generated irrelevant filler, went off on an unprompted tangent, hallucinated an answer to a completely different question, or attempted to evade answering.

---

Threshold Configuration & Blocking Mechanics

Developers configure minimum acceptable threshold values for both Grounding and Relevance when defining the Guardrail:

"contextualGroundingPolicyConfig": {
    "filtersConfig": [
        {
            "type": "GROUNDING",
            "threshold": 0.85
        },
        {
            "type": "RELEVANCE",
            "threshold": 0.75
        }
    ]
}

Execution and Intervention Logic

At runtime, when the foundation model completes generating a response, the Guardrail evaluates the output against both thresholds:

• Pass Condition: If ActualGroundingScore >= GroundingThreshold AND ActualRelevanceScore >= RelevanceThreshold, the output passes validation and is delivered to the client application.
• Intervention Condition: If either score falls below its respective configured threshold:
  1. The generated response is immediately discarded.
  2. The Guardrail sets action: "GUARDRAIL_INTERVENED".
  3. In the Converse API, the response returns stopReason: "guardrail_intervened".
  4. The client receives the configured blockedOutputsMessaging (e.g., "The generated response could not be verified against authoritative reference documentation.").

---

Regulated Industry Trade-offs: Threshold Tuning

A critical domain tested on the AIP-C01 exam is calibrating threshold values to balance factual precision against false refusal rates (over-blocking):

The False Refusal Trade-off

• Setting the Grounding threshold excessively high (e.g., 0.98 or 1.00) creates severe usability degradation. Natural language models frequently synthesize answers using connective transitional phrases (e.g., "Based on the internal company records provided above, the primary protocol is..."). Because the introductory phrase itself does not exist in the source document, an overly strict threshold may penalize the text and trigger false refusals.
• Conversely, setting the Grounding threshold too low (e.g., < 0.50) allows subtle, dangerous hallucinations to pass through undetected.

---

Integrating Contextual Grounding in RAG Pipelines

Depending on the system architecture, developers integrate Contextual Grounding through either managed Amazon Bedrock Knowledge Bases or custom RAG pipelines:

Pattern 1: Managed Bedrock Knowledge Bases

When using Bedrock Knowledge Bases via the RetrieveAndGenerate API, integration is fully automated. By associating a Guardrail with the Knowledge Base or passing guardrailConfiguration in the API call, Bedrock automatically routes the retrieved document passages as grounding sources directly into the Guardrail evaluation engine without custom coding.

Pattern 2: Custom RAG Pipelines with ApplyGuardrail

For custom RAG architectures utilizing self-managed vector stores (e.g., Amazon Aurora PostgreSQL with pgvector, Pinecone, or LangChain agents), developers evaluate outputs using the apply_guardrail runtime API:

import boto3

bedrock_runtime = boto3.client('bedrock-runtime', region_name='us-east-1')

# User prompt and retrieved vector chunks from custom vector database
user_query = "What is the maximum reimbursement limit for business travel meals?"
retrieved_context_chunk = "Section 4.2: Business travel meals are reimbursed up to a maximum of $75 per day with itemized receipts."
model_generation = "The daily reimbursement limit for meals during corporate travel is $75, provided you submit itemized receipts."

response = bedrock_runtime.apply_guardrail(
    guardrailIdentifier='gr-finance-safety-prod',
    guardrailVersion='1',
    source='OUTPUT',
    content=[
        {
            'text': {
                'text': model_generation,
                'qualifiers': ['grounding_source']  # Contextual reference passed here
            }
        }
    ],
    # For Contextual Grounding, pass query and source passages in the request structure
)

# Inspect assessment trace
for assessment in response.get('assessments', []):
    cg_policy = assessment.get('contextualGroundingPolicy', {})
    for cg_filter in cg_policy.get('filters', []):
        filter_type = cg_filter['type']       # 'GROUNDING' or 'RELEVANCE'
        score = cg_filter['score']             # e.g., 0.94
        threshold = cg_filter['threshold']     # e.g., 0.85
        action = cg_filter['action']           # 'NONE' or 'BLOCKED'
        print(f"{filter_type} -> Score: {score}, Threshold: {threshold}, Action: {action}")

---

Performance, Latency, and Cost Considerations

• Inference Latency Overhead: Contextual Grounding evaluation requires running a secondary natural language inference model over the generated completion and reference chunks. In production systems, developers should anticipate an additional 150 to 350 milliseconds of evaluation latency on the output stream.
• Pricing Structure: Amazon Bedrock Guardrails pricing applies a per-text-unit fee for policy evaluations. Contextual Grounding evaluations are billed based on the number of text units evaluated across the input prompt, grounding context, and model output.
• CloudWatch Observability: Guardrails emits real-time metrics to Amazon CloudWatch under the AWS/Bedrock namespace:
  • ContextualGroundingFilterIntervened: Counts the number of times output was blocked due to grounding or relevance failure.
  • ContextualGroundingGroundingScore: Emits statistical distribution of grounding scores, enabling developers to detect degradation in Knowledge Base retrieval quality.
  • ContextualGroundingRelevanceScore: Tracks prompt responsiveness over time.

---

Common Exam Traps & High-Stakes Scenarios

• Trap: Believing Contextual Grounding Evaluates Input Prompts. Contextual Grounding evaluates model outputs in relation to retrieved context and user queries. It cannot evaluate input prompts because grounding context only validates generated claims.
• Trap: Conflating Grounding Score with Relevance Score. Grounding measures factual support by reference context (factuality). Relevance measures responsiveness to the user prompt (alignment). A response can be 100% grounded (quoting facts accurately from context) while scoring 0.0 on relevance (completely ignoring the user's question).
• Trap: Assuming Contextual Grounding Only Works with Bedrock Knowledge Bases. Contextual Grounding functions seamlessly with any RAG architecture (including custom LangChain, LlamaIndex, or Aurora pgvector pipelines) via the standalone ApplyGuardrail API.
• Trap: Over-tuning Thresholds to 1.0 in Production. Setting Grounding or Relevance thresholds to 1.0 causes widespread false refusals because natural connective syntax or polite conversational phrasing will fail a strict mathematical 100% entailment test.