11.7 Troubleshooting Common GenAI Failure Modes & Agent Traces

11.7 Troubleshooting Common GenAI Failure Modes & Agent Traces

This independent study guide by OpenExamPrep helps candidates prepare for the AWS Certified Generative AI Developer - Professional (AIP-C01) examination. Operating production generative AI applications introduces novel failure modes that do not exist in conventional software engineering. Applications fail not only from standard network timeouts or database errors, but also from semantic drift, vector index retrieval mismatches, autonomous agent reasoning loops, malformed tool schemas, context window overflows, and prompt injection attacks.

Diagnosing these issues requires a systematic troubleshooting methodology and deep familiarity with AWS diagnostic tooling, specifically Amazon Bedrock Agent traces, Amazon Bedrock Guardrails logs, and AWS X-Ray distributed tracing.

---

1. RAG Failures: Hallucination vs. Retrieval Misalignment

When a RAG application yields incorrect answers, the developer must determine whether the failure occurred in the retrieval stage or the generation stage.

Diagnosing Retrieval Failures:

• Chunking Boundary Fragmentation: If a document is split using arbitrary fixed-size chunking (e.g., 200 tokens without overlap), critical semantic relationships—such as tabular data rows, mathematical formulas, or conditional clauses—are severed across chunks. Symptom: The retrieved chunks contain fragmented, incomplete sentences. Remedy: Implement semantic chunking or hierarchical chunking (parent-child chunking) with at least 15–20% token overlap.
• Embedding Model Asymmetry & Distance Thresholds: Using a general-purpose embedding model for highly technical or domain-specific legal text results in poor cosine similarity ranking. Furthermore, setting an excessively permissive distance threshold causes the vector store to return irrelevant documents. Symptom: Low Context Precision and Low Context Recall. Remedy: Transition to Amazon Titan Text Embeddings V2, enable hybrid search (BM25 lexical + dense vector), and apply a Cohere Rerank stage.

Diagnosing Generation Failures (Hallucination):

• Symptom: Inspection of the raw retrieved chunks confirms the correct facts are present, but the model emits contradictory or unverified statements.
• Remedy: Attach an Amazon Bedrock Guardrail configured with Contextual Grounding Checks. The grounding filter evaluates the generated response against the reference chunks. If the grounding confidence falls below the specified threshold (e.g., 0.75), Bedrock automatically blocks the hallucinated response and substitutes a configured fallback message.

---

2. Amazon Bedrock Agent Execution & Trace Diagnostics

Amazon Bedrock Agents utilize a ReAct (Reasoning + Acting) orchestration engine to interpret user requests, decompose complex tasks, invoke Action Group tools, and query Knowledge Bases. When an agent fails, developers debug execution by analyzing the trace stream returned by the InvokeAgent API.

The InvokeAgent Trace Architecture

When enableTrace=True is passed to InvokeAgent, Bedrock streams detailed diagnostic events categorized into four trace components:

┌─────────────────────────────────────────────────────────────────────────────┐
│                            InvokeAgent Trace Stream                         │
│                                                                             │
│  1. preProcessingTrace:                                                     │
│     - User query sanitization & input guardrail validation                  │
│     - Intent classification & contextual routing decisions                  │
│                                                                             │
│  2. orchestrationTrace (The ReAct Loop):                                    │
│     - rationale: Model's internal reasoning ("I need to check inventory")   │
│     - invocationInput: Target Action Group, API path, and extracted params  │
│     - observation: Raw JSON output returned by Lambda / Knowledge Base      │
│                                                                             │
│  3. postProcessingTrace:                                                    │
│     - Final response synthesis from observations                            │
│     - Output guardrail validation & sensitive data redaction                │
│                                                                             │
│  4. failureTrace (Emitted on Abort):                                        │
│     - Failure reason code & exact orchestration exception message           │
└─────────────────────────────────────────────────────────────────────────────┘

Common Agent & Action Group Failure Modes:

A. OpenAPI 3.0 Schema Discrepancies

The foundation model inspects the OpenAPI schema registered with an Action Group to determine which tool to call and how to format parameters. Common schema traps include:

• Type Mismatches: The schema defines a parameter as integer, but the model extracts it as a string (e.g., "1042"), or the schema expects an ISO-8601 date string while the model passes a timestamp.
• Missing Required Fields: An action definition marks a property as required in the OpenAPI schema, but the model cannot extract it from the user prompt and invokes the tool with null values.
• Diagnostic: The orchestrationTrace.invocationInput reveals invalid parameter formatting that fails API Gateway or Lambda input validation.

B. Missing Lambda Resource-Based Permissions

When an Action Group invokes an AWS Lambda function, the invocation is executed by the Amazon Bedrock service principal (bedrock.amazonaws.com). If the Lambda function lacks an IAM resource-based policy granting Bedrock invocation rights, the agent halts with an immediate failure.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowBedrockAgentInvocation",
      "Effect": "Allow",
      "Principal": {
        "Service": "bedrock.amazonaws.com"
      },
      "Action": "lambda:InvokeFunction",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:OrderManagementFunction",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:bedrock:us-east-1:123456789012:agent/*"
        }
      }
    }
  ]
}

C. Malformed actionResponse JSON Envelopes

The Lambda function backing an Action Group must return a response adhering strictly to the Bedrock Agent response contract. If the Lambda returns arbitrary JSON, Bedrock fails to parse the observation and emits a failureTrace.

{
  "messageVersion": "1.0",
  "response": {
    "actionGroup": "InventoryActionGroup",
    "apiPath": "/check-inventory",
    "httpMethod": "POST",
    "httpStatusCode": 200,
    "responseBody": {
      "application/json": {
        "body": "{\"sku\": \"SKU-9042\", \"availableStock\": 42, \"warehouse\": \"us-east-1\"}"
      }
    }
  }
}

[!WARNING] The Stringified Body Trap: The value of body inside responseBody.application/json must be a stringified JSON string, not a nested raw JSON object. Failing to stringify the body payload is the leading cause of ActionGroupInvocationException in Bedrock Agents.

---