11.2 Human, RAG & Continuous Evaluation
Key Takeaways
- Programmatic human evaluations use a SageMaker flow definition and private work-team ARN.
- Evaluate retrieval and response generation separately to target the actual failure stage.
- Continuous evaluation reruns after model, prompt, Guardrail, retrieval, tool, or incident changes.
11.2 Human, RAG & Continuous Evaluation
Human-based evaluation
Human review is appropriate for tone, usefulness, complex reasoning, policy interpretation, and high-risk sign-off. Current Bedrock workflows use a private work team. When created programmatically, the workflow uses a SageMaker AI flow definition and work-team ARN. The evaluation job connects prompts and responses to that human workflow.
Write a scoring rubric with observable anchors. 'Good' is not reproducible; 'all claims are supported by the cited policy and no required exception is omitted' is testable. Use multiple reviewers for subjective or high-risk tasks, measure agreement, adjudicate disagreements, and train reviewers with examples.
Do not claim that proprietary prompts can never leave an organization's boundary solely because a private workforce was selected. Verify data flow, worker identity, S3 and KMS access, retention, browser environment, contractual controls, and the organization's definition of its security boundary.
RAG evaluation
Evaluate retrieval separately from response generation. Bedrock RAG evaluations support retrieve-only and retrieve-and-generate jobs for supported sources. Retrieve-only metrics include whether context is relevant and covers expected evidence. Generate-stage metrics include correctness, completeness, helpfulness, logical coherence, faithfulness, citation precision and coverage, harmfulness, stereotyping, and refusal, depending on the chosen workflow.
A retrieval failure requires changes to source data, chunking, embeddings, metadata, query transformation, top-k, or reranking. A generation failure with good evidence requires changes to prompt, model, Guardrails, or response validation. An end-to-end score alone obscures this distinction.
Continuous evaluation and release gates
Run a fixed regression suite for every material model, prompt, Guardrail, retrieval, or tool change. Add incident examples to a separate regression set after remediation. Compare the candidate with the active release using confidence intervals or repeated samples where nondeterminism matters. Require safety and policy thresholds as hard gates even when average quality improves.
A production gate can require:
- Dataset and configuration validation.
- Offline model and RAG metrics above thresholds.
- Red-team and structured-output tests.
- Human review for high-risk slices.
- Canary traffic with rollback triggers.
- Post-deployment monitoring and sampled reevaluation.
Evaluation reports should identify the model, configuration, dataset version, metric implementation, timestamp, sample size, uncertainty, failures, and approval decision. A green average without named failures is not sufficient evidence for a professional production release.
Human rubric and stakeholder report
A rubric defines each metric, scale, and anchor. For a five-point faithfulness score, describe what fully supported, partially supported, contradicted, and unverifiable answers look like. Show reviewers examples, randomize candidate order for pairwise tasks, and measure agreement. Adjudicate low-agreement items rather than averaging confusion.
Limit each reviewer to authorized data. Use a private work team, role-scoped S3 access, optional KMS controls, time-bounded task availability, and a retention plan. Review worker instructions for accidental disclosure and avoid placing secrets in free-text examples.
The final report includes the evaluated release, dataset version, model or source configuration, metrics, thresholds, slice results, uncertainty, human-review method, failed examples, known limitations, and release decision. Visualizations should help stakeholders find tradeoffs rather than conceal them in one composite score.
After a canary, join offline metrics with user feedback, task completion, overrides, incidents, latency, and cost. Production data can reveal distribution shift, but feedback is biased toward users who choose to respond. Sample deliberately and continue scheduled benchmark runs.
Feedback loop controls
User ratings are signals, not labels automatically suitable for training. Detect spam, selection bias, duplicate feedback, and complaints caused by factors outside the model. Route high-risk negative feedback to review, redact sensitive content, and obtain the rights needed before reuse.
Annotation workflows need a queue, instructions, reviewer identity, disagreement handling, and quality checks such as known control items. Preserve the connection among prompt, response, cited evidence, release version, and rating. Otherwise the team cannot tell which component improved.
Close the loop by creating a candidate fix and rerunning the fixed benchmark plus the broader regression suite. Do not update production prompts continuously from raw ratings without review, because malicious or unrepresentative feedback can steer behavior.
Track evaluation debt: known failed slices, deferred metrics, reviewer disagreement, and missing production evidence. An approval can be conditional, but the condition needs an owner, due date, traffic limit, and automated guardrail against silent expansion.
For high-impact workflows, retain a small adjudicated benchmark that remains stable across releases. It provides continuity when user traffic and feedback distributions change, while the larger evolving dataset captures new failure modes.
Which current Bedrock API creates model or RAG evaluation jobs?
A RAG response is unfaithful even though the expected evidence was retrieved. Which stage should be remediated first?