11.1 Amazon Bedrock Model Evaluation (Automatic & Human Workflows)

11.1 Amazon Bedrock Model Evaluation (Automatic & Human Workflows)

Evaluation determines whether a model or RAG system meets a defined task, safety, and business contract. Current Amazon Bedrock evaluations support programmatic model evaluation, human-based evaluation, judge-model evaluation, and RAG evaluation. They can evaluate supported Bedrock resources and, for some workflows, responses produced outside Bedrock. Use the current CreateEvaluationJob API; the older name CreateModelEvaluationJob is not the current control-plane operation.

Start with a testable acceptance contract

Define the task, population, risks, metrics, thresholds, dataset, and decision before running the job. Accuracy alone is insufficient for a customer assistant that also needs faithfulness, safety, latency, and appropriate refusal. Include ordinary cases, rare edge cases, adversarial inputs, protected-group slices where relevant, and previously observed incidents.

Keep a final holdout set that prompt and model designers did not use for tuning. Version the dataset and preserve provenance, licensing, consent, redaction, and access controls. A test set copied from production logs can contain secrets or personal data and requires the same governance as the source workload.

Programmatic and judge-model evaluation

Programmatic evaluation runs a selected model over built-in or custom prompt data and produces task metrics. Available task types, datasets, and metrics vary by current service support. A custom JSONL dataset in Amazon S3 is the normal choice for domain-specific acceptance testing. The service role needs access to the exact input and output locations, permission to invoke selected models, and KMS permissions when customer-managed keys protect the data.

Judge-model evaluation uses another foundation model to score and explain candidate responses. It scales semantic review better than exact-match metrics, but it inherits judge bias, prompt sensitivity, and position effects. Calibrate it against expert-reviewed examples and periodically test agreement. Never treat a judge score as ground truth merely because it is numeric.

Metric design and uncertainty

Choose metrics from the failure cost. Exact match fits a narrow deterministic answer but penalizes harmless wording changes. Semantic judge scores capture meaning but require calibration. Toxicity or harmfulness metrics address safety but do not prove factual correctness. Structured applications also need schema validity and business-rule checks.

Report sample size and distribution, not just a mean. Slice results by language, task, risk class, prompt length, and user population. Review the worst failures and compute uncertainty where practical. A candidate whose average improves by one point but has a new critical safety failure does not pass.

Generative outputs can vary. Fix model and prompt configuration, repeat nondeterministic cases, and distinguish evaluator variance from candidate variance. Preserve raw response identifiers and evaluator explanations under appropriate access so reviewers can audit disputed scores.

Use custom datasets for the actual domain, while retaining a smaller general-capability set to detect broad regressions. Prevent data leakage by keeping the final set away from prompt tuning, fine-tuning, and judge-prompt development.

Evaluation job mechanics

The service role should trust Bedrock and grant only the S3 prefixes, KMS keys, and model invocation resources needed by the job. Keep input and output buckets in supported locations and validate Cross-Origin Resource Sharing settings when the documented workflow requires them. An AccessDenied failure is an IAM or key-policy investigation, not evidence that custom datasets are unsupported.

CreateEvaluationJob accepts an evaluation configuration, inference source, dataset metrics, output location, and role appropriate to the selected workflow. Use current API schemas from the SDK rather than copying an obsolete sample. Store the submitted configuration beside the report so later reviewers know what was actually scored.

Judge and RAG jobs incur model inference and evaluation cost. Limit development samples, then run the approved holdout once the rubric stabilizes. Repeatedly tuning against the final set converts it into another development set.

Evaluation datasets and reports are production assets. Apply bucket versioning or immutable release prefixes, encryption, restricted roles, retention, and deletion procedures. Avoid placing one customer's confidential prompts in a shared benchmark visible to unrelated teams.

Treat a failed evaluation job separately from a failed model candidate. IAM, dataset schema, unsupported metrics, or evaluator access can fail the job before a meaningful score exists. Do not convert infrastructure failure into a zero quality score or a release pass.