5.2 Structured Output and Evaluation

Schema-First Habit

If downstream code must parse the answer, design the schema before the prompt. Enumerate required fields, enum values, nested objects, and — critically — what happens on a validation failure. Then write the prompt around that contract. This prevents the classic failure the exam loves: output that reads beautifully to a human but breaks JSON.parse or a strict validator in production.

A schema-first object for a support-ticket extractor might require category (enum: billing, technical, account), priority (enum: low, medium, high), summary (string), and needs_human (boolean). Every field is named, every constrained field is an enum, and required lists the fields that must always appear. With the contract fixed, the prompt simply instructs Claude to fill it, and the validator rejects anything that does not conform.

Keep the schema as simple as the task allows. Strict structured-output schemas support common JSON types, enums, and nested objects, but some constraints (numeric ranges, string-length limits, recursive structures) are best enforced in your own validation code after parsing rather than expressed in the schema. The exam's point is the ordering discipline: fix the contract, generate against it, then validate the parsed object — not the precise list of supported keywords.

Programmatic Enforcement vs Prompt-Based Guidance

The exam draws a sharp line between asking the model for a format and enforcing it. Programmatic enforcement is stronger and is the preferred answer:

The robust production shape is: enforce the schema with structured outputs or strict tool use, then validate the parsed object in code and retry with the validation error fed back if it somehow fails. Always define the failure path: after N failed retries, return a safe default, escalate to a human, or surface a typed error — never ship unvalidated output downstream.

Evaluate Failure Paths, Not Just Demos

Evaluation is how an architect proves a system works and keeps working. The most common CCA-F trap is an answer that only evaluates happy paths. A real eval set deliberately includes hard cases:

• Ambiguous or underspecified user requests
• Tool calls that error or time out
• Invalid or unparseable JSON output
• Stale or empty retrieval results
• Adversarial prompts and safety-sensitive scenarios

A system that passes only on clean demo inputs is not production-ready. State the testable failure mode first, then pick the control that detects or mitigates it; that converts a vague "make it better" goal into observable behavior a team can measure and alert on.

Grading Methods

Different outputs need different graders, and the exam expects you to match them:

Programmatic checks are cheap and deterministic; reserve LLM-as-judge for genuinely open-ended quality where a rubric is the only practical grader.

Regression Evals as a Change Gate

Evals earn their keep when they run as a regression gate. Maintain a small, version-controlled eval set and re-run it before and after any change that could move quality: a model upgrade, a prompt edit, a new tool or MCP integration, or a retrieval change. If a score drops, the change is blocked or flagged before it reaches users. Anthropic frames evals as the safety net that catches problems before real users do — and the exam consistently rewards the answer that adds a regression eval over the answer that simply adds another retry or reads the output by hand.

Close the loop: sample real production traffic into the eval set so the suite reflects how users actually behave, and alert when a live metric (schema-validation failure rate, refusal rate, citation-missing rate, p95 latency, cost per request) drifts past a threshold. An eval suite that never grows from real failures slowly stops representing reality.

Structured Output and Eval Decision Table

Match the symptom to the control, define the failure path explicitly, and prefer programmatic enforcement over hoping the prompt holds — that discipline is exactly what this half of Domain 4 measures.