6.2 Model Selection and Reliability
Key Takeaways
- Model selection is a cost-latency-quality trade-off: route high-volume simple turns to a fast, cheap model and escalate only the hard turns to a more capable one.
- Do not default to the largest model; the exam penalizes paying for capability a task does not need, and rewards matching the model to the cost-of-error.
- Reliability controls are deliberate choices: retries with backoff for transient errors, timeouts to bound latency, fallbacks to degrade gracefully, and guardrails for unsafe output.
- Retries belong only on transient failures (429, 5xx); retrying a deterministic validation or auth error just burns tokens — fix the cause instead.
- Grounding plus an abstain instruction is the primary defense against confident hallucination, and monitoring closes the loop by feeding real failures back into evals.
Model Selection Is a Trade-off, Not a Maximization
The CCA-F frames model choice as a cost-latency-quality triangle, never as "use the biggest model." The Claude family spans tiers: fast, inexpensive models for high-volume simple work, mid-tier models for balanced general use, and the most capable models for the hardest reasoning and long-horizon agentic tasks. The architect's job is to match the tier to the task.
| Task profile | Favor | Reasoning |
|---|---|---|
| High-volume classification / routing | A fast, cheap model | Simple decision, cost dominates at scale |
| General assistant, moderate reasoning | A balanced mid-tier model | Good quality without top-tier cost |
| Complex multi-step reasoning, hard agentic work | The most capable model | Quality and reliability justify the cost |
| A wrong answer is catastrophic (regulated, medical) | The more capable model + validation | Cost-of-error raises the quality bar |
A recurring efficient pattern is routing by difficulty: a cheap model handles the common easy turns and escalates the rare hard turn to a stronger model. This is the routing workflow pattern (Chapter 2.1) applied to model selection. The exam penalizes answers that pay for top-tier capability on a task a cheap model handles, and answers that under-power a high-stakes task to save money.
A worked example makes the trade-off concrete. Suppose 90% of 100,000 daily turns are simple and 10% are hard. Routing the 90,000 simple turns to a model at roughly $1 input / $5 output per million tokens, and the 10,000 hard turns to one at $5 / $25, costs far less than sending all 100,000 to the expensive tier — while preserving quality where it matters, because only the hard turns ever touch the stronger model. The architect quantifies this rather than guessing; the exam expects you to recognize that a difficulty-routed mix beats a single-tier choice when the input distribution is skewed toward easy cases.
The Reliability Control Set
Reliability is a set of deliberate controls, each matched to a failure mode. Know what each one is for:
| Control | Use it for | Failure mode it addresses |
|---|---|---|
| Retry with backoff | Transient errors (429, 5xx) | A single dropped request fails the task |
| Timeout | Bound latency | A hung call stalls the user |
| Fallback | Degrade gracefully | Total outage with no Plan B |
| Guardrail / classifier | Reduce unsafe output | Harmful or off-policy responses |
| Eval / regression set | Measure expected behavior | Shipping a regression blind |
| Monitoring | Observe production outcomes | Silent quality decay |
The most-tested nuance: retries belong only on transient failures. Retrying a deterministic error — malformed schema, bad credentials, a 400 invalid-request — just burns tokens and time, because it will fail identically every time. Fix the cause instead. Use exponential backoff so a burst of retries does not amplify an overload, and bound the retry count so the loop terminates.
Grounding and Hallucination Control
Many reliability questions describe a system that confidently invents facts. The fix is rarely "a better prompt" alone; it is grounding. Retrieve authoritative context, instruct Claude to answer only from the supplied material, and tell it to say "I don't know" or return a sentinel value when the context lacks the answer. Asking for citations to the retrieved passages both improves faithfulness and gives your monitoring a way to detect unsupported claims. Pair this with an eval that plants a question whose answer is not in the retrieved set, and verify the system abstains rather than fabricates.
Guardrails are a complementary layer: a classifier or rule check on the input or output can block unsafe, off-policy, or out-of-scope responses before they reach the user. The exam favors the layered answer — ground to prevent fabrication, guardrail to block unsafe output, and validate to enforce shape — over any single silver bullet.
A practical distinction the exam draws: grounding addresses faithfulness (is the answer supported by the sources?), while guardrails address safety and policy (is the answer allowed at all?). They are not interchangeable. A perfectly grounded answer can still be off-policy, and a safe answer can still be a fabrication. Reach for grounding when the symptom is invented facts, and for a guardrail when the symptom is harmful or out-of-scope output, and apply both when the scenario shows both failures.
Monitoring Closes the Loop
Production reliability is a loop, not a one-time setup. Log inputs, outputs, tool calls, latencies, and error rates; sample real traffic into your eval set so the suite reflects how users actually behave; and alert when a metric drifts beyond a threshold — refusal rate, schema-validation-failure rate, citation-missing rate, p95 latency, or cost per request. The exam favors answers that close the loop, capturing real failures and feeding them back into evals, over answers that simply add another retry.
Reliability Decision Table
| Symptom in the scenario | First control to reach for |
|---|---|
| Confident wrong facts | Retrieval grounding + abstain instruction |
| Intermittent 429 / 5xx errors | Retry with exponential backoff |
| Occasional hung requests | Timeout + fallback response |
| Quality slipped after an edit | Regression eval set |
| Paying top-tier cost for trivial turns | Route easy turns to a cheaper model, escalate hard ones |
| Unsafe or off-policy outputs | Guardrail / classifier check |
| A deterministic 400 keeps recurring | Fix the cause — do not retry a non-transient error |
When two options both seem reasonable, choose the one whose control matches the named failure mode in the stem. Matching symptom to control — and matching the model tier to the task's cost and risk — rather than stacking every control at once, is the behavior the CCA-F is designed to measure.
A high-volume system classifies 50,000 short support messages a day into three categories, a task a small fast model handles accurately. Following CCA-F principles, which model choice is best?
An application keeps automatically retrying every failed Claude request, including ones that fail with a 400 invalid-request error from a malformed schema. Why is this a problem?
You've completed this section
Continue exploring other exams