4.4 The Claude Application Life Cycle & Configuration Management
Key Takeaways
- LLM features degrade statistically rather than deterministically, the prompt is production code written in prose, and the model itself can be retired - which is why the life cycle differs from ordinary software delivery.
- The six stages are prototype, evaluate, integrate, harden, deploy, and operate, and the golden eval set is built before application code rather than after.
- Model ID, effort level, max_tokens, turn ceilings, and routing flags must be externalized to configuration so canary rollout and rollback need no code deploy.
- System prompts are versioned artefacts: store them in version control, log the version identifier with every response, and gate changes on the eval suite so regressions are attributable.
- Dev, staging, and production must share prompt versions and tool schemas while deliberately differing on model tier, effort, rate limits, and logging verbosity.
The Claude Application Life Cycle: Prototype to Production
Exam Blueprint Focus: Systems Life Cycle (2.8%) sits inside Applications and Integration, and Configuration Management (4.1%) is graded alongside it. Together they test whether you can describe how a Claude feature moves from a notebook to production and stays maintainable once several people own it.
Why the LLM Life Cycle Differs from Ordinary Software
Traditional software fails deterministically: a bug either reproduces or it does not. An LLM feature degrades statistically, and three things move underneath you that have no analogue in ordinary deployment:
- The prompt is production code, but it is prose, so it slips past code review habits.
- The model can be retired, forcing a migration you did not schedule.
- Quality is a distribution, not a boolean. "Works on my three examples" is not a signal.
Everything in this section follows from those three facts.
The Six Stages
1. Prototype
Work in a notebook or the Console. The goal is a single question: is this task tractable with a prompt at all? Use a capable tier here even if you plan to ship on a cheaper one — you are testing feasibility, not economics. Collect the failure cases you stumble into; they are the seed of your eval set.
Exit criterion: a prompt that works on a handful of realistic inputs, and a written list of the ways it failed.
2. Evaluate
Turn those failures into a golden dataset before writing application code. Twenty well-chosen cases covering the real distribution — including edge cases and inputs that should be refused — beat a thousand generated ones. Record the model ID with every score, because a score without a model ID is not a baseline.
Exit criterion: a runnable eval with a pass rate you would defend to a stakeholder.
3. Integrate
Now write the application. This is where the design decisions from the previous section land: statelessness handled by your database, the output contract enforced by a tool schema, retries and backoff around the API call, and the prompt structured for cache stability from the start.
Exit criterion: the feature works end to end behind a flag, with structured logging.
4. Harden
Add what production requires and prototypes never have: rate limiting and quota enforcement, timeout and turn ceilings, graceful degradation when the API returns 429 or 529, redaction before logging, and prompt-injection defences on any path that ingests untrusted content.
Exit criterion: the failure modes have owners and runbooks.
5. Deploy
Roll out progressively. Shadow first if the change is risky, then canary at 1% → 10% → 100% with automated rollback thresholds defined before the rollout starts.
Exit criterion: the feature is at 100% and the rollback path is still one config change away.
6. Operate
Watch the signals that matter for LLM systems specifically — cache hit ratio, stop_reason distribution, tool error rate, cost per transaction, p95 latency — and re-run the eval suite on a schedule, not only on prompt changes. Track the retirement date of every model ID you depend on.
Configuration Management for LLM Systems
Everything that can change without a code change belongs in configuration, and everything in configuration needs the same discipline as code.
What must be externalized
| Value | Why it cannot be a literal |
|---|---|
| Model ID | Canary, rollback, and migration all require changing it without a deploy |
effort level | The primary cost/quality dial; needs tuning per environment |
max_tokens, turn and timeout ceilings | Environment-specific safety limits |
| Feature flags for routing and cascading | Enable and disable tiers independently |
| API keys and endpoints | Never in source; see the secrets section |
Prompts are versioned artefacts
Treat the system prompt like a schema migration. Store it in version control, give it an explicit version identifier, log which version produced each response, and gate changes on the eval suite. When an incident report says "the assistant started refusing valid requests on Tuesday," the first question is which prompt version shipped Tuesday — and you can only answer that if the version was recorded with each response.
PROMPT_VERSION = "claims-extract-v7"
SYSTEM_PROMPT = load_prompt(PROMPT_VERSION) # from version control, not a literal
resp = client.messages.create(
model=settings.CLAUDE_MODEL, # externalized
max_tokens=settings.MAX_TOKENS,
output_config={"effort": settings.EFFORT},
system=[{"type": "text", "text": SYSTEM_PROMPT,
"cache_control": {"type": "ephemeral"}}],
messages=messages,
)
log.info("claude_call", prompt_version=PROMPT_VERSION,
model=settings.CLAUDE_MODEL, usage=resp.usage)
Three things that log line buys you: attribution of any regression to a prompt version, cost per version, and a cache-hit signal from usage that tells you immediately if a prompt edit broke prefix stability.
Environment parity, with deliberate exceptions
Dev, staging, and production should share prompt versions and tool schemas so that what you test is what you ship. Deliberately differ on model tier (cheaper in dev), effort (lower in dev), rate limits, and logging verbosity. Never differ on the prompt itself: a staging prompt that drifts from production makes staging worthless.
Claude Code configuration is the same discipline
.claude/settings.json and CLAUDE.md are committed configuration governed by the same rules: shared settings in version control, personal overrides in gitignored .claude/settings.local.json, and secrets in neither.
CI/CD for Prompt Changes
A prompt change should be gated exactly like a code change:
PR opened
-> lint the request shape (schema validation on tool definitions)
-> run the golden eval on the pinned model
-> compare against the stored baseline for that model ID
-> block the merge if the pass rate regresses beyond tolerance
-> report cost-per-case delta in the PR
The cost delta is the part teams forget. A prompt change that adds 400 tokens to the system prefix is a permanent cost increase on every request, and it is far cheaper to notice in review than in the monthly invoice.
Common Traps
- Editing the prompt directly in production. Without a version identifier in the logs, the next regression is unattributable.
- Running evals only when the prompt changes. The model retirement calendar and your own data distribution both move on their own schedule.
- Hardcoding the model ID. It makes canary and rollback into code deploys, which is exactly what you do not want during an incident.
- Letting staging prompts drift from production. Testing a prompt you will not ship tests nothing.
- Treating the eval baseline as model-independent. A baseline without a model ID cannot tell you whether a change or a migration caused a shift.
An incident report reads: "On Tuesday the assistant began refusing valid requests." The team cannot determine whether a prompt edit, a config change, or a model migration caused it. Which single practice would most directly have made this diagnosable?
A team hardcodes model="claude-sonnet-5" across twelve request handlers. During a production incident they need to fall back to a previously validated model immediately. What is the consequence, and what should the design have been?
A team's CI runs the golden eval on every prompt change and blocks merges on quality regression. Six months in, monthly API spend has tripled while the eval pass rate has stayed flat. What is the most likely cause and the correct fix?