5.4 Latency & Throughput Optimization

Latency & Throughput Optimization

Exam Blueprint Focus: Latency optimization is a primary domain on the CCDV-F exam. Developers must master the exact mathematical and architectural mechanics of request latency, distinguish between the prefill and decode phases, properly configure prompt caching breakpoints, utilize Server-Sent Events (SSE) streaming for perception optimization, and avoid common misconceptions surrounding max_tokens.

Deconstructing Latency: The Four Foundational Metrics

To optimize Claude-powered systems effectively, engineers must decompose the end-to-end HTTP request lifecycle into four distinct, measurable telemetry metrics:

+-----------------------------------------------------------------------------------+
|                             REQUEST LATENCY TIMELINE                              |
+-----------------------------------------------------------------------------------+
| Client Dispatches |        Server Prefill Phase        | Server Autoregressive Decode   |
| HTTP Request      | (Prompt Ingestion & Cache Check)   | (Token-by-Token Generation)    |
+-------------------+------------------------------------+--------------------------------|
| t = 0             | <---------- TTFT ----------------> |                                |
|                   |                                    | <--- ITL ---> <--- ITL --->    |
|                   |                                    | Token 1      Token 2  Token N  |
|                   | <--------------------- Total Latency -----------------------------> |
+-----------------------------------------------------------------------------------+

1. Time-to-First-Token (TTFT)

Time-to-First-Token (TTFT) measures the elapsed time from when the client transmits the HTTP request payload to when the client receives the very first token emitted by Claude. TTFT represents the duration of the Prefill Phase, where the server ingests, tokenizes, and computes internal Key-Value (KV) attention matrices across the prompt, system instructions, and historical messages.

2. Inter-Token Latency (ITL)

Inter-Token Latency (ITL) is the time interval required for the model to generate each subsequent token during the Autoregressive Decode Phase. In modern transformer inference engines, tokens are generated sequentially one by one, where each token requires a forward pass over the model's parameters and KV cache.

3. Tokens Per Second (TPS)

Tokens Per Second (TPS) measures the throughput of the generation engine, representing the mathematical inverse of ITL:

$\text{TPS} = \frac{1}{\text{ITL}}$

While TTFT is heavily influenced by prompt size, TPS is largely independent of prompt size and is determined by model parameter size, GPU memory bandwidth, and server-side speculative decoding optimizations.

4. Total Generation Latency

Total request duration is governed by the fundamental formula:

$T_{\text{total}} = \text{TTFT} + (N_{\text{actual\_tokens}} \times \text{ITL})$

Where $N_{\text{actual_tokens}}$ is the number of tokens actually emitted before hitting a stop condition.

---

Anatomy of the Prefill Phase: What Governs TTFT?

TTFT is determined entirely prior to the emission of token 1. The primary architectural factors influencing TTFT include:

1. Prompt Ingestion Volume: A prompt containing 50,000 tokens of retrieved documents requires significantly more prefill compute than a prompt containing 500 tokens. The prefill phase exhibits $O(N)$ to $O(N^2)$ attention computational complexity.
2. Prompt Caching Hits vs. Misses: This is the single most powerful lever for reducing TTFT. When a prompt hits an active Anthropic KV cache breakpoint, the server completely bypasses prompt re-computation, reducing TTFT by up to 80% to 90% (e.g., dropping TTFT from 3,500ms down to 350ms on large contexts).
3. Multimodal Encoding Overhead: Passing images or PDF documents introduces visual and document parsing overhead. High-resolution images are split into multiple tiles (each costing ~1,600 tokens), which must pass through vision encoder layers before text attention begins.
4. Server Concurrency & Queueing: High traffic volume on public API endpoints can introduce queuing delays before GPU compute allocation occurs.

---

Anatomy of the Autoregressive Decode Phase: What Governs Generation Latency?

Once the first token is emitted, the generation phase dominates total request time. Understanding the physics of autoregressive decoding resolves several critical misconceptions:

The max_tokens Latency Myth

Critical Exam Concept: Setting max_tokens to a large value (e.g., max_tokens: 4096) does NOT cause Claude to take longer to respond if the answer is brief. max_tokens specifies an absolute upper ceiling, not a target allocation.

If Claude formulates a complete answer in 85 tokens, it immediately emits the end_turn stop token, terminates the decode loop, and returns the response. The generation phase lasts only $85 \times \text{ITL}$. Setting max_tokens: 100 vs max_tokens: 4096 produces identical latency if the output is 85 tokens.

However, setting an appropriately constrained max_tokens limit is a crucial architectural safeguard: it prevents runaway generation loops if a prompt inadvertently triggers recursive or overly verbose outputs.

Extended Thinking Reasoning Budgets

When using Claude Sonnet 5 with extended thinking enabled, Claude generates internal "thinking tokens" before generating visible response tokens. Every thinking token incurs the standard ITL decode cost. For example, if Claude spends 3,000 thinking tokens followed by 500 response tokens, the total decode phase spans 3,500 tokens. While extended thinking unlocks extraordinary reasoning accuracy, it expands generation latency proportionally.

Model Parameter Footprint

Smaller models have significantly smaller parameter matrices to load from GPU memory per token:

• Claude Haiku 4.5: ~100-140 TPS (ITL ~7-10ms)
• Claude Sonnet 5: ~50-80 TPS (ITL ~12-20ms)
• Claude Opus 5: ~25-40 TPS (ITL ~25-40ms)

---

Architectural Levers for Latency Optimization

To achieve production-grade performance, software developers must apply targeted architectural levers across both prefill and decode phases:

+---------------------------------------------------------------------------------------+
|                           LATENCY OPTIMIZATION LEVERS MATRIX                          |
+---------------------------------------------------------------------------------------+
| Architectural Lever       | Target Phase     | Latency Impact     | Primary Trade-off |
|---------------------------+------------------+--------------------+-------------------|
| SSE Streaming             | Perceived        | 70-90% reduction   | Client state      |
| (stream: true)            | Latency          | in user wait       | complexity        |
|---------------------------+------------------+--------------------+-------------------|
| Prompt Caching            | Prefill (TTFT)   | 80-90% reduction   | Requires static   |
| Breakpoints               |                  | on cache hits      | prompt prefix     |
|---------------------------+------------------+--------------------+-------------------|
| Context Pruning           | Prefill (TTFT)   | Linear reduction   | Risk of omitting  |
| (Trimming RAG baggage)    |                  | with token count   | marginal context  |
|---------------------------+------------------+--------------------+-------------------|
| Parallel Tool Calling     | Agent Workflow   | Multi-turn latency | Higher burst      |
| (Concurrent I/O)          | Round-trips      | collapsed into one | API concurrency   |
|---------------------------+------------------+--------------------+-------------------|
| Assistant Prefilling      | Decode Phase     | Saves 50-200ms of  | Requires format   |
| (Skip preamble)           |                  | filler generation  | pre-commitment    |
|---------------------------+------------------+--------------------+-------------------|
| Model Tier Downsizing     | Decode (TPS) &   | 2x-3x speedup      | Lower reasoning   |
| (Sonnet -> Haiku)         | Prefill (TTFT)   | across all phases  | depth ceiling     |
+---------------------------------------------------------------------------------------+

1. Server-Sent Events (SSE) Streaming for Perceived Latency

In synchronous non-streaming requests, the client HTTP connection blocks until the entire generation completes. If Claude generates a 600-token response at 60 TPS, the user stares at a blank screen for 10 seconds ($TTFT + 10s$).

By setting stream: true, the API immediately pushes tokens to the client over an SSE connection as they are emitted. The user's perceived latency drops from 10 seconds down to the TTFT (~600ms), creating an instantaneous, highly responsive user experience.

// Implementing Streaming in TypeScript for Minimal Perceived Latency
import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic();

async function streamResponse(userPrompt: string) {
  const stream = await client.messages.stream({
    model: 'claude-sonnet-5',
    max_tokens: 1024,
    messages: [{ role: 'user', content: userPrompt }],
  });

  // Stream events yield tokens immediately as they are decoded
  stream.on('text', (textDelta) => {
    process.stdout.write(textDelta);
  });

  const finalMessage = await stream.finalMessage();
  console.log(`\nGeneration complete. Total output tokens: ${finalMessage.usage.output_tokens}`);
}

2. Prompt Caching Breakpoints for Long-Context Applications

For workflows involving recurring documents, system prompts, or tool schemas, adding cache_control: {"type": "ephemeral"} eliminates redundant prefill compute. When querying a 40,000-token knowledge base across multiple turns:

• Turn 1 (Cache Write): TTFT is ~2,500ms (normal prefill).
• Turn 2+ (Cache Read): TTFT drops to ~300ms (an 88% reduction), while cutting input token costs by 90%.

3. Context Window Pruning & System Prompt Streamlining

Bloated prompts degrade both TTFT and accuracy. Best practices include:

• Stripping Redundant RAG Chunks: Rerank retrieved context chunks and discard low-scoring passages. Passing three high-relevance 400-token chunks is vastly faster and less error-prone than dumping twenty raw chunks.
• Eliminating System Prompt Verbosity: Avoid repetitive negative constraints (Do not say hello, never introduce yourself, avoid pleasantries...). Replace with concise declarative directives or assistant prefilling.

4. Concurrent Tool Execution (Parallel Tool Calling)

When an autonomous agent must query multiple external APIs (e.g., checking weather, stock price, and flight status), prompting the model to emit all independent tool calls in a single turn allows the client to execute them concurrently using Promise.all() or asyncio.gather().

• Sequential Execution: 3 tool calls across 3 turns = $3 \times (\text{TTFT} + \text{Decode} + \text{Tool latency}) \approx 9.0\text{s}$.
• Parallel Execution: 3 tool calls in 1 turn = $1 \times (\text{TTFT} + \text{Decode}) + \max(\text{Tool latencies}) \approx 2.5\text{s}$.

5. Assistant Prefilling to Eliminate Preamble Overhead

By prefilling the assistant turn with "{" for JSON responses or <result> for XML, Claude skips generating conversational boilerplate like "Certainly! Here is the information you requested:". This saves 15 to 30 output tokens per request, reducing generation latency by 200ms to 500ms per interaction.

---

Common Traps & Antipatterns

• The TTFT Reduction Illusion: Believing that reducing max_tokens will speed up how quickly the first token appears. max_tokens has zero influence on the prefill phase or TTFT.
• Buffering Streams on the Server: Enabling stream: true on the Anthropic API call, but buffering all SSE events inside your backend API gateway before returning the complete payload to the frontend. This introduces client-side implementation complexity without delivering any latency benefits to the user.
• Excessive Cache Breakpoints: Placing cache breakpoints on frequently mutating user messages rather than stable prefix context. Cache misses negate TTFT gains and incur cache write pricing overhead.