10.5 Prompt Caching, Token Optimization & Latency Reduction

10.5 Prompt Caching, Token Optimization & Latency Reduction

Repeated long prompt prefixes consume tokens and delay generation. Amazon Bedrock prompt caching can reduce latency and input-token cost for supported models by reusing eligible prompt context. The current service supports implicit and explicit caching, with behavior that varies by model and API. It is incorrect to teach one universal minimum length, checkpoint count, time to live, or discount.

Implicit and explicit prompt caching

Implicit caching lets Bedrock and the model attempt to reuse eligible repeated prefixes without application checkpoints. It is best effort: an identical prompt does not guarantee a hit. Explicit caching places model-supported cache controls or checkpoints after a stable prefix. The request still succeeds when a checkpoint is below the model's minimum, but the prefix is not cached.

Many models use prefix matching. Put stable tool definitions, system instructions, schemas, and shared reference content before changing user input. Changing an earlier component can invalidate later cached components. Model-specific implementations may tokenize and order fields differently, so reason from the documented request format rather than calling matching universally byte-for-byte.

Cache writes can cost more than ordinary input while reads cost less. A prefix reused only once may not save money. Estimate break-even from the selected model's prices, prefix size, expected reuse, and expiry window. Monitor response usage fields to distinguish hoped-for hits from actual hits.

Prompt caching is available for supported on-demand inference. Current Bedrock documentation says it is not supported with batch inference. Cross-Region inference can be combined with caching for supported models, but routing among Regions can affect cache-write behavior.

Context reduction before caching

Caching avoids reprocessing repeated text; it does not make irrelevant text useful. Reduce context first:

1. Retrieve fewer, better passages with metadata filters and reranking.
2. Remove duplicate chunks and boilerplate.
3. Summarize old conversation turns while preserving decisions and unresolved constraints.
4. Put tool results into compact structured fields.
5. Limit output with a clear task and appropriate maximum tokens.
6. Use a smaller model when evaluation proves it meets the requirement.

For RAG, track retrieval quality as context is pruned. Cutting top-k can improve focus and cost but reduce recall. A golden dataset should determine the safe threshold. For conversations, keep authoritative state in application storage instead of expecting an unlimited transcript to function as a database.

Application-level caching

Exact response caching works only when the request and authorization context are equivalent. Include model version, prompt version, Guardrail version, retrieval corpus version, tenant, locale, and relevant permissions in the cache key. Do not serve one tenant's generated content to another because the natural-language query matched.

Semantic caching finds similar requests by embedding distance. It can save more work but risks returning an answer for a request whose details differ. Set conservative thresholds, exclude high-risk or time-sensitive tasks, attach freshness rules, and validate authorization before lookup and delivery. Deterministic fingerprinting is easier to audit than semantic matching.

Cache invalidation is a correctness feature. Expire results when source documents, price data, access policy, prompt, Guardrail, or model changes. Record cache-hit quality and stale-answer incidents, not only hit rate.

Streaming and perceived latency

ConverseStream and InvokeModelWithResponseStream return output incrementally for supported models. Streaming improves time to first visible output but does not necessarily reduce total generation time or cost. The client must parse the event stream, handle midstream exceptions, support cancellation, and decide whether partial output may be shown before final safety or schema validation.

For structured JSON, partial rendering can expose invalid fragments. Buffer until validation when correctness matters more than progressive display. For ordinary chat, render safe chunks while keeping an abort control. Backpressure prevents a slow client from accumulating unbounded server memory.

Measure time to first token, time to last token, output tokens per second, cancellation rate, and end-to-end task completion. Provisioned capacity can improve consistency for supported workloads, but it does not convert a synchronous client into a streaming one.

The exam distinction is straightforward: prompt caching reuses supported repeated prefixes, application caching reuses completed results under application-defined equivalence, context optimization removes unnecessary tokens, and streaming changes delivery. Each has different correctness and security risks.