10.2 Capacity Measurement, Scaling Patterns & Resilience

10.2 Capacity Measurement, Scaling Patterns & Resilience

Measure the workload before purchasing capacity

Collect input tokens, requested and actual output tokens, request rate, concurrency, latency percentiles, streaming duration, retry volume, and traffic seasonality. Separate interactive traffic from offline processing. Measure the full application path because retrieval, Guardrails, tools, and network hops contribute to user latency even when model inference is stable.

Use controlled load tests with representative prompt sizes. Ramp traffic gradually and watch successful calls, throttles, transient capacity errors, queue depth, and latency. A test made only of tiny prompts can conceal the token pressure of production RAG requests.

If a hypothetical model unit supports 300,000 input tokens per minute and 60,000 output tokens per minute, and the measured peak requires 600,000 input tokens and 150,000 output tokens per minute, calculate both constraints:

• Input requirement: ceiling of 600,000 divided by 300,000 equals 2 units.
• Output requirement: ceiling of 150,000 divided by 60,000 equals 3 units.
• Required capacity is the larger result, so the answer is 3 units.

This arithmetic is useful only when the request rate and unit specification are valid. Concurrency multiplied by 60 divided by latency is an estimate of completed requests per minute under a steady pipeline, not a universal capacity formula. Validate it with a load test and headroom for variance.

Scaling patterns

Do not claim that Amazon Bedrock Provisioned Throughput is a scalable target in Application Auto Scaling unless current service documentation explicitly supports that integration. The Bedrock control plane can create, inspect, modify, and delete supported throughput resources, but provisioning is not an instant per-request autoscaling mechanism. Capacity changes require quotas, lead time, supported settings, and deployment controls.

For predictable cycles, schedule an operational review and modify capacity ahead of the event when supported. For unexpected bursts, bound client concurrency, queue noninteractive work, use cross-Region inference when policy allows, apply backoff with jitter, and degrade gracefully. A fallback model is acceptable only if it still meets the task's safety and quality contract.

Route offline summarization, embedding, or bulk generation to batch processing when supported. This keeps interactive paths focused on user-facing latency and uses the separate batch workflow. Semantic or prompt caching can avoid duplicate computation, but cache hits are workload- and model-dependent.

Cost and reliability decision

Estimate monthly cost under realistic utilization, including cache reads and writes, retries, evaluation traffic, vector search, logging, and idle provisioned hours. Compare cost per successful business task. A fixed allocation can be economical for sustained traffic but wasteful for a nightly ten-minute job. On-demand can be economical for irregular traffic but needs quota and capacity handling.

Define service objectives and failure policies: maximum queue delay, acceptable fallback, timeout, retry budget, and what happens when all model paths are unavailable. Monitor the exact endpoint and target used. Current AWS guidance distinguishes bedrock-runtime and bedrock-mantle quota accounting; some models expose requests-per-minute quotas, while token accounting and burndown behavior vary. Check the quotas for the exact model rather than teaching a universal RPM-plus-TPM rule.

The professional-level answer combines measurement, current support checks, economics, and resilience. Provisioned Throughput is not synonymous with unlimited or error-free capacity, and custom model is no longer synonymous with mandatory Provisioned Throughput.

Load-test design and headroom

Replay representative short, median, and long prompts with realistic output limits and concurrency. Increase traffic in controlled steps, hold each level long enough to observe queues and tail latency, and capture request IDs for failures. Test the steady baseline, expected peak, sudden burst, dependency slowdown, and Region or model failure path.

Headroom covers variance and recovery, but it should be explicit. Document the maximum accepted concurrency, queue depth, queue-age objective, retry budget, and threshold for degrading to retrieval-only or delayed processing. If a fallback model produces a different schema or weaker safety behavior, it is not compatible until the application adapts and evaluates it.

Capacity changes are deployments. Verify quotas and supported unit changes, schedule them before known events, observe the new allocation, and maintain rollback. Do not make a long commitment automatically in reaction to a brief alarm. Cost and reliability approvals should reflect the commitment term.

Recovery objective

State how long the service may be degraded and how much queued work may be lost or replayed. These objectives determine whether an explicit failure, a delayed batch result, or a compatible fallback is acceptable. Test recovery, not only failover.