1.9 Optimizing Gemini Applications for Cost, Latency and Availability

1.9 Optimizing Gemini Applications for Cost, Latency and Availability

Blueprint reference: Section 1.2, "Optimizing Gemini-based applications for cost, latency, and availability."

This bullet is where a lot of otherwise-strong candidates lose points, because the levers are specific and mutually exclusive in ways that are easy to blur. A scenario will state exactly one binding constraint — a monthly budget, a p95 latency target, or a throughput guarantee — and exactly one lever is right for it.

The Cost Levers, in Order of Impact

1. Model tier. The largest single cost decision. Generative model families ship in tiers, and the price gap between a fast tier and a frontier tier is typically severalfold. Most production workloads are over-modelled: a classification or extraction task that a fast-tier model handles at 95% accuracy does not need a frontier model at eight times the price. Measure on your eval set before assuming you need the top tier.

2. Context caching. When many calls share a large, unchanging prefix — a long system instruction, a policy manual, a codebase, a video — context caching stores that prefix so it is not reprocessed and re-billed on every request. Cached input tokens are billed at a 90% discount to the standard input rate on Gemini 2.5 and later models (75% on Gemini 2.0). Creating the cache costs one ordinary input-token charge, and the cache carries an hourly storage charge for as long as its TTL keeps it alive — which is why caching pays off on a large prefix reused often, and loses money on a small one reused rarely.

The decision rule: caching pays off when the shared prefix is large and reused frequently within the cache's lifetime. A 40-token system prompt is not worth caching. A 200-page policy document queried hundreds of times an hour is the canonical win.

3. Batch mode. Generative batch prediction submits many requests as one job, returns results to Cloud Storage or BigQuery when complete, and is priced well below interactive inference. If nobody is waiting — nightly summarization, bulk enrichment, backfilling classifications over a warehouse table — batch is almost always the correct answer, and choosing interactive inference for an offline job is a recognisable wrong answer.

4. Output control. Output tokens usually cost more than input tokens. Capping max_output_tokens, asking for structured output instead of prose, and using a constrained response schema all reduce spend directly. AI.COUNT_TOKENS (or the equivalent token-counting call) lets you estimate a job's cost on a sample before running it over the full table.

Real Latency Versus Perceived Latency

These are different problems with different fixes.

Streaming is the highest-leverage fix for interactive chat. Total generation time is unchanged, but the user sees the first tokens in a few hundred milliseconds instead of staring at a spinner for several seconds. If a scenario complains that "users perceive the assistant as slow" while total latency is acceptable, streaming is the answer and switching models is over-correction.

Availability: Provisioned Throughput Versus Shared Quota

Pay-as-you-go generative capacity is served from dynamic shared quota — a shared pool with no reserved allocation. It is cost-efficient and adequate for most workloads, but under contention a request can be throttled with a 429.

Provisioned Throughput reserves dedicated capacity for a committed term. You pay for the reservation whether or not you use it, and in return you get predictable, guaranteed throughput.

Choose Provisioned Throughput when the scenario mentions a guaranteed or predictable throughput requirement, a customer-facing SLA, a launch event with a known traffic profile, or intolerance of throttling. Choose pay-as-you-go when traffic is spiky, low-volume, or experimental — reserving capacity for a bursty prototype wastes the reservation between bursts.

A common production pattern combines both: Provisioned Throughput sized to the steady-state floor, with overflow spilling to pay-as-you-go.

Retry and Fallback Patterns

Any generative client will eventually see 429 (throttled) and 503 (unavailable). The expected engineering response:

# Exponential backoff with jitter — the jitter matters, because synchronized
# retries from many clients recreate the same contention that caused the 429.
delay = min(BASE * (2 ** attempt), MAX_DELAY)
sleep(delay * random.uniform(0.5, 1.5))

• Retry only retryable codes. A 400 from a malformed request will fail identically forever; retrying it burns quota.
• Cap total attempts and surface a clean error rather than retrying indefinitely behind a user-facing request.
• Define a fallback. Route to a smaller model, a cached response, or a deterministic non-AI path. A feature that degrades is better than a page that fails.
• Idempotency. For expensive generations, key each request so a retry does not pay twice.

Putting It Together

Exam Traps

• Streaming to reduce cost. It does not; it reduces perceived latency only.
• Provisioned Throughput for a bursty prototype. You pay for idle reservation.
• Interactive inference for an offline job. Batch is cheaper and nobody is waiting.
• Retrying 400-class errors. Only 429 and 5xx are retryable.
• Caching a tiny prompt. The storage charge can exceed the savings.