4.3 Optimizing Gemini Applications for Cost, Latency & Availability
Key Takeaways
- Agent Platform offers five consumption options for generative models: Provisioned Throughput, Standard PayGo, Priority PayGo, Flex PayGo, and batch inference.
- Provisioned Throughput plans come in 1-week, 1-month, 3-month, and 1-year terms, and a 1-year commitment costs 26% less than a 1-month commitment.
- Flex PayGo and batch inference each cost about 50% less than Standard PayGo, in exchange for slower or asynchronous responses.
- Implicit context caching is on by default and gives a 90% discount on cached input tokens, and explicit caches default to a 60-minute TTL.
- Google recommends exponential backoff for 429 errors and the global endpoint to reduce throttling from regional capacity limits.
The exam guide lists optimizing Gemini-based applications for cost, latency, and availability as its own consideration. Most questions describe a workload pattern (interactive chat, nightly bulk processing, a launch-day spike) and ask which consumption option or optimization fits.
The Five Consumption Options
| Option | What you get | Ideal for | Pricing |
|---|---|---|---|
| Provisioned Throughput (PT) | Guaranteed throughput for a commitment period | Critical, steady, always-on workloads needing an SLA | Commitment: 1 week, 1 month, 3 months, or 1 year |
| Standard PayGo | Flexible pay-per-use, no commitment | Default for everyday and variable traffic | Standard per-token rate |
| Priority PayGo | Higher reliability through priority processing, still pay per use | Important workloads needing more reliability than Standard | Premium per-token rate |
| Flex PayGo | Lower priority, longer latency, more throttling | Latency-tolerant tasks: offline analysis, annotation, evaluation, catalog building | About 50% off Standard |
| Batch inference | Asynchronous processing of large request files | Large backlogs where results can wait | About 50% off Standard |
Details worth remembering:
- Traffic above your PT baseline (spillover) is handled by Standard PayGo by default. With Priority PayGo, spillover performance is more reliable at a premium.
- A 1-year PT commitment costs 26% less than 1-month PT, and you can switch the model tied to your PT units to newer models.
- Flex PayGo (preview at the time of writing) is used through the global endpoint only. You select it with a request header, and request timeouts can be as long as 30 minutes.
- For very large jobs that can wait up to a day, Google recommends batch inference over Flex for throughput and success rate. Batch input can come from Cloud Storage (JSONL) or BigQuery.
- A 429 on PayGo reads "Resource exhausted, please try again later." Under PT it reads "Too many requests. Exceeded the Provisioned Throughput."
Combining Options
Google's recommended pattern for latency-sensitive production traffic:
- Analyze traffic at minute or second granularity.
- Cover the baseline (your highest-priority, steady traffic) with Provisioned Throughput. Don't buy PT for peak, because unused reserved capacity wastes money.
- Let spillover go to Standard PayGo, or Priority PayGo if spillover needs steadier performance.
- Move latency-tolerant background work to Flex PayGo, and bulk backlogs to batch inference.
Latency Levers
Latency has two measures. Time to first token (TTFT) matters for streaming chat. Time to last token (TTLT) is the total generation time.
| Lever | Effect |
|---|---|
| Choose a smaller tier (Flash or Flash-Lite instead of Pro) | Faster generation and lower cost, if quality holds on your eval set |
| Reduce prompt size | Lower TTFT |
Limit output: concise instructions and max_output_tokens | Lower TTLT, since latency scales with tokens generated. Too low a limit cuts answers off |
| Lower the thinking level or budget | Fewer internal reasoning tokens. Keep enough for hard tasks |
| Stream responses | Users see text sooner, which improves perceived latency |
| Provisioned Throughput | Most consistent performance, avoiding queueing during high traffic |
Gemini 3 models use a thinking_level setting (such as MINIMAL, LOW, MEDIUM, HIGH, depending on the model), while Gemini 2.5 models use a numeric thinking budget. For all Gemini 3 models, Google deprecated the temperature, top_p, and top_k sampling parameters and recommends removing them from requests.
Availability Levers
- Retry with exponential backoff on 429 errors, especially on Standard PayGo.
- Hybrid capacity: don't rely only on PayGo for critical apps. PT plus PayGo gives the best protection against resource exhaustion.
- Global endpoint: draws on Google's global capacity pool to reduce throttling from regional limits. Don't use it when data must be processed in a specific region, because you can't control where global-endpoint requests run.
- Monitor PT usage with alert policies, and add PT units before known events such as product launches.
- Smooth traffic and move deferrable work off-peak, since aggregate model demand follows a daily cycle.
Cost Levers
| Lever | How it saves |
|---|---|
| Context caching, implicit | On by default. Cached input tokens get a 90% discount. Improve hit rates by putting large, shared content at the start of the prompt and sending similar prefixes close together in time |
| Context caching, explicit | You create a cache and reference it by name. The discount on cached input tokens is guaranteed (90% on Gemini 2.5 and later). The default TTL is 60 minutes, adjustable, and storage is billed by time stored |
| Batch inference / Flex PayGo | About 50% lower per-token price for work that can wait |
| Right-size PT | Cover a percentile of traffic, not the peak |
| Smaller model or tuned model | Flash-Lite pricing, or tuning that replaces long few-shot prompts with shorter ones |
| Token counting | Estimate prompt tokens before sending large jobs, and trim unneeded context |
Measuring Before Optimizing
Tie each optimization to a metric so you can tell whether it worked:
- Latency: track TTFT and TTLT percentiles (p50, p95, p99) by model and route. Averages hide tail latency.
- Availability: track 429 and 5xx rates by consumption option and endpoint.
- Cost: track input, output, and cached tokens per request. The
cachedContentTokenCountfield in response metadata shows how many input tokens came from cache. - Quality: re-run the evaluation set after every model, prompt, or thinking-level change. A cheaper configuration that quietly lowers answer quality isn't a real saving.
Worked Scenario
A legal-tech app lets users chat with a 500-page contract. Every question resends the contract. Users complain about slow first answers, and costs are high.
- Put the contract at the beginning of the prompt and use explicit context caching for the session. This cuts input cost and the processing of repeated content.
- Stream responses and cap
max_output_tokensto improve perceived and actual latency. - Evaluate whether Flash matches Pro quality on a test set of contract questions.
- Run the nightly job that pre-summarizes new contracts as batch inference.
- If usage is steady during business hours, cover that baseline with Provisioned Throughput and let spikes spill to PayGo.
A retailer must classify 80 million product reviews once a quarter. Results are needed within a day, and cost is the top priority. Which consumption option fits best?
A customer-support chatbot sees steady weekday traffic with occasional marketing spikes. Leadership wants predictable performance for the core load without paying for idle capacity. What does Google recommend?
A European insurer's Gemini app gets frequent 429 errors on a regional endpoint. An engineer proposes the global endpoint, but regulation requires all processing to stay in the EU. What is the best response?