14.2 Latency, Throughput, Token Use, and Cost Metrics
Key Takeaways
- End-to-end latency follows TTLT = TTFT + (TBT × tokens generated). Always pair a latency metric with a token-count metric before calling a regression.
- For streaming, use Time to Response (AzureOpenAITimeToResponse) as TTFT. For overall duration, use Time to Last Byte (AzureOpenAITTLTInMS). Do not use the legacy Cognitive Services Latency metric for Azure OpenAI.
- Throughput is observed RPM and TPM. Standard quota is admission control, not guaranteed capacity. Agent dashboard hints: latency above about 10 seconds, run success rate below about 95 percent.
- Pay-as-you-go bills input and output tokens; output tokens usually cost more and dominate TTLT. Cache hits appear as cached_tokens under prompt_tokens_details. Cached tokens do not consume PTU capacity.
- PTU billing is hourly for deployed PTUs, idle or busy (Utilization V2 at 100 percent returns 429). Optimize with a smaller model, shorter prompts, prompt caching, then PTU for steady load and spillover for bursts.
Latency, Throughput, Token Use, and Cost Metrics
Quick Answer: Measure time to first token (TTFT) separately from end-to-end latency. TTLT = TTFT + (TBT × tokens generated). Throughput is requests per minute (RPM) and tokens per minute (TPM), not "the quota number." Cost splits input, output, and cached tokens. Provisioned throughput units (PTUs) are billed hourly for deployed capacity; pay-as-you-go bills tokens. Optimize with smaller models, shorter prompts, prompt caching, and PTU for steady load.
Domain 4 pairs performance metrics (latency, throughput, response times) with cost metrics (tokens, resource usage). Chapter 11 taught you to buy PTUs. This section teaches you to read whether those PTUs, or a Standard deployment, are healthy and cheap enough.
Latency: TTFT is not end-to-end
Azure OpenAI in Microsoft Foundry publishes a formula you should treat as exam-ready:
TTLT = TTFT + (TBT × Tokens Generated)
- TTFT (Time to First Token): prompt submitted until the first token. Dominated by prefill (prompt size), queueing, and cache misses.
- TBT (Time Between Tokens): average milliseconds between generated tokens (decode speed).
- TTLT (Time to Last Token / Time to Last Byte): end-to-end generation time.
Because TTLT scales with output length, a doubling of wall-clock time with a doubling of Generated Completion Tokens is often expected, not a platform regression. Always pair a latency metric with a token metric.
| Question you have | Metric (display / REST) | Notes |
|---|---|---|
| How long until the user sees the first word (streaming)? | Time to Response / AzureOpenAITimeToResponse | Maps to TTFT at the API gateway. Grows with prompt size and cache misses. Does not include your client. |
| How long until the full answer exists? | Time to Last Byte / AzureOpenAITTLTInMS | Maps to TTLT. Use for non-streaming SLAs. |
| Is decode slow? | Time Between Tokens / AzureOpenAINormalizedTBTInMS | Generation rate. |
| Are long prompts making TTFT look worse than it is? | Normalized Time to First Byte / AzureOpenAINormalizedTTFTInMS | For comparing prompt sizes, not absolute customer latency. |
| How many input / output tokens? | Processed Prompt Tokens / Generated Completion Tokens | Pair with latency. |
Do not use the Cognitive Services - HTTP Requests Latency metric for Azure OpenAI. Microsoft documents that legacy metric as misleading for these workloads.
On the agent dashboard, latency above about 10 seconds is a clue for throttling, heavy tool chains, or network, not a hard SLA Microsoft publishes for every model. Timeouts are a client-side twin of TTLT: a 60-second non-streaming call is cancelled by many gateways and browsers. Streaming (stream: true) does not shorten TTLT, but it delivers tokens as they exist so the client is less likely to time out and the user perceives a faster start (TTFT).
Gateway metrics still omit browser, VPN, and your orchestration delay. If the portal looks healthy and users do not, log a correlation ID on the client (Section 14.3).
Throughput: RPM, TPM, and admission versus capacity
System throughput is what the deployment can process: RPM and TPM. Per-call latency is a different question. Quota on a Standard (pay-as-you-go) deployment is admission control, not a guaranteed TPM. Under load you see HTTP 429 before you "use the quota number."
Estimate TPM from Azure Monitor Processed Prompt Tokens + Generated Completion Tokens, or from each response usage object times observed RPM. That estimate ignores cache hits, so it is conservative for PTU sizing.
Agent run success rate below about 95 percent is Microsoft's dashboard investigation hint: look at failed runs, not only average latency. Split AzureOpenAIRequests by StatusCode and ModelDeploymentName.
Separate workloads onto separate deployments. A bulk summarizer batched with an interactive copilot both lengthens short calls and destroys cache hit rate.
Tokens and cost: input, output, cached
Pay-as-you-go (Standard) bills tokens. Output tokens cost more than input on current Azure OpenAI price lists, and they dominate TTLT because decode is sequential. Reasoning models add reasoning_tokens inside completion details — they count toward cost and latency even when the user never sees them.
Prompt caching (Microsoft Learn, updated 2026-08-11):
- Requires about 1,024 tokens minimum, and the first 1,024 tokens must be identical.
- A single character change in that prefix is a cache miss (
cached_tokens= 0). - Hits appear as
cached_tokensunderprompt_tokens_details. - Place stable system instructions, tools, and files first; put the user question last.
- On Standard, cache reads are discounted versus ordinary input. On provisioned deployments, cached tokens get up to a 100 percent discount on input for capacity: cached tokens do not consume PTU capacity.
- Monitor Prompt Token Cache Match Rate (
AzureOpenAIContextTokensCacheMatchRate) and Active Tokens (total minus cached) on PTU.
Do not invent a cache discount percentage for the exam unless the item prints the price list. Know where the signal lives (cached_tokens, cache match rate) and that cache rate is a PTU sizing input.
PTU utilization versus PAYG spend
Provisioned-managed Utilization V2 (AzureOpenAIProvisionedManagedUtilizationV2) is (PTUs consumed / PTUs deployed) × 100. At 100 percent or more, the service throttles with 429. The older Utilization metric is deprecated.
PTU billing is hourly for the deployed count, idle or busy. You cannot pause a provisioned deployment; billing stops when you delete (or resize). PAYG bills tokens and has no latency SLA. Spillover can send overflow to Standard when PTUs return non-200. Azure Reservations discount the PTU meter; they do not reserve capacity (Chapter 11).
| Pattern | Cost lever | Performance lever |
|---|---|---|
| Variable / dev | Standard PAYG | Accept variable latency |
| Steady high volume, latency SLA | PTU, then a reservation once capacity is deployed | Watch Utilization V2; size with RPM, prompt, completion, cache rate |
| Bursts on top of PTU | Spillover to Standard | Avoid 429s without buying peak PTUs |
| Long static prefixes | Prompt cache | Lower TTFT and PTU burn |
Optimize in this order (cheap to expensive): pick a smaller model that still passes Chapter 13 evals → shorten prompts and lower max_tokens / add stop sequences → cache the prefix → stream for perceived latency → PTU for steady load → reservations for the PTU meter. Mixing eval judges onto the production PTU deployment shows up as utilization and cost that is not user traffic.
Content filtering adds safety latency. That tradeoff is real; do not "fix TTFT" by silently disabling filters on a production agent without a policy decision.
Scenario
A support copilot's p95 TTLT jumps from 4 seconds to 9 seconds after a prompt change. TTFT is flat. Generated Completion Tokens doubled because the new prompt asked for "detailed citations and a full policy restatement." Utilization V2 is 40 percent. Resizing PTUs will not fix this. Cut the completion, add a stop sequence, or summarize citations. If instead TTFT rose while prompt tokens were flat and cache match rate collapsed, look at a prefix change in the first 1,024 tokens.
Common trap
Treating quota TPM as observed throughput, or treating TTLT as a platform outage without checking output tokens. Second trap: using the legacy Cognitive Services Latency metric. Third: assuming PTU hourly cost drops at night — it does not, unless you delete, and you may not get capacity back in the morning.
Operator checklist
- Chart TTLT or TTFT with Generated Completion Tokens and Processed Prompt Tokens.
- Chart 429 on
AzureOpenAIRequestssplit byStatusCode. - Chart Utilization V2 on PTU; token spend on PAYG.
- Chart cache match rate after every prompt-template change.
- Keep judge traffic off the user-facing deployment when you can.
End-to-end latency doubled but first-token time is unchanged and completion tokens doubled. Using TTLT = TTFT + (TBT × tokens generated), what should you conclude?
Which statement about cached prompt tokens is correct for Foundry / Azure OpenAI monitoring and PTU sizing?
You are diagnosing Azure OpenAI latency in Azure Monitor for a streaming chat workload. Which metric pair is the correct starting point?
Steady weekday traffic saturates a Standard pay-as-you-go deployment with 429s, while nights are idle. Which cost and capacity plan matches Microsoft guidance?