12.4 KQL for Metrics & Troubleshooting Scenarios

From logs to metrics-minded troubleshooting

Section 12.3 focused on log/trace tables. AI-200 also asks you to analyze metrics and apply KQL in realistic troubleshooting scenarios—especially latency and errors in multi-stage AI pipelines. Metrics tell you whether a problem is widespread; traces/logs tell you why a representative operation failed.

Metrics sources in Log Analytics

Even if a stem says “metrics,” duration percentiles from requests/dependencies are valid metric-style analysis.

Core aggregations for latency

Average alone hides tail latency. Prefer percentiles for user-facing AI APIs:

AppRequests
| where TimeGenerated > ago(4h)
| where Name == "POST /chat"
| summarize
    count(),
    avg(DurationMs),
    p50=percentile(DurationMs, 50),
    p95=percentile(DurationMs, 95),
    p99=percentile(DurationMs, 99),
    failures=countif(Success == false)

Time-bin to see when the regression started:

AppDependencies
| where TimeGenerated > ago(6h)
| where Target has "openai"
| summarize p95=percentile(DurationMs, 95), errors=countif(Success == false) by bin(TimeGenerated, 10m)
| order by TimeGenerated asc

A step-change at a deployment timestamp is a strong exam clue: correlate with revision change, model deployment update, or prompt size increase.

Split the AI pipeline by dependency target

Map stages to dependency targets (names vary by SDK instrumentation):

Example KQL:

AppDependencies
| where TimeGenerated > ago(2h)
| summarize
    calls=count(),
    p95=percentile(DurationMs, 95),
    err=countif(Success == false)
  by Type, Target
| order by p95 desc

If model Target p95 is 7s and Postgres is 50ms, do not “optimize the index” first—address model latency, quota, or prompt size.

Custom metrics (tokens, hits, queue depth)

When OTel meters export custom metrics into AppMetrics (schema fields vary slightly by ingestion path), summarize by metric name and dimensions:

AppMetrics
| where TimeGenerated > ago(1h)
| where Name in ("gen_ai.client.token.usage", "cache_hit", "jobs_queue_depth")
| summarize Total=sum(Sum), Samples=sum(Count) by Name, bin(TimeGenerated, 5m)

Exam takeaway: custom metrics are how you watch token burn and cache effectiveness—signals that raw HTTP status codes alone may not show.

Troubleshooting playbooks

1) Sudden latency spike after deploy

1. Chart AppRequests p95 by bin(TimeGenerated, 5m) around the deploy time.
2. Split AppDependencies p95 by Target for the same window.
3. Pick one slow operation_Id from AppRequests ordered by DurationMs desc.
4. Join traces/exceptions for that ID.
5. Confirm whether a new revision, larger context window, or cold start pattern explains it.

2) Elevated error rate with 429s

AppDependencies
| where TimeGenerated > ago(3h)
| where ResultCode == "429" or ResultCode == 429
| summarize count() by bin(TimeGenerated, 5m), Target

Then verify request volume on AppRequests did not simply triple. Remediation themes: quota, concurrency limits, backoff, caching embeddings, batching.

3) Timeouts only on first requests (cold start)

Look for long durations on Functions or scale-from-zero Container Apps shortly after idle periods. Metrics may show instance count rising from zero; dependencies may be fine once warm. Solutions include min replicas, preload, or accepting warm-up SLOs—not blind Postgres scaling.

4) Cache miss amplification

If Redis dependency errors rise or cache hit custom metrics fall, Postgres/Cosmos dependency volume often rises in the same bins. Join the timeline:

AppDependencies
| where TimeGenerated > ago(2h)
| summarize RedisCalls=countif(Type has "Redis"), DbCalls=countif(Type has "SQL" or Target has "postgres") by bin(TimeGenerated, 5m)

Inverse movement (Redis down, DB up) is a classic amplification signature.

5) Worker lag in async generation

API requests succeed quickly (enqueue only) while users wait on job status. Investigate worker AppRequests/AppDependencies separately by cloud role name, and check Service Bus metrics or custom jobs_queue_depth. Application Map should show both roles if propagation is correct.

Operator reminder for metric scenarios

Scenario: end-to-end AI incident

Team Nova runs a RAG chat API on Container Apps with a Functions scoring worker. Users report “answers hang.”

1. AppRequests on the API role show normal p95—enqueue path is healthy.
2. Worker role AppRequests p95 jumped from 2s to 12s after 16:00.
3. AppDependencies shows Azure OpenAI p95 11s; pgvector stable.
4. ResultCode 429 counts rise on the model target.
5. One operation_Id join to AppTraces shows “insufficient quota” messages.

Root cause: quota exhaustion under a marketing traffic spike—not a vector index outage. The fix is quota/capacity plus backoff/cache, validated by watching 429 counts and p95 fall in the next time bins.

Exam decision habits

• Separate symptom (API slow) from stage (which dependency).
• Prefer percentile + bin over a single average.
• Use operation_Id when you need proof, not only aggregates.
• Include container console queries when telemetry stops entirely.
• Choose remediations that match the signature (429 ≠ restart Postgres).

Chapter synthesis

OpenTelemetry SDKs create correlated traces and metrics; Azure Monitor/Application Insights store and visualize them; KQL turns both logs and metrics into diagnoses. Together they satisfy the Domain 4 skill to monitor and troubleshoot Azure solutions for AI backends—exactly the operational loop AI-200 expects you to demonstrate.