12.1 Distributed Tracing with OpenTelemetry SDKs

Why OpenTelemetry matters on AI-200

Domain 4 asks you to secure, monitor, and troubleshoot Azure solutions. For AI backends, “monitor” is not a single App Service log stream. A typical request may hit a Container Apps API, call Azure OpenAI or a custom model endpoint, retrieve embeddings from PostgreSQL/pgvector or Cosmos DB, cache with Managed Redis, and publish work to Service Bus or Event Grid. When latency spikes or errors cascade, you need one correlated story across those hops.

OpenTelemetry (OTel) is the industry-standard, vendor-neutral telemetry framework Microsoft expects you to recognize. You instrument once with OTel SDKs, then export to Azure Monitor / Application Insights. On the exam, prefer OTel + Azure Monitor over inventing custom logging-only correlation.

Signals: traces, metrics, and logs

OTel describes three complementary signals. AI-200 scenarios lean hardest on traces, but metrics and correlated logs complete the picture.

Do not treat these as interchangeable. A metric can show that p95 latency doubled; a trace shows which span (vector search vs model call) caused it; a log shows the exception message on that span.

Traces and spans

A trace represents one logical operation traveling through a distributed system—for example, POST /chat that triggers retrieval and generation. Every trace has a trace ID shared by all participating services.

A span is a single timed unit of work inside that trace. Spans form a tree (or directed acyclic graph) via parent/child relationships:

• Root span: inbound HTTP request to your API
• Child span: vector_search against PostgreSQL
• Sibling or nested child: chat_completions against Azure OpenAI
• Downstream child: servicebus_send when you enqueue follow-up work

Each span carries:

• Name (operation identity such as GET /health or embeddings.create)
• Start/end timestamps (duration)
• Span kind (server, client, producer, consumer, internal)
• Attributes/tags (model name, deployment, HTTP status, DB system, custom ai.prompt.tokens)
• Status (ok vs error) and optional error description
• Span events (timed log-like annotations on the span)

On AI-200, expect questions that distinguish “add a child span around the model call” from “emit an unstructured console line with no trace ID.”

W3C Trace Context propagation

Correlation only works if every hop forwards context. The W3C Trace Context standard defines HTTP headers:

When Service A calls Service B, A’s active span ID becomes B’s parent. The trace ID stays constant. Messaging and Functions need the same idea: inject context into Service Bus application properties or Event Grid event metadata, then extract it in the consumer so the consumer span is a child of the producer span—not an orphan root.

If you forget propagation, Azure Monitor shows disconnected islands of traces. Application Map looks sparse, and KQL joins on operation_Id fail to reconstruct the full path.

SDK building blocks (conceptual map)

Regardless of language, OTel SDKs share the same mental model:

For Python services common on AI-200 labs:

1. Install OpenTelemetry API/SDK packages and Azure Monitor OpenTelemetry distro or exporter packages as appropriate.
2. Configure a TracerProvider with a resource (service.name, service.namespace, cloud role).
3. Enable auto-instrumentation for inbound/outbound HTTP.
4. Add manual spans around AI-specific work that libraries do not cover well (chunking, reranking, tool-calling loops).
5. Attach attributes that you will later query: deployment name, temperature, retrieval top-k, cache hit/miss.
6. Register an exporter so spans leave the process (OTLP or Azure Monitor).

Manual vs automatic instrumentation

Automatic instrumentation captures framework and client calls with less code. It is ideal for HTTP servers, HTTP clients, and many database drivers.

Manual instrumentation is mandatory for AI business logic:

• Span for “build retrieval query”
• Span for “similarity search”
• Span for “compose prompt”
• Span for “model completion”
• Span for “post-process / safety filter”

Without manual spans, your trace may show only a fat HTTP server span and hide where the eight-second delay occurred.

Metrics alongside traces

OTel meters create counters, histograms, and gauges. Useful AI metrics include request counts, error counts, token usage histograms, and cache hit ratios. Metrics answer fleet-level questions; traces answer incident questions. Export both when possible so Azure Monitor workbooks and KQL can correlate.

Sampling and cost awareness

Full tracing of every token-heavy request can be expensive. Samplers (for example, parent-based or ratio-based) decide which traces to export. On the exam, understand the tradeoff: lower sampling reduces cost but can miss rare failures; head-based sampling decisions should propagate so children agree with the parent’s sampled flag in traceparent.

Scenario: instrumenting a RAG API

Priya builds a Container Apps Python API. Users report “chat is slow.” Without OTel, she only sees container CPU is fine. After adding OTel:

1. Inbound FastAPI request creates a server span.
2. Auto-instrumentation creates a client span for the PostgreSQL query.
3. Manual span wraps Azure OpenAI chat completions with attributes for deployment and token counts.
4. W3C headers flow to a downstream Functions worker via Service Bus properties.

Priya opens the trace and sees model completion at 6.2s while retrieval is 180ms. The fix is deployment capacity or prompt size—not “scale the database.” That is the operational skill AI-200 is testing.

Common exam pitfalls

• Confusing trace ID (shared) with span ID (per operation)
• Logging only to stdout and assuming Azure can reconstruct distributed causality
• Creating a new root span on every Service Bus consumer message instead of extracting parent context
• Putting secrets or raw PII prompts into span attributes
• Instrumenting only the API gateway and ignoring worker/container side paths

Checklist before you move on

You should be able to define trace vs span vs metric, explain W3C traceparent, list SDK provider/exporter roles, and describe where manual spans belong in an AI pipeline. Section 12.2 connects these spans to Application Insights and Azure Monitor so correlation becomes visible in the portal.