14.3 Logging, Distributed Tracing, and Production Debugging

Logging, Distributed Tracing, and Production Debugging

Quick Answer: Foundry tracing is OpenTelemetry. Each agent run is a trace of nested spans (LLM call, tool call, retrieval, agent-to-agent). Export to Azure Monitor Application Insights. Debug a bad answer by walking trace → retrieved chunks → prompt version → model deployment. Redact personally identifiable information (PII). Viewing logs needs Log Analytics Reader; protected gen-AI content needs Privileged Monitoring Data Reader.

Domain 4 asks you to configure detailed logging, tracing, and debugging for production troubleshooting. Sections 14.1 and 14.2 tell you that quality or latency moved. This section tells you which span did it.

OpenTelemetry in Foundry

Traces record a request's journey. Spans are timed operations inside that journey. Attributes are key-value metadata. Foundry follows OpenTelemetry generative-AI semantic conventions and W3C Trace Context, so a traceparent correlation ID can follow the call from your API gateway into the agent.

Typical spans on a RAG agent:

• Parent invoke_agent / user run
• Retrieval or execute_tool (Azure AI Search, function, Model Context Protocol tool)
• LLM / model completion (llm_spans)
• Nested agent_to_agent_interaction or agent_planning on multi-agent systems

Microsoft, with Cisco Outshift, extended conventions for multi-agent systems. Integrations include Foundry, Microsoft Agent Framework, LangChain, LangGraph, and the OpenAI Agents SDK. Use consistent span attribute names across agents so one Kusto query works everywhere. Correlate evaluation run IDs with traces so a groundedness failure and the retrieval span share a screen.

Tracing is generally available for prompt and hosted agents. Workflow and external agents are preview. Server-side tracing turns on when you connect Application Insights to the project; no code change is required for hosted agents, and traces show up within minutes. Client-side instrumentation (opentelemetry-sdk, azure-core-tracing-opentelemetry) is for your surrounding application code. Local OpenTelemetry Protocol (OTLP) export via the Foundry Visual Studio Code toolkit is for development without cloud access.

The portal Traces view searches about the last 90 days. Retention and cost follow the Application Insights / Log Analytics plan — tracing is not free, and high-cardinality prompt payloads are an ingestion bill as well as a privacy bill.

Correlation, conversations, and Azure Monitor

A Conversation is the durable thread between user and agent. In the Foundry portal you can jump from Response ID or Trace ID to conversation history, tokens, ordered tool calls, and inputs/outputs. In Application Insights the same records land in dependencies, traces, and (for sensitive payloads) AppGenAIContent.

Put your own correlation ID on inbound HTTP and copy it into application logs so a Service Bus or Azure Functions hop is still joinable. Do not rely on "the user said it was around 3 p.m."

To view traces you need Log Analytics Reader on the connected Application Insights / Log Analytics workspace. If tables are protected, also assign Privileged Monitoring Data Reader. Empty traces usually mean the Application Insights connection is missing, there is no recent traffic, or ingestion is delayed a few minutes. Client-side traces missing after server-side works usually mean the SDK packages or connection string were never configured.

Resource logs on the Cognitive Services / Foundry Models account (Request and Response, Trace, Audit) complement Application Insights when you need gateway status codes. Turn diagnostic settings on purpose; full request/response logs are another PII store with their own export cost.

Redact PII and restrict sensitive spans

Traces will capture prompts, outputs, tool arguments, and system instructions. Treat them as production telemetry with the same access controls and retention as logs.

Do not put secrets, API keys, or credentials in prompts, tool arguments, or span attributes.

Microsoft treats these OpenTelemetry attributes as sensitive:

• gen_ai.input.messages
• gen_ai.output.messages
• gen_ai.system_instructions
• gen_ai.tool.definitions
• gen_ai.tool.call.arguments
• gen_ai.tool.call.result
• gen_ai.evaluation.explanation

To lock them down:

1. Register the protectGenAISensitiveData feature flag on the subscription so payloads route to the AppGenAIContent table.
2. Set that table's protection level to Protected (deny by default).
3. Grant Privileged Monitoring Data Reader only to people who must read raw prompts. Privileged Identity Management can make that access time-bound.
4. Verify a Log Analytics Reader-only user sees pointers, not the raw messages.

Before 30 September 2026, values may still duplicate into AppDependencies, AppTraces, and AppEvents. After that date, new ingest keeps keys in the old tables and stores values only in AppGenAIContent. Update custom Kusto alerts and workbooks before then. Registering optOutProtectGenAISensitiveData only delays the cutover until 30 September 2027. Redact in application code as well — table protection is not a substitute for sending fewer secrets.

Production debugging playbook

A user says the agent cited the wrong policy. Do not start by fine-tuning. Walk the trace:

1. Find the run by Response ID, Conversation ID, or timestamp plus a user hash.
2. Read the retrieval span. Which chunks were actually passed into the prompt? If the right PDF never appears, this is a search, index, or access-control bug (Chapter 15), not a model bug.
3. Read the LLM span prompt. Which prompt version (Git tag from Chapter 12) and tool definitions were on the wire? A silent portal edit is a common outage.
4. Read the model deployment. Did spillover send this call to a different Standard deployment or an older version?
5. Read token and timing on each span. A 12-second tool span with a 200 ms LLM span is not a PTU problem.
6. Cross-link evaluation. If continuous eval scored groundedness 0.2 on this trace, you have a labeled failure for the regression suite.

Record the prompt version, index alias, and deployment name as span attributes so the next incident does not require guessing.

Scenario

P95 latency is fine, but 4 percent of claims answers mention a retired form. The trace for a failing Conversation ID shows retrieval returning chunk IDs from index alias claims-v3, while production should have flipped to claims-v4. The prompt version is correct; the model deployment is correct. The tool argument still hard-codes the old alias. Fix the tool configuration, add a span attribute for index alias, and alert when it is not claims-v4. Fine-tuning the model would have hidden the retrieval miss.

Common trap

Logging full prompts to a shared Application Insights instance that every developer can query, then "turning on tracing" in production without role-based access control or redaction. Second trap: debugging only from the final assistant message without opening child spans — you will fine-tune a model for a retrieval miss. Third: expecting traces without Application Insights connected, or without Log Analytics Reader.

Debugging checklist

• Connect Application Insights; confirm a trace after one test run.
• Instrument client code if the bug is outside the hosted agent.
• Standardize span names and attributes; include prompt version, deployment name, and index alias.
• Redact PII; protect AppGenAIContent.
• Walk trace → chunks → prompt version → deployment before you retrain.