13.3 Monitor Agents with Application Insights

13.3 Monitor Agents with Application Insights

Quick Answer: In the agent Settings → Advanced → Application Insights, paste the Azure connection string, enable the logging options your privacy policy allows, then diagnose production behavior with Logs (Kusto), customEvents/customDimensions, and the Copilot Studio Dashboard workbook.

Building Foundry-backed answers and catalog-powered prompts is incomplete without observability. AB-620’s third Azure integration skill is monitor agents by using Application Insights—agent-level telemetry into Azure Monitor for reliability, latency, topic behavior, and custom diagnostics.

Why Application Insights for agents?

Copilot Studio analytics in the product UI help makers, but enterprise operations teams standardize on Azure Monitor / Application Insights for:

• Cross-app correlation with APIs, App Services, and Functions the agent calls
• Kusto (KQL) queries, alerts, and workbooks
• Longer retention and security-team access patterns
• Exception and dependency signals when tools fail
• Custom events authors emit from topics for business KPIs

Think of Application Insights as the production flight recorder for conversational AI—not a replacement for responsible AI evaluation test sets (Domain 3), but the runtime half of quality.

Connection configuration

Microsoft’s agent-level telemetry article describes this maker path:

1. Create (or reuse) an Application Insights resource in an appropriate Azure subscription and region strategy.
2. Copy the resource connection string (modern guidance prefers connection strings over legacy instrumentation keys alone).
3. In Copilot Studio, open the agent Settings page.
4. Select Advanced.
5. In Application Insights, paste the Connection string.
6. Optionally enable detailed logging switches (next section).
7. Save, then exercise the agent in test canvas and published channels.
8. In Azure portal → Application Insights → Logs, confirm events arrive.

Telemetry is emitted when users interact—including during testing in Copilot Studio—so empty logs usually mean wrong connection string, blocked connector policy, or queries aimed at the wrong resource/time window.

Logging options and privacy

Microsoft documents optional settings such as:

Exam and real-world rule: more logging is not automatically “more mature.” Contoso’s privacy office may require Log conversation details off in production customer channels while still allowing coarse event metrics. Document the decision; do not silently enable full message capture.

What you can diagnose

Custom dimensions worth memorizing

Much of Copilot Studio’s detail rides in customDimensions on customEvents:

Important caveat: unique user counts are meaningful when identities are stable and authenticated. Anonymous channels that mint a new ID per conversation inflate “users.”

Example KQL patterns (conceptual)

Distinct users per day (last 14 days)—useful for adoption charts:

let queryStartDate = ago(14d);
customEvents
| where timestamp > queryStartDate
| summarize uc=dcount(user_Id) by bin(timestamp, 1d)
| render timechart

Exclude test canvas traffic:

customEvents
| extend isDesignMode = customDimensions['designMode']
| where isDesignMode == "False"

Start investigations with customEvents then narrow by time, channel, and event name. Learn documents that a simple table query is valid—complexity is optional.

Copilot Studio Dashboard workbook (preview)

Application Insights can show a Copilot Studio Dashboard workbook under Monitoring → Workbooks. It consolidates conversations, latency, exceptions, tool usage, and topic analytics. Teams can Edit the workbook, add KQL tiles for custom attributes, Save variants, and Share with colleagues who have at least Reader on the Application Insights resource.

Use the workbook for stand-ups; use raw Logs for deep incident response.

Environment-level telemetry and product analytics

Microsoft also documents broader telemetry stories (environment-level Application Insights preview, Power Platform export scenarios, and product analytics). For AB-620’s wording—monitor agents by using Application Insights—prioritize agent-level connection string configuration, logging toggles, custom dimensions, and diagnostic queries. Mention environment-level options as supplemental, not a substitute for knowing the agent Advanced settings path.

Governance: disabling agent-level telemetry

To prevent makers from enabling agent-level Application Insights telemetry, admins can block the Application Insights in Copilot Studio connector via a data policy in the Power Platform admin center:

1. Security → Data and privacy → Data policy.
2. Create or edit a policy; add target environments.
3. Find Application Insights in Copilot Studio.
4. Move it to Blocked.
5. Publish.

When enforced, makers cannot use the connector in governed environments. Exam stems that say “compliance forbids exporting utterances to Azure” often want DLP block, not “delete the agent.”

Scenario — diagnosing a slow Foundry prompt path

Contoso’s claims agent spiked to 12-second responses after a model catalog change. Ops opens Application Insights:

1. Workbook latency tile confirms the regression after Tuesday 16:00 UTC.
2. Logs filter designMode == False and channelId == "msteams".
3. Node execution events show long gaps around the custom prompt tool.
4. Correlation with Azure metrics on the Foundry deployment shows throttling.
5. Team either scales the deployment or switches the high-volume prompt to a mini model (Section 13.2 tradeoffs).

Without App Insights, the team would only hear “Teams is slow” with no evidence.

Scenario — false alarm from test traffic

A dashboard shows thousands of daily “users,” but production publish is limited to a pilot group. Querying without filtering designMode counted maker test canvas sessions. After excluding design mode, real pilot usage matches expectations. Always separate lab traffic from production analytics.

Pitfalls

1. Pasting the wrong resource’s connection string.
2. Enabling full conversation text logging without privacy approval.
3. Alerting on test-canvas noise.
4. Assuming anonymous channels yield trustworthy unique user counts.
5. Forgetting Reader permissions when sharing workbooks.
6. Ignoring DLP when “settings refuse to save” telemetry.
7. Using App Insights alone as a substitute for formal evaluation test sets.

Success checklist

• Configure Settings → Advanced → Application Insights connection string.
• Explain each logging toggle and its sensitivity.
• Query customEvents and extend customDimensions (especially designMode, channelId).
• Use workbooks for health overviews and KQL for incidents.
• Diagnose latency/errors by correlating topic/node/tool signals.
• Govern telemetry with data policies when required.

With Search+Foundry generative answers, Foundry model catalog prompts, and Application Insights monitoring fluent, you can design, power, and operate Azure-integrated Copilot Studio agents for AB-620’s Domain 2 Azure section end to end.