7.2 Managing Agents: Microsoft Agent 365, Monitoring & Evaluations

Why Managing Agents Matters

As organizations move from a single pilot agent to dozens or hundreds built across departments, individually testing and trusting each agent stops scaling. The PL-900 blueprint pairs the skills for building agents with a matching set of skills for managing them, because an ungoverned population of AI agents is a real organizational risk — much like ungoverned apps or ungoverned devices was in earlier eras of IT.

Three capabilities anchor this side of the exam objective: Microsoft Agent 365 for organization-wide governance, monitoring for usage and adoption visibility, and evaluations for measuring response quality.

Microsoft Agent 365: Governing Agents at Scale

Microsoft Agent 365 provides organization-wide governance, management, and security for the AI agents an organization builds — treating agents as a managed population the way Microsoft's other admin tools manage devices, apps, and user identities. Rather than each maker's agent existing as an invisible, one-off creation, Agent 365 gives administrators a consistent way to see and control agents across the tenant.

For the exam, remember the core distinction: Copilot Studio is where a maker builds an individual agent; Microsoft Agent 365 is how an administrator governs the whole population of agents an organization has created. This distinction matters because a maker in a single department can publish an agent in minutes — but without a governance layer sitting above every maker's work, an organization has no way to know how many agents exist, what data each one can reach, or which ones should be retired. Microsoft Agent 365 closes that gap by extending the same identity and security thinking Microsoft already applies to user accounts and devices to the growing population of AI agents.

Monitoring Agent Usage and Adoption

Once an agent is published, makers and admins need visibility into whether it's actually working. Copilot Studio provides analytics that show how an agent is performing in production, including:

• Session volume — how many conversations the agent is handling over time
• Resolution rate — the share of conversations the agent resolves without escalating to a human
• Escalation rate — how often users are handed off to a live agent
• Topic engagement — which topics and trigger phrases are used most, and which questions repeatedly fall through to the fallback topic
• User satisfaction — direct feedback signals collected from conversations

This data drives real maker decisions. A high fallback rate on a particular kind of question, for example, signals that the agent needs a new topic or an additional knowledge source to cover that gap; a low resolution rate on an otherwise popular topic can point to a broken tool or an outdated knowledge source that needs attention. At the organizational level, admins can view adoption trends across the tenant — which departments have deployed agents, how usage is growing over time, and where governance attention is needed most. Monitoring closes the loop between building an agent and improving it: without usage data, a maker is guessing at what to fix next.

Agent Evaluations: Measuring Response Quality

Evaluations let a maker systematically test whether an agent's responses match expected outcomes, rather than manually re-testing every scenario by hand after every change. A maker defines evaluation test cases that pair a sample user input with the response or outcome the agent is expected to produce, then runs the evaluation to check the agent's actual behavior against that expectation.

Evaluations matter most at moments of change: after a maker edits a topic, updates a knowledge source, or adjusts how the agent's generative answers behave, an evaluation run can catch a quality regression before it reaches real users — much like a suite of automated tests catches a bug before software ships. This makes evaluations a core part of responsible, quality-controlled agent management rather than a one-time check performed only when an agent is first built. Where monitoring tells a maker what real users experienced in production, evaluations let a maker check quality proactively, on a controlled set of test cases, before a change is ever exposed to a real user — the two capabilities are complementary, not interchangeable.

Bringing It Together: A Governance Checklist

For the exam, connect the three capabilities to the stage of the agent lifecycle they belong to:

• Before publishing — run evaluations to confirm the agent's responses meet expectations
• After publishing — use monitoring/analytics to track real usage, resolution rate, and gaps
• Across the organization — rely on Microsoft Agent 365 to keep every agent inventoried, identity-governed, and secured, regardless of which maker built it

Together, these three capabilities complete the second half of the Copilot Studio exam objective: it isn't enough to know how to build an agent — PL-900 also expects candidates to understand how organizations keep a growing population of agents trustworthy, visible, and under control once they're live.