14.3 Review Test Results

Why reviewing results matters on AB-620

Review test results closes the evaluation loop. A test set and perfect method pack still fail the skill if you cannot interpret scores, diagnose failures, decide whether to publish, and prove that a fix did not create regressions. Domain 3 pairs evaluation with ALM—exam scenarios often ask what you do after a run, not only how you create cases.

How to run an evaluation

From the agent’s Evaluation experience:

1. Select a saved test set (or finish creating one and choose Evaluate).
2. Confirm user profile and connections if the agent needs authenticated knowledge or tools. Broken connections surface a manage-profile dialog; fix connections before trusting results.
3. Start the run. Processing can take up to a few minutes; product guidance notes you can run only one test at a time in the UI.
4. When complete, open the run under Recent results to view the evaluation summary and per-case outcomes.

You can also re-run from recent results (Evaluate test set again) after changing instructions, knowledge, topics, or tools. Each run is stored so you can compare iterations.

Admin dependency: Evaluations that use user authentication require access through the Microsoft Copilot Studio connector. If an admin disables that connection path, authenticated evaluation runs can be blocked—another builder-vs-admin collaboration item for exams.

What a results view typically includes

Use the summary for go/no-go signals and the case drill-down for why. AB-620 items often reward candidates who look beyond a single percentage and inspect tool path or grounding evidence.

Pass/fail metrics and thresholds

Interpret metrics in the language of the methods you chose:

Define release thresholds before the run—not after you see a disappointing number. Example policy:

• Golden set tool use pass rate 100% for critical paths.
• Compare meaning average ≥ 70 on knowledge FAQs.
• General quality ≥ agreed floor on conversational smoke pack.
• Zero Invalid cases (configuration hygiene).

Document thresholds with the Center of Excellence or product owner so evaluation is a contract, not a maker-only preference.

Failure analysis workflow

When cases fail, classify before rewriting the entire agent:

Always open the activity map and resources used before blaming “the model.” Many “AI quality” failures are actually routing, permission, or content problems.

Regression testing before publish

Treat evaluation like automated regression for agent software:

1. Baseline: Run the golden set on the last known good published version; store scores/export.
2. Change: Update topics, knowledge, tools, or prompts in a dev/test environment.
3. Re-run the same set (do not silently swap cases mid-comparison).
4. Diff: Compare method scores and newly failing cases.
5. Fix or accept: Resolve regressions or explicitly accept with change-control notes.
6. Publish / promote only when gates pass.

Exam trap: Publishing after a single happy-path test chat while the golden set is red. Domain 3 expects evaluation-aware release discipline.

Export, retention, and evidence

• Results remain available in Copilot Studio for 89 days.
• Export to CSV for longer archives, dashboards, or audit packets.
• Keep golden-set definitions (questions + expected answers + methods) in source control or a solution-aware process alongside the agent where possible so evidence is reproducible.

Linking results to ALM

Evaluation is not a separate hobby from solutions, environment variables, and Power Platform Pipelines (covered in the next chapter). Connect them deliberately:

Automation paths Microsoft documents:

• Run evaluations from the Copilot Studio UI.
• Trigger evaluations programmatically with the Power Platform REST API (agent test set ID, environment, OAuth token).
• Integrate API calls into CI/CD or Power Automate so a pipeline can fail a deployment when quality regresses.
• Use connectors/actions in tools or flows where your automation strategy lives.

API automation helps you:

• Validate quality after agent updates.
• Run recurring checks against staging or production agents (with appropriate risk controls).
• Detect regressions early in the lifecycle.

You still review failures as a human for root cause—the API does not replace diagnosis.

Compare runs over time

Healthy teams keep a simple history:

Without comparable sets and saved runs, “we tested it” is not measurable.

Scenario: Contoso release blocked by evaluation

Contoso prepares to promote the benefits agent via a Power Platform Pipeline.

1. Pipeline checklist requires the Benefits-golden-2026 set to meet tool-use 100% and compare-meaning average ≥ 75.
2. Run fails: three enrollment cases no longer call the custom connector after a description edit.
3. Maker opens case details and activity map, restores the tool description’s “when to use” language, and adds a fourth negative case that must not call the tool.
4. Re-run shows gate green; CSV export attached to the change request.
5. Pipeline promotes to test, then production. Weekly scheduled API evaluation watches for drift after SharePoint knowledge updates.

Act-on-results playbook

Based on outcomes, Microsoft-aligned practice is to change the agent, not only the score target:

• Update knowledge sources, topic triggers, instructions, or tool configuration.
• Adjust evaluation methods only when the method was the wrong ruler (exact match on free-form FAQs).
• Expand the test set with new production themes so the suite stays honest.
• Re-run until fixed cases pass and previously passing cases still pass.

Exam traps for review results

• Reading only the headline percentage and ignoring Invalid or tool-path failures.
• Changing the test set between runs and calling it a fair regression comparison.
• Assuming results live forever without CSV export (remember 89 days).
• Treating evaluation as unrelated to pipelines and solution promotion.
• “Fixing” quality by lowering thresholds without business approval.

Checklist

1. Run → summary → case drill-down (transcript, activity, resources).
2. Classify failures; fix agent or fix method configuration deliberately.
3. Re-run the same golden set for regression.
4. Export evidence; respect retention limits.
5. Wire gates into ALM and optional Power Platform API automation.