14.1 Create a Test Set

Why creating a test set matters on AB-620

Create a test set is the first skill under Evaluate agent performance in Domain 3 (Test and manage agents, about 20–25% of AB-620). Microsoft does not treat evaluation as optional polish. Enterprise agents that call tools, ground on knowledge, and serve internal or external users need a repeatable way to prove quality before publish and after every material change.

Agent evaluation is automated, structured testing: you define test cases, group them into a test set, run the set against the agent, and score each response with one or more test methods (graders). That is different from the authoring test chat, which is interactive and useful for exploration but weak as regression evidence because it is hard to rerun the same suite after every topic, knowledge, or tool change.

Core vocabulary

When you run an evaluation, Copilot Studio sends the questions, records the agent’s responses, compares them to expectations or quality criteria, and scores each case. You can inspect the transcript, activity map, and which resources the agent used—details that ad-hoc chat rarely captures consistently.

Single-response vs conversational test sets

Choose the set type based on what behavior you need to prove:

Exam framing: If the stem says “validate that order status always invokes the REST tool and returns the same SKU code format,” think single-response with tool-use and exact/keyword graders. If the stem says “confirm the agent remembers the employee’s location across three turns before recommending a policy,” think conversational evaluation.

How to create a single-response test set (product path)

1. Open the agent in Copilot Studio and go to the Evaluation page (or Evaluate tab depending on UI density).
2. Select New evaluation → Single response.
3. Choose how to populate cases (see creation methods below).
4. Name the test set clearly (for example HR-policy-golden-v3 or Order-status-regression).
5. Add or adjust test methods and expected answers as required.
6. Save, then Evaluate, or save first and run later from Recent results.

Important operational limit: test results are retained in Copilot Studio for 89 days. Export to CSV if you need longer evidence for audits, change boards, or ALM gates.

Ways to populate test cases

Microsoft supports multiple authoring paths so you do not have to invent every prompt by hand:

CSV / spreadsheet import shape

For import files, Microsoft’s template pattern uses two columns in order:

1. Question
2. Expected response

Rules that show up in both product docs and exam-style questions:

• Each question is limited to about 1,000 characters including spaces.
• Save as .csv or .txt (or follow the downloadable template under the data-source step).
• Expected response is optional at import, but you need expected responses (or keywords/tools, depending on method) to run exact match, text similarity, compare meaning, and related reference-based methods. Without them, those methods return Invalid for the case.

Designing coverage: intents, tools, knowledge, and risk

A weak test set is a list of happy-path FAQs. A exam-ready and production-ready set is a coverage matrix. Build cases across at least these dimensions:

Golden set discipline: Mark a smaller core of cases (often dozens, not necessarily the full 100) as release-blocking. Those cases get human-reviewed expected answers, stable naming, and ownership. Use broader AI-generated cases for discovery; promote only proven cases into the golden set.

Expected answers that grade well

Writing expected answers is part of test design, not an afterthought:

• For exact match, keep answers short and deterministic (“24 hours”, “SKU-1044”, “Contact HR at hr@contoso.com”).
• For compare meaning or general quality, write the intent of a correct answer in clear prose; do not over-constrain wording if paraphrase is acceptable.
• For keyword match, list the mandatory terms (policy name, legal phrase, product code) rather than a full essay.
• For tool use, select the expected tools/topics per case (or bulk-edit when every case shares the same tool expectation).

User profiles and connections

Many enterprise agents behave differently based on identity and connected systems. When creating or editing a test set, you can attach a user profile and verify connections (green/active vs broken). Automated evaluation uses that profile’s authentication. You can run some tests without a profile, but if the agent needs knowledge tools or connectors that require auth, a broken profile produces failed or invalid runs rather than trustworthy quality signal.

Scenario: Contoso employee benefits agent

Contoso builds a Copilot Studio benefits agent with SharePoint knowledge, a custom connector for enrollment status, and a topic for “open enrollment deadlines.”

1. Maker creates a single-response test set named Benefits-golden-2026.
2. Imports 40 CSV questions covering plan options, deadlines, and out-of-scope payroll tax advice.
3. Adds expected responses for factual cases and expected tool use for enrollment-status questions.
4. Generates a quick question set of 10 more prompts from instructions, then deletes duplicates and weak generic items.
5. Pulls three failing prompts from test chat into the set after a pilot.
6. Tags twelve cases as the golden gate for pipeline promotion; keeps the rest as exploratory coverage.
7. Documents that results will be exported weekly because stakeholders need longer than 89 days of history.

Exam traps for create-a-test-set

• Confusing test chat with a test set—only the latter is the evaluation batch artifact.
• Assuming every method works without expected responses—many return Invalid without them.
• Designing only positive FAQs and forgetting tool, refusal, and knowledge-miss cases.
• Ignoring single-response vs conversational limits when the scenario clearly needs multi-turn context.
• Never naming or versioning sets, so regression comparison becomes impossible.

Checklist before you leave this skill

1. Can you define test case, test set, expected response, and golden set in one sentence each?
2. Can you list at least four ways to create cases (manual, CSV, AI generate, test chat, themes)?
3. Do you know the approximate case caps for single-response vs conversational sets?
4. Can you design a coverage matrix across intents, tools, knowledge, and safety?
5. Do you remember 89-day result retention and CSV export for longer evidence?