14.2 Choose an Evaluation Method

Why evaluation methods matter on AB-620

Choose an evaluation method is the second Evaluate agent performance skill. Creating cases without graders is unfinished work: the method defines what “good” means. Microsoft’s agent evaluation model supports multiple methods on the same test set, so a single run can prove both that the agent called the right tool and that the spoken answer meets quality or compliance criteria.

In product language, test methods are also called graders. You add them while creating or editing a test set (Add test method), configure pass scores or expected fields where required, then save. Selecting an existing method lets you edit criteria or remove it.

Method catalog at a glance

Any method that needs expected answers, keywords, or tools produces an Invalid result for a case that is missing those fields. Memorize that for exam distractors that “just enable Exact match with empty expected responses.”

Exact match

Exact match checks whether the agent’s answer equals the expected response character for character, word for word. Same → pass. Any difference → fail.

Use when:

• Short, deterministic outputs: order numbers, SKU codes, fixed status phrases, numeric limits, mandated one-line disclosures.
• You truly require identical text (for example a regulated phrase that must not be paraphrased).

Avoid when:

• Users can correctly phrase the answer many ways (“We open at 9” vs “Our hours start at 9:00 AM”).
• Generative answers naturally vary wording even when meaning is right.

Exact match is the strictest surface grader. On AB-620, if the stem emphasizes a fixed code or immutable phrase, exact match (or keyword match for a required substring) is the defensible choice.

Text similarity

Text similarity scores how closely the agent’s wording and sentence structure match the expected response. Product guidance describes a cosine similarity style metric from 0 to 1 (1 = very close wording). You set a pass score threshold for what counts as pass.

Use when:

• Wording must be nearly identical (legal templates, scripted customer promises) but tiny formatting differences might exist.
• You need a scored wording check rather than binary exact equality.

Contrast with neighbors:

• Exact match requires full equality of the whole answer.
• Keyword match only requires presence of terms, not overall construction.
• Compare meaning accepts different wording if intent aligns.

Pair text similarity with compare meaning when you care about both wording and meaning—common for regulated generative content.

Compare meaning (AI-assisted semantic grading)

Compare meaning evaluates whether the agent’s answer reflects the intended meaning of the expected response using intent similarity, not exact wording. You set a pass threshold (default commonly 50 on a 100% scale) and supply expected answers.

Use when:

• Multiple correct phrasings are acceptable.
• You have a human-approved “ideal answer” but do not want brittle string equality.
• FAQ and knowledge answers should be right in substance more than in script.

Exam tip: Compare meaning still needs expected answers. It is not the same as general quality, which can run without them.

General quality (AI-assisted quality grading)

General quality uses a large language model to score how effectively the agent answered. It is the default method on new test sets and works for single-response and conversational sets. Microsoft’s documented criteria include:

High quality requires meeting the criteria set; incomplete or poorly supported answers score lower. You do not need expected answers for general quality, which makes it ideal for early broad checks and multi-turn conversation quality.

Nuance: Reducing knowledge sources does not guarantee better general-quality scores—retrieved context size and relevance still matter.

Keyword match

Keyword match passes if the agent’s answer contains the expected words or phrases. Configure Any (at least one keyword) vs All (every keyword required).

Use when:

• Mandatory terms must appear (product name, policy ID, “not legal advice” disclaimer fragment).
• You care about presence more than full sentence structure.

Avoid when:

• Keywords can appear in a wrong or unsafe sentence and still “pass.” Keyword match is not a full safety proof.

Tool use

Tool use is pass/fail based on whether the agent used the expected tools or topics for that case. Configure per case or bulk-edit expectations across cases.

Use when:

• Generative orchestration must select the correct REST tool, MCP tool, connector action, or topic.
• You are validating routing design more than prose quality.

Combine tool use with a content grader (compare meaning or general quality) so the agent does not pass by calling the tool while returning a useless message.

Custom methods (manual-style criteria at scale)

Custom lets you define:

1. Evaluation instructions — goal-oriented criteria (what to check).
2. Labels — named outcomes (for example Compliant / Non-Compliant), each marked Pass or Fail toward the method’s pass rate.

Good instructions are goal-oriented, organized with bullets/headings, and limited to allowed characters. Label names support letters, numbers, spaces, and a small set of symbols (- _ / & + .).

Use when:

• HR compliance, brand voice, privacy rules, or multi-factor rubrics do not fit a single similarity score.
• You need human-readable pass/fail labels for stakeholders.

This is the product’s path for “manual review criteria” expressed as an automated labeler. True human spot-checks still matter for calibrating custom instructions and for high-risk releases, but custom methods scale rubric logic beyond one person reading every transcript.

Decision guide: which method when

Manual review vs automated methods

Automated methods do not eliminate human judgment:

• Use manual review of failing transcripts and activity maps to diagnose why a score failed (wrong knowledge, bad instructions, auth profile, tool description).
• Use automation for breadth and regression; use humans for calibration, disputed failures, and safety edge cases.
• For AI-judged methods (general quality, compare meaning, custom), align with stakeholders on pass thresholds and label definitions—treat thresholds as product decisions, not magic constants.

Scenario: Contoso order status agent

The agent must call GetOrderStatus and reply with a status code plus a short customer-friendly sentence.

1. Method pack: Tool use (expect GetOrderStatus) + Exact match on a separate case that only returns the code SHIPPED for a fixed order fixture + Compare meaning for the friendly sentence case with an expected narrative.
2. Keyword method on disclosure cases requires the phrase “tracking number.”
3. General quality remains on for exploratory conversational flows about shipping delays without rigid expected text.
4. Custom method labels privacy violations if the agent invents other customers’ orders.

Exam traps for evaluation methods

• Choosing exact match for long generative FAQs that legitimately vary.
• Forgetting expected answers for similarity/compare meaning/exact match.
• Using keyword match Any when compliance requires All mandatory terms.
• Measuring only prose quality while ignoring wrong tool selection.
• Believing general quality needs golden expected strings (it does not).

Checklist

1. Name each method and whether it is scored or pass/fail.
2. Map answer shape → method using the decision table.
3. Know defaults: general quality present by default; compare meaning often starts near a 50 pass score.
4. Plan multi-method packs for enterprise agents.
5. Keep humans in the loop for calibration and failure analysis.