7.2 How Testing Center Evaluations Work
Key Takeaways
- Default turn-level scorers are Response Evaluation, Subagent Assertion, and Action Assertion, and they run on every test.
- Conversation-level default scorers are Task Resolution, Subagent Evaluation, and Actions Evaluation.
- Response quality scorers such as completeness, coherence, and conciseness use an LLM judge on a 0-5 scale, where 3 or higher is a pass.
- Subagent and action assertions score 1 for pass and 0 for fail; conversation-level action evaluation passes if all expected actions occurred, even if extra actions ran.
- Custom scorers define LLM-judge criteria in a prompt template and return labels, with allowed and fallback values, or free-form results.
7.2 How Testing Center Evaluations Work
Quick Answer: Every Testing Center run applies default scorers. At turn level these are Response Evaluation, Subagent Assertion, and Action Assertion. At conversation level they're Task Resolution, Subagent Evaluation, and Actions Evaluation. You can add quality scorers: Completeness, Coherence, Conciseness, Latency, and Instruction Adherence. Quality scores come from an LLM judge on a 0–5 scale with 3 or higher passing. Custom scorers let you write your own judge prompt for criteria such as tone or policy adherence.
What an "LLM Judge" Is
Several scorers use LLM-as-judge: one LLM evaluates another LLM's output. The judge receives a prompt with the task and criteria (factual accuracy, relevance, coherence, faithfulness), determines the expected response, compares it with the agent's response, and returns a score, ranking, or written reasoning. Salesforce designs these judge prompts for accurate, useful results. You can inspect the reasoning for any score in the results detail panel.
Default Scorers
Turn-level
| Scorer | How it works | Result |
|---|---|---|
| Response Evaluation | The test utterance goes to the agent, and an LLM judge determines the expected response, compares it with the actual response, and scores it | 0–5. 5 = precise, complete, and on-brand; 3–4 = minor omissions or ambiguity; 1–2 = unclear or missing key elements, possibly asking for data that should come from the CRM; 0 = generic and doesn't address the intent |
| Subagent Assertion | Checks whether the agent selected the expected subagent for the utterance | 1 = pass, 0 = fail (an unexpected subagent was selected) |
| Action Assertion | Checks whether the agent selected all the right actions within the subagent | 1 = pass, 0 = fail (wrong actions, or a missing action) |
A common Response Evaluation mistake: the Expected Response field must contain the reply you expect, not instructions such as "the agent should mention the order number and be polite." Instructions in that field produce inaccurate scores.
Conversation-level
| Scorer | Result |
|---|---|
| Task Resolution | Pass or fail, based on whether the agent resolved the user's issue |
| Subagent Evaluation | Whether the agent routed to the expected subagents during the conversation |
| Actions Evaluation | Whether every expected action occurred. It passes even if the agent also ran additional actions |
Actions Evaluation example:
| Expected actions | Agent did | Result |
|---|---|---|
| Look up account, send email | Look up account, send email, create case | Pass |
| Look up account, send email | Look up account, create case | Fail, because send email is missing |
Quality Scorers (Optional)
Response quality scorers measure three criteria: accuracy (correct information), relevance (addresses intent and context), and voice and tone (style and brand).
| Scorer | What it measures | Agent testing scale |
|---|---|---|
| Completeness | Whether the response covered all important information expected | 0–5; pass at 3 or higher |
| Coherence | Whether information was turned into grammatical, conversational language instead of raw data such as JSON | 0–5; 0–2 means raw data was delivered |
| Conciseness | Whether the response is short but accurate | 0–5; 0–2 means lengthy, repetitive, or irrelevant |
| Latency | Test execution time in milliseconds | Not scored. Unusually high values suggest utterance or infrastructure issues |
| Instruction Adherence | How fully the agent followed the subagent instructions | High, Low, or Uncertain |
The pass/fail threshold for quality scorers is 3. If that doesn't fit your needs, add a custom scorer with your own criteria.
Prompt Template Testing Scorers
When batch-testing prompt templates in the new Testing Center, scorers include Completeness, Coherence, Conciseness, and:
| Scorer | Scale |
|---|---|
| Factuality | 0–5. 3–5 = accurate, well-supported; 1–2 = errors or unsupported claims; 0 = significantly inaccurate or fabricated |
| Instruction Following | 1, 3, or 5 (Low, Uncertain, High) |
| Response Match | 0–5 closeness to the Expected Output column you provide |
Custom Scorers
Custom scorers evaluate criteria you define, such as tone, brand voice, policy adherence, sentiment, resolution quality, or whether a response is valid JSON.
- In the test suite, click Select Scorers → Add → LLM Judge.
- Name the scorer and choose the Response Data Type:
- Labels: classify each session into predefined values. Configure Allowed Labels (such as TRUE, FALSE) and a Fallback Label (such as UNSURE) for when the judge can't decide or the prompt fails.
- Free-form: Boolean, DateTime, Number, Text, or URL for richer output such as summaries or sentiment.
- Write the evaluation prompt in the Prompt Template Composer and select the AI model to act as judge. Useful merge fields:
{!$Input:AllowedLabels}and{!$Input:FallbackLabel}so the judge knows the valid outputs{!$SalesforceDataAction:getSession.chatTranscript}for session-level scoring{!$SalesforceDataAction:getLastInteraction.userUtterance}and...agentResponsefor turn-level scoring
- Optionally save multiple template versions, set the passing threshold, and save. The scorer is automatically selected for the suite.
Session-level and turn-level tests use different data models, so match the merge fields to the scope. Custom scorers also surface after deployment in Agent Analytics, so the same quality definition can monitor production.
Reading and Acting on Results
Results appear as a grid: one row per test case, one column per scorer.
- Select a cell to open a detail panel with the full JSON and the judge's reasoning.
- Add JSON fields as columns (hover a field → + Column) to surface reasoning in the grid and in CSV downloads.
- Look for recurring failures. They often point to unclear instructions, missing context, or configuration problems.
| Failure | Recommended remediation |
|---|---|
| Subagent Assertion fails | Retest the utterance in Agentforce Builder and review reasoning. Refine subagent descriptions and instructions to guide the correct choice |
| Action Assertion fails | Refine instructions for both the expected action and the subagent, and restrict the incorrect action |
| Low Response Evaluation | Check subagent selection, instructions, actions, and knowledge gaps such as outdated articles |
| Low Instruction Adherence | Refine instructions and set clearer constraints |
| Low prompt template scores | Adjust the template's instructions in Prompt Builder |
Salesforce reminds you that quality scores are guidance, not absolute measures. A low conciseness score might be acceptable if extra context serves your customers. Interpret scores against your goals.
Scenario Practice
| Scenario | Answer |
|---|---|
| Responses return raw JSON from actions instead of sentences | Low Coherence score. Make sure outputs are summarized, for example by turning off Show in conversation so the LLM phrases them |
| The team needs to know whether every response follows the brand's "no promises about delivery dates" policy | Custom scorer with Labels (COMPLIANT / NON_COMPLIANT) and a fallback label |
| A conversation test expects "look up order" and "create return," and the agent also sends a confirmation email | Actions Evaluation passes, because extra actions don't fail it |
| Expected Response column says "Be empathetic and give the tracking number" | Replace it with an actual expected reply, or scores will be inaccurate |
Exam Traps
- Latency is reported, not scored.
- Quality scorers pass at 3 or higher on a 0–5 scale. Assertions are 1 or 0.
- Instruction Adherence for agents is High, Low, or Uncertain, not 0–5.
- Custom scorers are LLM judges defined with prompt templates, not Apex test classes.
Which default scorers run on every turn-level Testing Center test?
A conversation-level test expects the agent to 'look up account' and 'send email.' The agent looks up the account, sends the email, and also creates a case. What is the Actions Evaluation result?
On what scale do Testing Center response quality scorers such as Completeness and Conciseness score agent responses, and what counts as a pass?
A compliance team wants every tested session classified as COMPLIANT or NON_COMPLIANT against a disclosure policy, with UNSURE when the judge can't decide. What should they configure?