13.1 Test Datasets and Data Mapping for Evaluation
Key Takeaways
- Exam AI-300 Domain 4 (10–15%) starts with create test datasets and data mapping for comprehensive model evaluation. A row is not 'some CSV of chats' — it is query, optional context, optional ground_truth, and response, mapped to the fields each evaluator requires.
- Microsoft Foundry portal field mapping and SDK data_mapping use {{item.field}} for dataset columns and {{sample.output_text}} (or {{sample.output_items}}) when a model or agent target generates the response at run time. If a required field is Unassigned, that evaluator fails.
- Upload JSONL (turn-level or conversation messages) or CSV (turn-level only) as a versioned project dataset. Column names do not have to be query; map {{item.question}} when the file uses question.
- Coverage beats raw size: intents, languages, empty-retrieval and adversarial rows, plus a frozen golden set and a sampled production set. Judge-model cost scales with rows × evaluators, so size the batch to quota, not to 'as many as the lake has.'
- For RAG, concatenate retrieved chunks into one context string with a separator such as a blank line. Groundedness cannot score a response against documents it never sees.
Test Datasets and Data Mapping for Evaluation
Quick Answer: A Microsoft Foundry evaluation dataset is a versioned JSONL or CSV of query, optional context, optional ground_truth, and response. Data mapping (
{{item.field}}in the SDK, field mapping in the portal) binds those columns to each evaluator. Cover intents, languages, and adversarial rows. Keep a frozen golden set and a production sample. Size the set against judge-model cost, not against how many logs you can dump.
Domain 4 of Exam AI-300 — Implement generative AI quality assurance and observability (10–15%) — opens with create test datasets and data mapping for comprehensive model evaluation. Domain 3 already versioned prompts and agents. This chapter is how you prove a change is better before it ships. Chapter 14 covers continuous monitoring of live traffic. This section is the offline design of the test set those evaluators will score.
Why an evaluation dataset is not production logs
Playground eyeballing of three happy-path questions is not evaluation. Neither is pasting last week's Application Insights dump into a spreadsheet and calling it a test. An evaluation dataset is a controlled sample with a schema the evaluators understand, labels (or at least known intents), and a version you can rerun after a prompt or model change.
Microsoft Foundry accepts:
- Existing dataset — you already have queries and, often, responses. Score them as a dataset evaluation without calling the model again.
- Model or agent target — you have queries (and maybe context / ground truth). Foundry generates responses at evaluation time, then scores them. Map the live output with
{{sample.output_text}}. - Synthetic queries (preview) — you have almost nothing; the service generates queries from a prompt and optional seed file, then hits a target.
- Traces / existing conversations — later in the flywheel; still need mapping so query and response come out of the span or message array.
JSONL is the workhorse: one JSON object per line, turn-level rows or a messages array for conversation-level runs. CSV is turn-level only. Upload with project_client.datasets.upload_file (name + version) so CI reuses file_id instead of attaching a one-off blob. Inline file_content is for tiny experiments and for agent response-ID evaluation, which does not accept file_id.
The four columns evaluators actually consume
Built-in evaluators do not guess your business names. They look for query, response, context, and ground_truth (plus agent extras such as tool_calls and tool_definitions, and conversation messages).
| Column | What it is | Who needs it |
|---|---|---|
query | The user (or test) prompt | Relevance, coherence, safety, similarity, retrieval; required in portal single-turn mapping |
response | The model or agent output | Almost every quality and safety evaluator |
context | Retrieved grounding text (chunks concatenated) | Groundedness, Groundedness Pro, retrieval, ungrounded attributes |
ground_truth | Human or expert expected answer | Similarity, F1, BLEU/GLEU/ROUGE/METEOR, response completeness |
A query is the stimulus. A response is what you are scoring. Context is the retrieved evidence the RAG or tool path supplied — not the entire knowledge base and not the system prompt. Ground truth is the answer a reviewer would accept, used for overlap and completeness metrics, not as a substitute for context.
For multi-chunk retrieval, Microsoft documents concatenating chunks into one string with a separator such as a blank line between passages. Do not leave context as an array of objects unless you have written a custom evaluator that knows that shape. For conversation-level runs the portal maps messages (chat format) and, for agents, tool_definitions.
Data mapping: teaching the evaluator which column is which
The portal Evaluation wizard (Step 5 when you use existing data) auto-maps columns that already match the expected names. If a field shows Unassigned, you pick the dataset column from a dropdown. Required fields are marked with an asterisk. Evaluators fail if required fields stay unassigned. That is the exam trap in UI clothing.
The cloud SDK is explicit: always include data_mapping with the inputs each evaluator lists. Syntax:
{{item.field_name}}— a column on the dataset row (or item schema).{{sample.output_text}}— text the model or agent target just generated.{{sample.output_items}}— structured agent output including tool calls.
If your JSONL uses question and answer because that is how the product team exports tickets, you do not rename the lake. You map query to {{item.question}} and response to {{item.answer}}. Local classic SDK evaluate() uses a similar idea as column_mapping with ${data.queries} style placeholders. New cloud runs use the {{item.*}} / {{sample.*}} pair. Either way, mapping is the contract, not the filename.
When the run generates responses, do not map response to a stale {{item.response}} column that is empty or from last week's model. Map it to {{sample.output_text}} and set include_sample_schema on the data source config so the sample fields exist.
Coverage: intents, languages, and adversarial rows
A 2,000-row file of near-duplicate "what is the refund window?" questions is not comprehensive. Design coverage on purpose:
- Intents / skills — every tool, every policy chapter, every "I don't know" path, and the top support tickets. Tag a column such as
intentfor slicing even if evaluators ignore it. - Languages and locales — if the agent serves
en-USandes-MX, both belong in the set. Coherence and fluency currently document English-language scoring; do not pretend an English judge is a multilingual certification. - Retrieval extremes — empty context, contradictory chunks, and the correct chunk buried at position k. Groundedness on empty context should not invent policy.
- Adversarial / red-team — jailbreak phrasing, indirect injection hidden in a retrieved document, copyrighted-lyric bait, and self-harm probes. These rows exist so risk and safety evaluators have something to fail. Keep them in a restricted dataset version; they are not playground demos.
- Schema / tool rows — missing
plan_code, malformed JSON, and "call the refund tool" cases if you will score tool accuracy later.
Synthetic generation (portal Synthetic data, or azure_ai_synthetic_data_gen_preview) is a bootstrap when you have no queries, not a replacement for a reviewed golden set. Seed it with a document or prompt that describes your domain. Red-team evaluation (azure_ai_red_team) generates adversarial prompts on purpose; that is a safety dataset, not a fluency dataset.
Golden set versus production sample, and size versus cost
A golden set is a frozen, human-reviewed batch — often 50–200 rows for an associate-scale product, more if you have labeled qrels for document retrieval. You change it only in a pull request. Rerunning the same golden set after a prompt variant is how Chapter 12's comparison becomes a number instead of a vibe.
A production sample is a time-boxed draw from real queries (or traces). It catches drift the golden set never imagined. It is not the golden set: labels may be missing, PII may be present, and you must sample rather than score the entire lake. Intelligent sampling of traces (Chapter 14) is the production-side cousin; here you still curate a file.
Size versus cost: AI-assisted quality evaluators call a judge deployment once per row per metric. Microsoft's portal troubleshooting says the judge counts against Azure OpenAI quota, that you should shrink the set or use a cheaper judge such as gpt-4.1-mini / gpt-5-mini when quota binds, and that GitHub's evaluation action should not run on every commit. A 5,000-row set × four LLM judges is a bill and a timeout, not a badge. Start with a few dozen rows to validate mapping, then scale the golden set to the intents you must cover.
Version the file (eval-claims-golden/3). Record which agent version and prompt file the responses came from. Dataset evaluation of precomputed outputs is cheaper than regenerating against a target every run; regenerate when the question is "does this agent version still pass?"
Exam scenario
Contoso Claims has a Foundry agent over an Azure AI Search index. The product manager emails a 40,000-row export with columns ticket_id, member_question, kb_snippets, agent_reply, and adjuster_answer. You upload JSONL version claims-eval/1. In the portal you map query → member_question, context → kb_snippets (chunks already joined), response → agent_reply, ground_truth → adjuster_answer. You add 30 adversarial rows and 20 Spanish intents in claims-eval/2. You freeze 120 reviewed rows as the golden set and sample 80 production tickets per week into a separate version. You do not map groundedness until kb_snippets is populated.
Common trap
Leaving query or response Unassigned because the CSV header says question / output, then blaming Foundry when every evaluator errors. A second trap is scoring groundedness without context (or with the system prompt stuffed in as context). A third is treating the golden set as a dump of production logs — PII, no intents, no adversarial cases, and a cost bomb for the judge model. A fourth is mapping {{item.response}} on a target run that is supposed to generate {{sample.output_text}}.
You upload a claims JSONL whose columns are member_question, kb_snippets, agent_reply, and adjuster_answer. You will run groundedness, relevance, and F1. What must you do so the evaluators receive the fields they require?
Contoso wants both a regression suite they can rerun after every prompt pull request and a check that live traffic has not drifted. How should those two datasets differ?
A teammate wants to score 8,000 near-duplicate English refund questions with four LLM-as-judge quality evaluators on every GitHub push. What is the evaluation-dataset design you should insist on instead?
You configure a cloud evaluation whose target is a Foundry agent. The JSONL has only query and context. Where should the response field for coherence come from?