13.4 Built-in and Custom Automated Evaluation Workflows
Key Takeaways
Domain 4 ends this cluster with automated evaluation workflows using built-in and custom metrics. Built-in evaluators live in the project evaluator catalog (builtin.*). Custom evaluators are code-based (grade() returns 0.0–1.0), prompt-based (LLM judge with ordinal, continuous, or binary result), or endpoint-based (your HTTP scorer).
Cloud evaluation in Microsoft Foundry (evals.create then evals.runs.create, or the portal wizard) is the default for scale and CI. Local azure-ai-evaluation evaluate() is for classic/dev-box spotting and can log to the project. Cloud supports dataset, model/agent targets, traces, synthetic, and red team sources.
microsoft/ai-agent-evals GitHub Action (v3-beta) runs catalog evaluators against Foundry agents from a data file. Authenticate with Azure Login (OIDC). Do not run the full suite on every commit. Fail the job when scores miss your thresholds.
Code-based custom evaluators sandbox at under 256 KB, two minutes, no network. Prompt-based custom judges need deployment_name and threshold. Endpoint-based scorers need a project connection and a 30-second JSON contract.
This chapter is eval design and the pre-production/CI run. Continuous evaluation of sampled production traffic and the monitoring dashboard are Chapter 14 — do not treat a one-shot cloud run as production observability.
Built-in and Custom Automated Evaluation Workflows
Quick Answer: Use built-in catalog evaluators plus optional custom code, prompt, or endpoint scorers. Run cloud evaluation in Microsoft Foundry for scale and CI; use the local Azure AI Evaluation SDK
evaluate()for classic/dev loops. Gate a GitHub Action (microsoft/ai-agent-evals) on thresholds. Continuous evaluation of live traffic is Chapter 14.
The last Domain 4 bullet in this cluster is set up automated evaluation workflows by using built-in and custom evaluation metrics. You now have a mapped dataset, quality metrics, and safety metrics. Automation is how those runs happen on every release candidate without a human clicking the portal.
Built-in catalog versus custom scorers
Built-in evaluators are builtin.coherence, builtin.groundedness, builtin.violence, and the rest of the catalog. They already know their inputs. You add them as testing_criteria of type azure_ai_evaluator with data_mapping and, when required, initialization_parameters.
Custom evaluators exist when the catalog cannot express your rule: JSON schema validity, "must cite a policy ID," brand tone, disclaimer present. Microsoft documents three types:
| Type | How it scores | Best for | Contract |
|---|---|---|---|
| Code-based | Python grade(sample, item) -> float in 0.0–1.0 | Format, length, keyword, regex, deterministic checks | Sandbox: under 256 KB, 2 min, no network, 2 GB RAM; packages such as numpy, pandas, jsonschema, rouge-score |
| Prompt-based | Judge prompt with {{response}} (and other fields) | Tone, rubric, semantic rules the catalog lacks | JSON {result, reason}; ordinal, continuous, or binary; needs deployment_name and threshold |
| Endpoint-based | POST to your HTTP API | Proprietary models, network, logic over the sandbox limits | JSON with score, status, optional passed; 30 s timeout; ApiKey or Entra ID connection |
Register custom evaluators in the project evaluator catalog (portal Evaluation → Evaluator catalog → Custom evaluator, or project_client.beta.evaluators.create_version). Then reference them in a run the same way as builtin.*. Code-based APIs still require a deployment_name in init parameters even though grade() does not call GPT — the orchestration schema expects it. If grade() throws or times out, that item scores 0.0 and is marked an error; write defensive try/except.
Do not write a custom judge that reimplements groundedness poorly. Use built-in groundedness, then a code evaluator that asserts "policy_id" appears in the JSON.
Cloud evaluation versus local SDK
Cloud evaluation is the current Foundry path: openai_client.evals.create (schema + testing criteria) then evals.runs.create (data source). Results live on the project. The portal wizard is the same pipeline with clicks: target (agent / model / dataset / traces), scope (turns vs conversations), data, mapping, criteria, submit. Status values include In Progress, Completed, Partial, Failed. Use cloud when you want scale, RBAC, CI, model/agent targets, synthetic or red-team generation, or App Insights traces. Role on the project: Foundry User (the renamed Azure AI User).
Local evaluation (pip install azure-ai-evaluation, evaluate(data=..., evaluators=..., evaluator_config=column_mapping)) runs on your machine or a pipeline agent. It still needs JSONL and column mapping. You can pass azure_ai_project to log the studio URL. Local is the classic SDK loop and still appears in hubs/prompt-flow shops. AI-300 expects you to know both: cloud for Foundry GenAIOps, local for a dev-box or an air-gapped debug. Cloud is what Microsoft tells you to use for most pre-deployment and CI scenarios so you are not babysitting compute.
Data sources you should recognize for automation:
- Versioned JSONL/CSV (
file_id) — golden set in CI. - Target completions — hit
azure_ai_agentorazure_ai_modelwith{{item.query}}. - Agent response IDs —
file_contentonly. - Traces — production-adjacent; more Chapter 14.
- Synthetic / red team — bootstrap or adversarial, not the only gate.
GitHub Action as a quality gate
Microsoft publishes microsoft/ai-agent-evals (documented as v3-beta) to evaluate Foundry agents in GitHub Actions. Inputs: azure-ai-project-endpoint, deployment-name (judge), data-path (JSON with name, evaluators, data rows), agent-ids as name:version. Optional baseline-agent-id for pairwise tests with confidence intervals. Authenticate with azure/login and OIDC (id-token: write). The action invokes the agent, runs catalog (and custom) evaluators, and writes a summary report.
Treat the job as a gate: fail the workflow if groundedness mean is below 3, if violence defect rate exceeds policy, or if the pairwise test says the new agent is worse than baseline. That is "automated evaluation workflow" in exam language.
Microsoft’s own tip: do not run evaluation on every commit — judge tokens and agent calls cost money. Typical triggers: pull request to main, a workflow_dispatch before a release, or a nightly on the golden set. Path filters so a README change does not spend GPT.
You can also call the cloud SDK from a workflow step (Azure CLI login, then Python evals.runs.create) if you need dataset evaluation without the agent-evals action. Same rule: threshold → exit code.
What this chapter is not
Continuous evaluation samples production agent responses on a schedule and feeds the monitoring dashboard. That is Chapter 14 (observability: continuous monitoring, latency, tokens, tracing). A cloud run against claims-eval/3 in CI is pre-deployment. Mixing them on the exam — "we turned on continuous eval so we can skip the golden set" — is the trap.
A complete automated loop for this chapter looks like: Git tracks the JSONL golden set and custom grade() source → PR or release workflow logs into Azure → cloud evaluation or ai-agent-evals runs built-in quality + safety + custom schema check → job fails on threshold → you do not deploy the agent version. After it is in production, Chapter 14 watches live scores.
Exam scenario
Contoso versions agent claims-triage:7 in Git. The workflow on pull request to main checks out the repo, azure/login@v2 with federated credentials, and runs microsoft/ai-agent-evals@v3-beta with agent-ids: claims-triage:7, data file listing builtin.groundedness, builtin.relevance, builtin.violence, builtin.hate_unfairness, and custom policy_json_valid. Pass thresholds: groundedness ≥ 3, violence defect rate 0 on the adversarial slice, custom code score ≥ 0.99. The previous version is baseline-agent-id. A prompt-only change that lifts fluency but drops groundedness fails the Action. Nobody clicks Submit in the portal on Friday night as the release process.
Common trap
Calling the playground "CI." A second trap is a custom prompt evaluator that duplicates groundedness with a vaguer rubric. A third is a code evaluator that calls out to the public internet (sandbox has no network) or exceeds 2 minutes. A fourth is running ai-agent-evals on every push of a markdown fix. A fifth is treating a successful local notebook evaluate() on 8 rows as the production gate, with no thresholds in GitHub. A sixth is configuring continuous production sampling and skipping this chapter’s pre-deploy run.
An MLOps engineer must evaluate 400 golden-set rows against a Foundry agent on every release candidate and keep results in the project. Which execution mode matches Microsoft’s current guidance?
Cloud evaluation in the Foundry project (portal wizard or evals.create / evals.runs.create), optionally triggered from CI — not a one-off local notebook as the only gate.
Only the Azure Machine Learning RAI dashboard gather pipeline, because it logs to the same studio.
Paste 400 rows into the agent playground and screenshot fluency.
Continuous production sampling only, skipping pre-deployment datasets.
You need a metric that the response JSON contains a policy_id matching ^POL-[0-9]{5}$, and a metric that a judge rates brand tone 1–5. Which custom evaluator types fit?
Endpoint-based for the regex (because regex needs the public internet) and code-based for tone (because grade() is an LLM).
Code-based grade() returning 0.0–1.0 for the schema/regex check, and a prompt-based custom evaluator with ordinal 1–5 for tone; do not reimplement built-in groundedness as a vague prompt.
Built-in fluency for the regex and built-in violence for tone.
Only Groundedness Pro, because binary Content Safety scores JSON schema.
How should a GitHub Actions workflow use Microsoft’s agent evaluation action as a deploy gate?
Run it on every documentation commit with no thresholds so the badge stays green.
Store API keys in the data JSONL query field so the action can call Azure OpenAI as the user.
Skip azure/login because the action uses anonymous Foundry access.
On pull request or release (not every commit), azure/login with OIDC, microsoft/ai-agent-evals with project endpoint, judge deployment, data-path, and agent-ids, then fail the job when quality or safety scores miss policy thresholds.
A teammate says that enabling Foundry continuous evaluation of sampled production traces means the team can delete the golden-set CI job. What is the AI-300 distinction?
They are identical; Microsoft uses the names interchangeably.
Continuous evaluation replaces safety evaluators but not groundedness.
This chapter’s automated workflow is the pre-production/CI design-and-run against a mapped dataset or agent target; continuous evaluation of live traffic is Chapter 14 observability and does not retire the golden-set gate.
Golden-set CI is only for Azure Machine Learning command jobs, never for Foundry agents.
Sections you finish are checked off in the contents.