13.4 Built-in and Custom Automated Evaluation Workflows

Built-in and Custom Automated Evaluation Workflows

Quick Answer: Use built-in catalog evaluators plus optional custom code, prompt, or endpoint scorers. Run cloud evaluation in Microsoft Foundry for scale and CI; use the local Azure AI Evaluation SDK evaluate() for classic/dev loops. Gate a GitHub Action (microsoft/ai-agent-evals) on thresholds. Continuous evaluation of live traffic is Chapter 14.

The last Domain 4 bullet in this cluster is set up automated evaluation workflows by using built-in and custom evaluation metrics. You now have a mapped dataset, quality metrics, and safety metrics. Automation is how those runs happen on every release candidate without a human clicking the portal.

Built-in catalog versus custom scorers

Built-in evaluators are builtin.coherence, builtin.groundedness, builtin.violence, and the rest of the catalog. They already know their inputs. You add them as testing_criteria of type azure_ai_evaluator with data_mapping and, when required, initialization_parameters.

Custom evaluators exist when the catalog cannot express your rule: JSON schema validity, "must cite a policy ID," brand tone, disclaimer present. Microsoft documents three types:

Register custom evaluators in the project evaluator catalog (portal Evaluation → Evaluator catalog → Custom evaluator, or project_client.beta.evaluators.create_version). Then reference them in a run the same way as builtin.*. Code-based APIs still require a deployment_name in init parameters even though grade() does not call GPT — the orchestration schema expects it. If grade() throws or times out, that item scores 0.0 and is marked an error; write defensive try/except.

Do not write a custom judge that reimplements groundedness poorly. Use built-in groundedness, then a code evaluator that asserts "policy_id" appears in the JSON.

Cloud evaluation versus local SDK

Cloud evaluation is the current Foundry path: openai_client.evals.create (schema + testing criteria) then evals.runs.create (data source). Results live on the project. The portal wizard is the same pipeline with clicks: target (agent / model / dataset / traces), scope (turns vs conversations), data, mapping, criteria, submit. Status values include In Progress, Completed, Partial, Failed. Use cloud when you want scale, RBAC, CI, model/agent targets, synthetic or red-team generation, or App Insights traces. Role on the project: Foundry User (the renamed Azure AI User).

Local evaluation (pip install azure-ai-evaluation, evaluate(data=..., evaluators=..., evaluator_config=column_mapping)) runs on your machine or a pipeline agent. It still needs JSONL and column mapping. You can pass azure_ai_project to log the studio URL. Local is the classic SDK loop and still appears in hubs/prompt-flow shops. AI-300 expects you to know both: cloud for Foundry GenAIOps, local for a dev-box or an air-gapped debug. Cloud is what Microsoft tells you to use for most pre-deployment and CI scenarios so you are not babysitting compute.

Data sources you should recognize for automation:

• Versioned JSONL/CSV (file_id) — golden set in CI.
• Target completions — hit azure_ai_agent or azure_ai_model with {{item.query}}.
• Agent response IDs — file_content only.
• Traces — production-adjacent; more Chapter 14.
• Synthetic / red team — bootstrap or adversarial, not the only gate.

GitHub Action as a quality gate

Microsoft publishes microsoft/ai-agent-evals (documented as v3-beta) to evaluate Foundry agents in GitHub Actions. Inputs: azure-ai-project-endpoint, deployment-name (judge), data-path (JSON with name, evaluators, data rows), agent-ids as name:version. Optional baseline-agent-id for pairwise tests with confidence intervals. Authenticate with azure/login and OIDC (id-token: write). The action invokes the agent, runs catalog (and custom) evaluators, and writes a summary report.

Treat the job as a gate: fail the workflow if groundedness mean is below 3, if violence defect rate exceeds policy, or if the pairwise test says the new agent is worse than baseline. That is "automated evaluation workflow" in exam language.

Microsoft’s own tip: do not run evaluation on every commit — judge tokens and agent calls cost money. Typical triggers: pull request to main, a workflow_dispatch before a release, or a nightly on the golden set. Path filters so a README change does not spend GPT.

You can also call the cloud SDK from a workflow step (Azure CLI login, then Python evals.runs.create) if you need dataset evaluation without the agent-evals action. Same rule: threshold → exit code.

What this chapter is not

Continuous evaluation samples production agent responses on a schedule and feeds the monitoring dashboard. That is Chapter 14 (observability: continuous monitoring, latency, tokens, tracing). A cloud run against claims-eval/3 in CI is pre-deployment. Mixing them on the exam — "we turned on continuous eval so we can skip the golden set" — is the trap.

A complete automated loop for this chapter looks like: Git tracks the JSONL golden set and custom grade() source → PR or release workflow logs into Azure → cloud evaluation or ai-agent-evals runs built-in quality + safety + custom schema check → job fails on threshold → you do not deploy the agent version. After it is in production, Chapter 14 watches live scores.

Exam scenario

Contoso versions agent claims-triage:7 in Git. The workflow on pull request to main checks out the repo, azure/login@v2 with federated credentials, and runs microsoft/ai-agent-evals@v3-beta with agent-ids: claims-triage:7, data file listing builtin.groundedness, builtin.relevance, builtin.violence, builtin.hate_unfairness, and custom policy_json_valid. Pass thresholds: groundedness ≥ 3, violence defect rate 0 on the adversarial slice, custom code score ≥ 0.99. The previous version is baseline-agent-id. A prompt-only change that lifts fluency but drops groundedness fails the Action. Nobody clicks Submit in the portal on Friday night as the release process.

Common trap

Calling the playground "CI." A second trap is a custom prompt evaluator that duplicates groundedness with a vaguer rubric. A third is a code evaluator that calls out to the public internet (sandbox has no network) or exceeds 2 minutes. A fourth is running ai-agent-evals on every push of a markdown fix. A fifth is treating a successful local notebook evaluate() on 8 rows as the production gate, with no thresholds in GitHub. A sixth is configuring continuous production sampling and skipping this chapter’s pre-deploy run.