6.3 Compare Model Performance Across Jobs

Compare Model Performance Across Jobs

Quick Answer: Compare training jobs in Azure Machine Learning studio (multi-select jobs → compare parameters, metrics, tags) or with MLflow mlflow.search_runs. Sweeps and pipelines produce parent/child runs; filter tags.mlflow.parentRunId. Sort on one primary metric computed on the same validation set. Register only the winner. SDK v2 does not query metrics natively — MLflow does.

Exam AI-300 Domain 2 asks you to compare model performance across jobs. Chapter 5 taught you to log with MLflow. This section is the decision that follows: which run is champion, and what you are allowed to conclude from the numbers.

Studio compare versus MLflow search

Two complementary tools; the exam expects both.

Studio. On the Jobs page, select two or more jobs (or child jobs under an experiment) and open compare. The compare experience (studio documents it as a preview panel you enable) lines up parameters, metrics, and tags so a human can see that run A used lr=0.01 and scored val_auroc=0.81 while run B used lr=0.001 and scored 0.84. Graph compare for pipelines (section 6.2) is the sibling feature: it diffs topology, inputs, and settings when you are debugging reuse, not when you are picking a model.

MLflow. Azure Machine Learning Python SDK v2 does not provide native logging or metric-query APIs. Query the workspace tracking store with the MLflow SDK (mlflow plus azureml-mlflow), pointed at the workspace tracking URI.

import mlflow
runs = mlflow.search_runs(
    experiment_names=["claims-seg"],
    filter_string="metrics.val_auroc > 0.8 and params.backbone = 'resnet50'",
    output_format="pandas",
)
winners = runs.sort_values("metrics.val_auroc", ascending=False)

search_runs returns a pandas DataFrame by default: run identity columns, params.<name>, metrics.<name> (the last logged scalar), and tags. That last-value rule matters: a loss curve collapses to the final point. For the full series, use MlflowClient().get_metric_history(run_id, "val_loss").

Useful filters (AND only; OR is not supported when MLflow is connected to Azure Machine Learning):

• Parameters: params.num_boost_round='100' (operators =, !=, like).
• Metrics: metrics.auc>0.8 (numeric comparators).
• Tags: tags.framework='torch'.
• Attributes: attributes.status = 'Finished', attributes.user_id = '...', attributes.duration, attributes.run_id IN (...).

Status names differ from Studio. Map them before you write a filter:

order_by supports attributes (attributes.start_time DESC, attributes.duration DESC — duration is an Azure Machine Learning convenience). It does not currently support metrics.* / params.* / tags.* expressions against Azure Machine Learning. Sort those in pandas: runs.sort_values("metrics.val_auroc", ascending=False).

Search across experiments with experiment_ids=[...] (an array — useful when two people logged the same model in different experiments) or search_all_experiments=True. If you pass none of those, MLflow searches only the active experiment (mlflow.set_experiment).

Parent and child runs

Hyperparameter sweeps (Chapter 5) and pipelines (section 6.2) mint a parent job plus children. The parent is the orchestrator: it does not usually hold the metric you care about. Each trial or step is a child.

MLflow records the relationship on the child as tag mlflow.parentRunId:

child_runs = mlflow.search_runs(
    filter_string=f"tags.mlflow.parentRunId='{parent_run_id}'"
)
best = child_runs.sort_values("metrics.val_auroc", ascending=False).iloc[0]

Compare siblings that share a parent when you are picking a sweep winner. Comparing a pipeline's prep step metric to a train step metric is meaningless. Comparing two unrelated command jobs is valid only if they logged the same metric on the same holdout.

Primary metric, tags, and lineage

Pick one primary metric before the meeting, the same way AutoML makes you pick one. Accuracy on an imbalanced fraud set is a vanity number; val_auroc or val_pr_auc is a decision number. Direction matters: higher AUROC wins; lower log loss wins. If one job logged accuracy and another logged val_auroc, you do not have a comparison — you have two posters.

Tags are the cheap index you will thank yourself for:

• framework=torch, backbone=resnet50, data=claims-images:2026-08-15
• role=candidate versus role=champion
• validation_set=holdout-2026-08 so a later engineer can filter to comparable runs

Studio lineage on a job shows data assets consumed and models produced. When a candidate looks too good, open lineage first: did it train on the evaluation set? Did it use a different data-asset version than the champion? Lineage plus tags beat screenshot metrics in an audit.

Download artifacts (mlflow.artifacts.download_artifacts) when the metric is not enough — confusion matrices, reliability plots, Responsible AI dashboards (Chapter 7). Load a logged model with mlflow.<flavor>.load_model(f"runs:/{run_id}/{artifact_path}") for a local sanity check before you promote.

Register only the winner

The compare step ends in a promotion decision, not a bulk register. Workflow:

1. Filter to completed children of the sweep or to comparable command jobs.
2. Confirm every remaining run used the same validation asset and split.
3. Sort on the primary metric; break ties with a secondary metric or cost (GPU hours, latency).
4. Spot-check lineage, tags, and a few artifacts on the top run.
5. Register that run's model into the workspace model registry (MLflow model format is the AI-300 default — Chapter 7). Do not register every trial "for completeness." Cluttered registries become the next outage.

The parent sweep job is not the model. Registering the parent by mistake packages an empty or incomplete artifact. Register the winning child run.

Scheduled pipelines from 6.2 should compare (or at least gate on a metric threshold) before they register. A weekly retrain that always registers model:latest without beating the champion is how production quietly regresses.

The validation-set trap

This is the skill-measured bullet's most important caution: a higher number is not a better model if the metric was computed on different data.

Concrete failure modes:

• Job A scored on a 10 percent random split of last week's tiles; job B scored on a hand-picked "easy" folder.
• Job A used azureml:claims-val:3; job B used azureml:claims-val:4 after a labeling correction.
• Job A reported training accuracy; job B reported validation AUROC; someone compared 0.97 to 0.84 and kept A.
• Prep changed (new augmentation) for B, but both jobs still named the metric val_auroc.

Equal metric names do not mean equal meaning. Before you sort, pin the validation data asset version (or an mltable that lists exact files) in both jobs and tag it. If you cannot prove the holdout is the same, you are not comparing model performance — you are comparing experimental setups.

Exam scenario

A sweep of 40 trials trains claims-segmentation models under parent job gentle_oak. Trials log val_auroc and val_loss via MLflow. A scientist opens the parent, sees val_auroc=0.79 on the parent row (a leftover diagnostic), and almost registers the parent. The MLOps engineer instead runs mlflow.search_runs with tags.mlflow.parentRunId='<gentle_oak_id>', keeps attributes.status = 'Finished', sorts metrics.val_auroc descending in pandas, and inspects the top three in studio compare. Two of the three used azureml:claims-val:2; the 0.91 run used a different, smaller val folder. They discard 0.91, register the 0.88 run that shares the agreed holdout, and tag it role=candidate pending Responsible AI checks in Chapter 7.

Common trap

Comparing metrics that were not computed on the same validation set — or comparing training loss to validation AUROC because both are "the number in studio." Sibling traps: registering every sweep child; querying metrics with Azure Machine Learning SDK v2 instead of MLflow; filtering MLflow status with Studio words (Completed instead of Finished); using OR in filter_string; treating the pipeline parent as the model; ranking on search_runs last-value metrics when you actually needed the best checkpoint mid-run from get_metric_history.