1.12 Identifying the Best Run with the MLflow Client API

1.12 Identifying the Best Run with the MLflow Client API

After a sweep of dozens or hundreds of trials, the best run has to be identified by code, not by eye — a promotion job cannot scroll the UI. MLflow exposes two search entry points, and the exam expects you to know both the syntax and the difference.

Two APIs, Two Return Types

import mlflow
from mlflow.tracking import MlflowClient
from mlflow.entities import ViewType

experiment = mlflow.get_experiment_by_name("/Shared/Experiments/churn_prediction")

# A) Fluent API -> pandas DataFrame, convenient for analysis in a notebook
runs_df = mlflow.search_runs(
    experiment_ids=[experiment.experiment_id],
    filter_string="metrics.val_f1 > 0.80 AND params.model_family = 'xgboost'",
    order_by=["metrics.val_f1 DESC"],
    max_results=1,
)
best_run_id = runs_df.iloc[0]["run_id"]

# B) Client API -> list of Run objects, preferred inside jobs and pipelines
client = MlflowClient()
best = client.search_runs(
    experiment_ids=[experiment.experiment_id],
    filter_string="metrics.val_f1 > 0.80",
    run_view_type=ViewType.ACTIVE_ONLY,
    order_by=["metrics.val_f1 DESC"],
    max_results=1,
)[0]

print(best.info.run_id, best.data.metrics["val_f1"], best.data.params["max_depth"])

Use the fluent version when you want a DataFrame to plot or aggregate. Use the client version in automation: it avoids materialising a DataFrame, exposes the full Run entity (info, data, inputs), and is the same object the rest of the client API consumes.

---

Filter-String Grammar

Every term must be prefixed with the entity it refers to:

Rules that trip people up:

• Parameters are stored as strings, so the comparison value must be quoted: params.max_depth = '6', never params.max_depth = 6.
• Conjunction is AND in upper case. OR is not supported in MLflow filter strings — a filter that needs a disjunction must be split into two searches.
• Metric comparisons use the metric's latest logged value for that key.

Comparators by Field Type

# Sweep runs that recorded an environment tag and cleared the F1 bar
client.search_runs(
    experiment_ids=[experiment.experiment_id],
    filter_string=(
        "tags.mlflow.runName ILIKE 'sweep%' "
        "AND tags.environment IS NOT NULL "
        "AND metrics.val_f1 > 0.8"
    ),
    order_by=["metrics.val_f1 DESC"],
)

Two more grammar rules the exam can lean on:

• Backtick awkward key names. A key containing a hyphen or starting with a digit must be wrapped: metrics.`cross-entropy-loss` < 0.5. Written bare, it is a parse error rather than an empty result.
• IN is narrow. It applies to attributes.run_id and the datasets.* fields only, and its members take single quotes. There is no IN over an arbitrary param.

Passing search_all_experiments=True to mlflow.search_runs searches every experiment the caller can read, which is convenient for a one-off audit and expensive as a habit — prefer explicit experiment_ids in scheduled jobs.

Ordering, Limits, and View Types

• order_by=["metrics.val_f1 DESC"] sorts server-side. Combined with max_results=1, only one row crosses the wire — the correct pattern for an experiment with thousands of runs.
• Multiple keys break ties: order_by=["metrics.val_f1 DESC", "attributes.start_time ASC"] prefers the earliest run among equal scores, which makes promotion deterministic.
• run_view_type defaults to ACTIVE_ONLY. Passing ViewType.ALL includes deleted runs, and ViewType.DELETED_ONLY returns only them. If a search keeps returning a run someone deleted, this parameter is why.

From Best Run to Registered Model

model_uri = f"runs:/{best.info.run_id}/model"
version = mlflow.register_model(model_uri=model_uri,
                                name="prod_catalog.ml.churn_classifier")

runs:/<run_id>/<artifact_path> is the URI form for an artifact that has not been registered yet. Once registered it is addressed as models:/catalog.schema.name/<version> or by alias — see Sections 1.14 and 1.15.

Finding the Trials of One Sweep

Hyperopt and AutoML write each trial as a child run whose mlflow.parentRunId tag holds the parent's run ID. A search across the experiment returns parents and children together, so ranking purely by the trial metric can surface a parent run whose metric was aggregated rather than measured. Constrain the search to one sweep:

children = client.search_runs(
    experiment_ids=[experiment.experiment_id],
    filter_string=f"tags.mlflow.parentRunId = '{parent_run_id}'",
    order_by=["metrics.val_f1 DESC"],
    max_results=1,
)

Exam tip: when a nested Hyperopt sweep is involved, remember that the parent run holds the summary while each trial is a child run. Searching for the best trial means searching the experiment's runs and ordering by the trial metric, not reading the parent.

Ordering, Ties, and Pagination

order_by accepts several keys and applies them in sequence, which is how ties are broken deterministically:

runs = client.search_runs(
    experiment_ids=[experiment.experiment_id],
    filter_string="attributes.status = 'FINISHED'",
    order_by=["metrics.val_f1 DESC", "attributes.start_time ASC"],
    max_results=1,
)

Without a tiebreaker, two runs holding identical metrics come back in an unspecified order, so re-running the same selection logic can register a different model version from one day to the next. Two further details matter in practice. search_runs paginates, so a result set larger than the page size requires following the returned page token rather than assuming everything arrived in one call. And failed or killed runs still appear in the results unless the filter excludes them — a run that crashed after logging one epoch's metric can otherwise win a DESC ordering outright, which is why attributes.status = 'FINISHED' belongs in the filter whenever a sweep might have partial runs in it.