5.4 Automate Hyperparameter Tuning

Automate Hyperparameter Tuning

Quick Answer: A sweep job (type: sweep) launches child jobs of your training script across a search space. Sample with random, grid, or Bayesian. Stop losers with Bandit, median stopping, or truncation selection. Optimize a primary metric (Maximize or Minimize) that the script logs with MLflow under the exact same name. Sweep ≠ AutoML: you bring the script; AutoML brings the algorithms.

Domain 2 asks you to automate hyperparameter tuning. In SDK/CLI v2 that product is the SweepJob, not the v1 HyperDrive object name. You start from a command job (section 5.3), replace some inputs with parameter expressions, and call .sweep() — or you author YAML with type: sweep.

Search space: discrete versus continuous

Hyperparameters are the knobs the optimizer does not learn as weights: learning rate, tree depth, batch size, number of layers. You declare a search space of expressions:

• Discrete / choice — a list, a range, or comma-separated values (batch_size: [16, 32, 64]).
• Quantized discrete — QUniform, QLogUniform, QNormal, QLogNormal (round a draw to a step q).
• Continuous — Uniform / LogUniform / Normal / LogNormal over a real interval.

CLI YAML:

search_space:
  learning_rate:
    type: uniform
    min_value: 0.01
    max_value: 0.2
  boosting:
    type: choice
    values: [gbdt, dart]

SDK v2 uses Uniform, Choice, and friends from azure.ai.ml.sweep, then command_job(learning_rate=Uniform(...), boosting=Choice(...)).sweep(...).

Every trial restarts training from scratch, including data loaders. Push heavy featurization into a pipeline step or a data asset so each child is not recopying a lake.

Sampling algorithms

Grid on a uniform learning rate is an exam fail: grid cannot enumerate an interval. Bayesian with a loguniform (or other unsupported distribution) is the other fail. Bayesian also converges better with fewer concurrent trials, because new samples need finished predecessors. Random is the safe default when the space is mixed or you want aggressive early stopping.

Objective, logging, and the best child

The sweep’s objective is:

• primary_metric — a string that must equal the metric name in mlflow.log_metric inside the trial script.
• goal — Maximize or Minimize.

If the script logs val_auc and the sweep asks for AUC_weighted, the service never sees a usable score and early termination / best-child selection is nonsense. Log often enough for policies to have intervals (each log_metric of that name is one interval). After the parent finishes, download the best trial outputs (ml_client.jobs.download(sweep_name, output_name="model") or az ml job download --name <sweep> --output-name model). Studio charts — metrics over time, parallel coordinates, 2D/3D scatter — are how you explain why that child won.

Early termination and limits

Policies compare children on the primary metric and cancel losers:

1. Bandit — slack around the current best (slack_factor ratio or slack_amount absolute). Example: maximize, best is 0.8, slack_factor 0.2 → children below about 0.67 can die.
2. Median stopping — stop if the trial’s best is worse than the median of running averages across jobs.
3. Truncation selection — at each evaluation, cancel the worst truncation_percentage (1–99). Optional exclude_finished_jobs.
4. None (default) — every trial runs to completion.

Shared knobs: evaluation_interval (apply every N metric reports; docs default 0 if you omit it, so set it on purpose) and delay_evaluation (give every configuration a minimum number of intervals). A conservative starter that Microsoft cites as roughly 25–35% savings without hurting the primary metric is median stopping with evaluation_interval=1 and delay_evaluation=5. Bandit with tight slack, or truncation with a large percentage, is more aggressive.

Resource limits on the parent:

• max_total_trials — 1 to 1000.
• max_concurrent_trials — 1 to 1000; cannot exceed what the compute actually has.
• timeout — seconds for the entire sweep.
• trial_timeout — seconds for one child.

If both max_total_trials and timeout are set, the sweep ends on whichever hits first.

Sweep versus AutoML

Keep this contrast memorized:

• AutoML — you choose a task and data. The service searches algorithms (and hyperparameters, and featurization, and ensembles).
• Sweep — you already chose the algorithm (your python train.py). The service searches that script’s hyperparameters.

A LightGBM sweep is the right answer when the team standardized on LightGBM. An AutoML classification job is the right answer when you still need to know whether LightGBM, XGBoost, or a stack ensemble should win. You can even sweep inside a pipeline by calling .sweep() on a command component; that is still “you brought the script.”

YAML sketch:

$schema: https://azuremlschemas.azureedge.net/latest/sweepJob.schema.json
type: sweep
trial: ...
search_space:
  learning_rate:
    type: uniform
    min_value: 0.01
    max_value: 0.2
sampling_algorithm: random
objective:
  goal: minimize
  primary_metric: test-multi_logloss
limits:
  max_total_trials: 20
  max_concurrent_trials: 4
  timeout: 7200
early_termination:
  type: median_stopping
  evaluation_interval: 1
  delay_evaluation: 5
compute: azureml:cpu-cluster

SDK equivalent: sweep_job = command_job_for_sweep.sweep(compute="cpu-cluster", sampling_algorithm="random", primary_metric="test-multi_logloss", goal="Minimize"), then set_limits(...) and early_termination = MedianStoppingPolicy(...).

Exam scenario

A claims model will ship as LightGBM. The engineer wraps src/train.py (already logging mlflow.log_metric("val_logloss", ...) ) as a command job, replaces learning_rate with Uniform(0.01, 0.2) and num_leaves with Choice([31, 63, 127]), and sweeps with Bayesian sampling, goal="Minimize", primary_metric="val_logloss", max_total_trials=40, max_concurrent_trials=4 on an 8-node cluster (leaving headroom), and median stopping delayed 5 evaluations. They download the best child’s MLflow model. They do not open AutoML for this run because the algorithm is already chosen.

Common trap

The classic failure is a name mismatch between primary_metric and log_metric. Next is using grid on continuous parameters, or Bayesian on unsupported distributions. Next is setting max_concurrent_trials larger than cluster nodes and wondering why Bayesian looks random. Next is treating a sweep as AutoML (or vice versa). Last is logging the metric only once at the end — Bandit and median stopping never get an interval to act on, so you pay for every doomed epoch.