5.2 Use Automated Machine Learning to Explore Optimal Models

Use Automated Machine Learning to Explore Optimal Models

Quick Answer: Automated machine learning (AutoML) tries many algorithms and parameters against a primary metric you choose. Tasks cover classification, regression, forecasting, computer vision, and NLP. Featurization becomes part of the model. Voting and stacking ensembles are on by default. Stop with exit criteria. Use studio (no-code) or SDK/CLI v2. AutoML is a search, not a promise it beats a well-tuned custom model.

Domain 2 of Exam AI-300 asks you to use automated machine learning to explore optimal models. The exam word is explore. AutoML is how an MLOps engineer gets a high-quality baseline — and often a production candidate — without personally iterating every estimator. It does not retire custom training scripts or sweep jobs (section 5.4).

How AutoML works

You identify the problem type, choose studio or SDK/CLI v2, point at labeled data, set the primary metric and limits, and submit a job. Azure Machine Learning then launches many child trials in parallel. Each trial pairs an algorithm with feature handling and hyperparameters. Each trial gets a score on your metric. The job stops when it hits exit criteria you defined (or when it stops making progress if you defined none).

The training job produces a serialized model (commonly a Python .pkl) that includes both the estimator and the preprocessing. Logged metrics for every child appear on the parent job. You inspect them in studio the same way you inspect any experiment: Jobs, then the AutoML parent, then child runs.

A minimal CLI v2 job looks like this:

$schema: https://azuremlsdk2.blob.core.windows.net/preview/0.0.1/autoMLJob.schema.json
type: automl
task: classification
training_data:
  path: ./train_data
  type: mltable
compute: azureml:cpu-cluster
primary_metric: AUC_weighted
target_column_name: is_fraud
limits:
  timeout_minutes: 180
  max_trials: 40
  enable_early_termination: true

Submit with az ml job create --file automl-classification-job.yml. SDK v2 uses factory helpers such as automl.classification(...) plus set_limits and set_training, then ml_client.jobs.create_or_update.

Task types you must name on the exam

Pick the task from the business problem, not from the library you like.

1. Classification — categorical label. Fraud, churn, image-or-text class. Featurizers include deep neural network text featurizers for tabular-plus-text problems.
2. Regression — numeric target. Price, delay minutes, risk score.
3. Forecasting — time series treated as multivariate regression: lagged values are pivoted into features along with extra predictors. Advanced knobs include holiday detection, Auto-ARIMA, Prophet, ForecastTCN, many-models grouping, rolling-origin cross validation, lags, and rolling-window aggregates.
4. Computer vision — authored from the Python SDK (studio still shows the resulting jobs). Tasks are multi-class image classification (one label per image), multi-label image classification (several labels per image), object detection (bounding boxes), and instance segmentation (pixel-level polygons). It integrates with Azure Machine Learning data labeling.
5. Natural language processing (NLP) — text classification and named entity recognition (NER). End-to-end deep networks on pretrained BERT models, 104-language multilingual support, distributed training with Horovod, and the same labeling integration.

Tabular AutoML v2 wants MLTable training data (a folder with an MLTable file plus the CSV or Parquet). Vision and NLP jobs use the labeled datasets those task guides describe. You are not required to memorize every supported estimator; you are required to know that you do not pick the algorithm unless you allow-list or block-list it (allowed_training_algorithms / blocked_training_algorithms).

Featurization is part of the model

Featurization is scaling, normalization, missing-value handling, and converting text to numbers. AutoML does this automatically (mode: auto), you can turn it off, or you can supply custom transformers. Whatever ran during training is serialized into the model. At scoring time the same steps run on new rows. That is why a raw CSV scored against an AutoML endpoint does not need a separate preprocessing service — and why you must not “simplify” the pickle by stripping preprocessors.

Validation is also automatic if you omit validation_data:

• Training set larger than 20,000 rows — 10% holdout for validation.
• 1,000–20,000 rows — 3-fold cross-validation.
• Fewer than 1,000 rows — 10-fold cross-validation.

You can pass an explicit validation set. A separate test dataset to evaluate the final recommended model is a preview feature — teach it as optional and experimental, not as a GA requirement.

Ensembles, metrics, and exit criteria

Ensemble models are enabled by default and usually appear as the last iterations. Two methods:

Selection follows the Caruana ensemble selection algorithm with sorted initialization: start with up to five best individual models that sit within a 5% band of the best score, then add a model only when it improves the ensemble. You can disable vote or stack ensembles in set_training when you need a single interpretable model.

The primary metric must be one the task allows. Classification often uses accuracy, AUC_weighted, average_precision_score_weighted, norm_macro_recall, or precision_score_weighted. On small or highly skewed labels, thresholded metrics such as accuracy can be a poor objective — AUC_weighted is the usual safer choice for fraud. Regression and forecasting lean on r2_score, normalized_root_mean_squared_error, and normalized_mean_absolute_error. NLP NER currently supports Accuracy as the primary metric. The name you put in YAML must match the metric the service actually logs.

Exit criteria live under limits:

• timeout_minutes — whole job; default is six days (8,640 minutes). Ensembling and explainability after the last trial are not counted in that timeout.
• trial_timeout_minutes — each child; default 43,200 minutes (one month) if unset.
• max_trials — default 1,000.
• max_concurrent_trials — match this to nodes on the cluster so children actually run in parallel; default is one concurrent child.
• enable_early_termination — stop if the short-term score is not improving.

SDK/CLI AutoML jobs run on a compute cluster or compute instance, not on your laptop kernel. Studio’s automated ML wizard is the no-code equivalent for tabular problems; vision and NLP authoring is SDK-first with studio for monitoring.

Studio versus SDK, and what AutoML is not

Use studio when an analyst should launch a classification run without Python. Use SDK/CLI v2 when the AutoML job is a pipeline step, when you need vision/NLP, or when you must pin blocked_training_algorithms, custom featurization, ONNX-compatible models, or distributed LightGBM / TCNForecaster on large data. You can convert many tabular winners to ONNX for C# / ML.NET scoring without a REST hop.

AutoML is not magic. Algorithms have inherent randomness, so two identical configs can differ slightly. A specialist with a well-designed feature set and a tuned LightGBM can beat AutoML on that problem. AutoML’s job in MLOps is to search the space quickly, apply sound defaults, and give you a model whose featurization is already packaged. You still register, evaluate with responsible AI, and compare against the custom candidate (later chapters).

Exam scenario

A bank wants a first fraud model on a 2-million-row tabular set with a 0.8% positive rate. The MLOps engineer creates an MLTable, submits type: automl with task: classification, primary_metric: AUC_weighted (not accuracy), enable_early_termination: true, and max_concurrent_trials equal to the cluster’s node count. Studio shows child trials plus a final voting ensemble. The engineer registers the MLflow model and, in parallel, keeps a custom XGBoost job as a challenger — they do not assume AutoML is automatically production.

Common trap

Do not say “AutoML always wins.” Do not pick accuracy as the primary metric on a brutal class imbalance just because it is the default in a tutorial. Do not ship the estimator and drop the featurizers — missing-value imputation and scaling are the model. Do not point tabular v2 AutoML at a raw uri_file and expect it to behave like MLTable. Do not confuse AutoML (the service searches algorithms) with a sweep job (you bring one training script and search its hyperparameters).