3.5 Tuning Hyperparameters with Hyperopt's fmin

3.5 Tuning Hyperparameters with Hyperopt's fmin

fmin is Hyperopt's entry point, and its signature encodes the whole workflow:

from hyperopt import fmin, tpe, hp, STATUS_OK, Trials

best = fmin(
    fn=objective_function,     # callable: params dict -> loss (or result dict)
    space=search_space,        # dict of hp.* distributions
    algo=tpe.suggest,          # search algorithm: tpe.suggest, rand.suggest, anneal.suggest
    max_evals=50,              # evaluation budget
    trials=Trials(),           # where results are stored (Trials or SparkTrials)
    rstate=np.random.default_rng(42),   # reproducibility
)

fmin returns a dictionary of the best hyperparameters found. Note the sharp edge: for hp.choice parameters it returns the index into the option list, not the value. hyperopt.space_eval(space, best) converts the returned dictionary back into actual parameter values, which is what you want before retraining a final model.

---

Defining Hyperopt Search Spaces (hyperopt.hp)

Hyperopt provides a comprehensive suite of stochastic distribution primitives in the hyperopt.hp module:

[!IMPORTANT] Integer Casting in hp.quniform: hp.quniform returns a floating-point number (e.g. 6.0). Passing floats to integer parameters like max_depth or n_estimators in Scikit-learn or XGBoost causes runtime exceptions. Always cast integer hyperparameters: int(params['max_depth']).

---

Objective Function Construction & Optimization Loop

The objective function is the evaluation harness executed on every trial. It must satisfy three strict rules:

1. Input: Accepts a single dictionary containing candidate hyperparameters sampled from the search space.
2. Optimization Direction: Hyperopt strictly minimizes the objective. If optimizing a metric where higher is better (e.g., ROC-AUC, Accuracy, $R^2$), you must return the negative metric (e.g., -roc_auc) or $1 - \text{metric}$.
3. Return Format: Returns either a single scalar loss (float) or a dictionary containing at minimum {'loss': loss_value, 'status': STATUS_OK}.

+-----------------------------------------------------------------------------+
|                        HYPEROPT OBJECTIVE FUNCTION CONTRACT                 |
|                                                                             |
|   def objective(params):                                                    |
|       # 1. Cast integer hyperparameters                                     |
|       max_depth = int(params['max_depth'])                                  |
|                                                                             |
|       # 2. Train model on training data                                     |
|       model = XGBClassifier(max_depth=max_depth, lr=params['lr'], ...)      |
|       model.fit(X_train, y_train)                                           |
|                                                                             |
|       # 3. Evaluate validation metric                                       |
|       val_auc = roc_auc_score(y_val, model.predict_proba(X_val)[:, 1])      |
|                                                                             |
|       # 4. Return negated loss for minimization                             |
|       return {'loss': -val_auc, 'status': STATUS_OK}                        |
+-----------------------------------------------------------------------------+

---

Complete Production Code Example

import numpy as np
import xgboost as xgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials
import mlflow
import mlflow.xgboost

# 1. Prepare Data
pdf = spark.table("lakehouse_gold.churn_features").toPandas()
X = pdf.drop(columns=["customer_id", "churned"])
y = pdf["churned"]
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

# 2. Define Hyperopt Search Space
search_space = {
    "max_depth": hp.quniform("max_depth", 3, 10, 1),
    "learning_rate": hp.loguniform("learning_rate", np.log(0.005), np.log(0.3)),
    "n_estimators": hp.quniform("n_estimators", 50, 400, 25),
    "subsample": hp.uniform("subsample", 0.6, 1.0),
    "colsample_bytree": hp.uniform("colsample_bytree", 0.6, 1.0),
    "reg_alpha": hp.loguniform("reg_alpha", np.log(1e-3), np.log(10.0)),
    "reg_lambda": hp.loguniform("reg_lambda", np.log(1e-3), np.log(10.0))
}

# 3. Define Objective Function
def objective_function(params):
    # Cast discrete hyperparameters to integers
    params["max_depth"] = int(params["max_depth"])
    params["n_estimators"] = int(params["n_estimators"])
    
    # Train candidate model
    model = xgb.XGBClassifier(
        **params,
        random_state=42,
        eval_metric="logloss",
        use_label_encoder=False
    )
    
    model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)
    
    # Predict and evaluate validation metric
    val_probs = model.predict_proba(X_val)[:, 1]
    val_auc = roc_auc_score(y_val, val_probs)
    
    # Return negated metric because fmin minimizes loss
    return {
        "loss": -val_auc,
        "status": STATUS_OK,
        "roc_auc": val_auc
    }

# 4. Execute Optimization with MLflow Tracking
mlflow.xgboost.autolog(log_models=True)
trials = Trials()

with mlflow.start_run(run_name="xgboost_hyperopt_parent"):
    best_hyperparameters = fmin(
        fn=objective_function,
        space=search_space,
        algo=tpe.suggest,
        max_evals=50,
        trials=trials,
        rstate=np.random.default_rng(42)
    )

print("Optimal Hyperparameters:", best_hyperparameters)

---

MLflow Automatic Experiment Tracking Integration

Databricks provides seamless, automatic integration between Hyperopt and MLflow. When mlflow.autolog() is activated before calling fmin(), MLflow automatically constructs a hierarchical nested experiment run structure:

+-----------------------------------------------------------------------------+
|                    MLFLOW NESTED RUN HIERARCHY FOR HYPEROPT                 |
|                                                                             |
|   PARENT RUN: "hyperopt_tpe_optimization"                                   |
|   +-- Best Parameters: {learning_rate: 0.038, max_depth: 6, subsample: 0.85}|
|   +-- Best Validation Loss: -0.9240 (ROC-AUC: 0.9240)                       |
|   |                                                                         |
|   +---> CHILD RUN 1: Trial 001 | lr=0.120 | depth=3 | loss=-0.8650 | 2.1s   |
|   +---> CHILD RUN 2: Trial 002 | lr=0.005 | depth=8 | loss=-0.8120 | 4.3s   |
|   +---> CHILD RUN 3: Trial 003 | lr=0.045 | depth=5 | loss=-0.9100 | 3.0s   |
|   +---> CHILD RUN 4: Trial 004 | lr=0.038 | depth=6 | loss=-0.9240 | 3.5s * |
|   +---> CHILD RUN N: Trial ...                                              |
+-----------------------------------------------------------------------------+

Each nested child run captures:

• The exact hyperparameter parameters evaluated in that trial.
• Training metrics, validation metrics, and epoch loss curves.
• Model artifacts and the environment dependencies needed to reload them.
• Trial execution start time, duration, and status.