3.6 Grid, Random, and Bayesian Search Strategies

3.6 Grid, Random, and Bayesian Search Strategies

Hyperparameter optimization is the systematic process of identifying the optimal combination of model hyperparameters that minimizes validation loss. While model parameters (such as tree split thresholds or linear coefficients) are learned directly during training, hyperparameters (such as learning rate, tree depth, and regularization penalties) govern the learning algorithm itself. Databricks natively bundles Hyperopt, an open-source Python library for Sequential Model-Based Optimization (SMBO) using Bayesian reasoning.

---

Hyperparameter Optimization Paradigms Compared

+-----------------------------------------------------------------------------+
|                  HYPERPARAMETER TUNING STRATEGY COMPARISON                  |
|                                                                             |
|   GRID SEARCH             RANDOM SEARCH           BAYESIAN OPTIMIZATION     |
|   (ParamGridBuilder)      (Random Sampling)       (Hyperopt / TPE)          |
|   +-----------------+     +-----------------+     +-----------------+       |
|   | . . . . . . . . |     |   .   .     .   |     |       . . . .   |       |
|   | . . . . . . . . |     | .       .     . |     |     . . . . . . |       |
|   | . . . . . . . . |     |     .     .   . |     |         . . .   |       |
|   | . . . . . . . . |     |   .     .   .   |     |                 |       |
|   +-----------------+     +-----------------+     +-----------------+       |
|   - Exhaustive grid       - Random distributions  - Learns from past trials |
|   - Combinatorial blowup  - Better coverage       - Focuses on optimal space|
|   - Wastes compute        - Uninformed trials     - Max Expected Improvement|
+-----------------------------------------------------------------------------+

Grid Search

• Mechanics: Exhaustively evaluates every point in the Cartesian product of predefined parameter candidate lists.
• Limitations: Suffers from the curse of dimensionality. If 5 hyperparameters each have 4 candidate values, Grid Search evaluates $4^5 = 1,024$ full training runs. Furthermore, if only 2 of the 5 hyperparameters strongly influence performance, Grid Search wastes 95% of its computational budget re-evaluating identical values of the important parameters across uninformative variations of unimportant ones.

Random Search

• Mechanics: Randomly draws candidate parameter combinations from specified continuous or discrete distributions over a fixed budget of $N$ evaluations.
• Advantages: Discovers superior parameter configurations faster than Grid Search because it tests unique values across continuous dimensions on every single trial.
• Limitations: Independent and memoryless; it fails to leverage results from previous trials to guide future exploration.

Bayesian Optimization & Tree of Parzen Estimators (TPE)

• Mechanics: Hyperopt implements the Tree of Parzen Estimators (TPE) algorithm. Rather than modeling the objective function response directly $P(y|x)$ (as in Gaussian Processes), TPE uses Bayes' theorem to model two conditional parameter distributions: $P(x|y) = \begin{cases} \ell(x) & \text{if } y < y^* \\ g(x) & \text{if } y \ge y^* \end{cases}$ where $y^*$ is a quantile cutoff (e.g., top 15% best performing trials), $\ell(x)$ is the density of hyperparameters associated with low validation loss, and $g(x)$ is the density associated with the remaining evaluations.
• Acquisition Function: TPE chooses candidate hyperparameter vectors $x$ that maximize the Expected Improvement (EI): $\text{EI}(x) = \frac{\gamma y^* \ell(x) - \ell(x) \int_{-\infty}^{y^*} P(y) dy}{\gamma \ell(x) + (1-\gamma) g(x)} \propto \frac{\ell(x)}{g(x)}$ Maximizing $\frac{\ell(x)}{g(x)}$ naturally samples parameters that have high probability under good models $\ell(x)$ and low probability under poor models $g(x)$.

---

Expressing Each Strategy on Databricks

Grid search — ParamGridBuilder with a Spark ML tuner

from pyspark.ml.tuning import ParamGridBuilder, CrossValidator

param_grid = (ParamGridBuilder()
              .addGrid(lr.regParam, [0.01, 0.1, 1.0])
              .addGrid(lr.elasticNetParam, [0.0, 0.5, 1.0])
              .build())          # 3 x 3 = 9 parameter combinations

ParamGridBuilder builds the full Cartesian product; the count is the product of the list lengths. Section 3.9 covers how that number combines with the fold count.

Random search — Hyperopt with rand.suggest

from hyperopt import fmin, rand, hp

best = fmin(fn=objective, space=search_space, algo=rand.suggest, max_evals=40)

Every trial draws fresh values from the distributions, so continuous hyperparameters are never re-tested at the same point — the key advantage over a grid.

Bayesian search — Hyperopt with tpe.suggest

from hyperopt import fmin, tpe

best = fmin(fn=objective, space=search_space, algo=tpe.suggest, max_evals=40)

TPE builds density models of the good and the poor trials and proposes candidates that maximise the ratio between them, so later trials exploit what earlier trials revealed.

Choosing a Strategy

The dimensionality argument

With 5 hyperparameters at 4 values each, a grid is $4^5 = 1{,}024$ evaluations. If only 2 of the 5 genuinely affect the score, the grid tests each important value 64 times under irrelevant variations of the others. Random search with the same budget tests 1,024 distinct values of every parameter — which is why random search dominates grid search on high-dimensional spaces at equal cost.

---

Defining the Search Space Correctly

The strategy only matters if the space it samples is well posed, and Hyperopt's space primitives are themselves examinable:

The log-uniform case is the one that changes results. Sampling a learning rate uniformly between 0.001 and 0.3 puts roughly 97% of the draws above 0.01, so the small values that often win are almost never tried. hp.loguniform spreads the budget evenly across each decade instead. Note that its bounds are given in natural-log units, so a range of 0.001 to 0.3 is expressed as hp.loguniform("lr", np.log(0.001), np.log(0.3)).

import numpy as np
from hyperopt import hp, fmin, tpe, space_eval

search_space = {
    "learning_rate": hp.loguniform("learning_rate", np.log(0.001), np.log(0.3)),
    "max_depth": hp.quniform("max_depth", 3, 12, 1),
    "booster": hp.choice("booster", ["gbtree", "dart"]),
}

best = fmin(fn=objective, space=search_space, algo=tpe.suggest, max_evals=40)
best_params = space_eval(search_space, best)   # {'booster': 'gbtree', ...}

The hp.choice return-value trap

fmin returns the index into the option list for any hp.choice parameter, not the value itself. Reading best["booster"] therefore yields 0, and passing that straight into an estimator either fails or silently trains something else. hyperopt.space_eval(search_space, best) translates the returned dictionary back into real hyperparameter values, and it is the step scenario questions leave out.

Budget

max_evals is the total number of trials, and it is the only real cost control. A useful default is roughly 10 to 20 evaluations per hyperparameter for TPE, which needs some completed trials before its density models are informative — the first several trials are effectively random regardless of the algorithm chosen.