3.9 Counting the Models Trained by Grid Search and Cross-Validation

3.9 Counting the Models Trained by Grid Search and Cross-Validation

One exam objective is purely arithmetic: given a parameter grid and a validation scheme, state how many models are trained. It is easy marks, provided you apply the formula rather than guessing.

The Formula

Step 1 — count the combinations. For hyperparameters $p_1 \dots p_n$ with $|p_i|$ candidate values each, the grid contains

$$C = \prod_{i=1}^{n} |p_i|$$

combinations. A grid is a Cartesian product, so the counts multiply.

Step 2 — multiply by the validation scheme.

Worked Example 1 — scikit-learn GridSearchCV

An SVM is tuned with GridSearchCV, 5-fold cross-validation, and this grid:

• C: [0.1, 1, 10] → 3 values
• kernel: ['linear', 'rbf'] → 2 values
• gamma: [0.01, 0.1, 1] → 3 values

$$C = 3 \times 2 \times 3 = 18 \text{ combinations}$$ $$\text{fits} = 18 \times 5 = 90$$

90 models are trained during the search. With refit=True (the default), GridSearchCV then trains one more model — the best combination on the entire training set — for a total of 91 fits. When a question asks how many models the search trains, the answer is 90; when it asks how many fits occur in total including the refit, it is 91. Read the wording carefully.

Worked Example 2 — Spark ML CrossValidator

param_grid = (ParamGridBuilder()
              .addGrid(lr.regParam, [0.01, 0.1, 1.0])       # 3
              .addGrid(lr.elasticNetParam, [0.0, 0.5, 1.0])  # 3
              .build())                                      # 9 combinations

crossval = CrossValidator(estimator=pipeline,
                          estimatorParamMaps=param_grid,
                          evaluator=evaluator,
                          numFolds=5)

$$9 \text{ combinations} \times 5 \text{ folds} = 45 \text{ models}$$

CrossValidator then refits the winning combination on the full training dataset, so 46 fits occur in total and cv_model.bestModel is that final refit.

Switching to TrainValidationSplit(trainRatio=0.8) with the same grid fits 9 models — one per combination — plus the final refit.

Why the Arithmetic Matters

Consider tuning XGBoost over five hyperparameters with four candidate values each, under 5-fold cross-validation:

$$4^5 \times 5 = 1{,}024 \times 5 = 5{,}120 \text{ model fits}$$

At two minutes per fit that is roughly 170 hours of sequential compute. This is the concrete reason the exam pairs this objective with the search-strategy objective: the multiplicative blow-up is what motivates random and Bayesian search, and it is why TrainValidationSplit exists for very large data.

A budgeting checklist

1. Multiply the grid sizes to get $C$.
2. Multiply by $K$ (and by repeats, if any).
3. Multiply by the average fit time.
4. Divide by the tuner's parallelism for an approximate wall-clock estimate.
5. If the number is unacceptable, cut the grid, cut $K$, or switch to Bayesian search — not the fold count alone, since $K < 3$ makes the estimate noisy again.

Common trap: parallelism changes how long the search takes, never how many models are trained. A question that adds parallelism=4 to a 45-model grid is testing whether you conflate the two — the answer is still 45.

Variations That Change the Count

Several details alter the arithmetic, and each one appears in question stems.

Conditional grids

scikit-learn accepts a list of grid dictionaries, and the totals add rather than multiply, because parameters that do not apply to a kernel are simply not enumerated for it:

param_grid = [
    {"kernel": ["linear"], "C": [0.1, 1, 10]},                    # 3 combinations
    {"kernel": ["rbf"], "C": [0.1, 1, 10], "gamma": [0.01, 0.1]}, # 6 combinations
]                                                                  # 9 in total

Nine combinations under 5-fold cross-validation is 45 fits. A flat product over the union of the keys — kernel (2) × C (3) × gamma (2) — would count 12 combinations and 60 fits, over-counting the three linear configurations that gamma does not apply to.

Early stopping does not reduce the count

Early stopping shortens each individual fit by halting boosting rounds; it does not remove any combination from the grid. The number of models trained is unchanged.

Nested cross-validation multiplies again

An outer loop of $K_{outer}$ folds wrapped around an inner tuning loop of $K_{inner}$ folds trains $K_{outer} \times (C \times K_{inner} + 1)$ models. With $C = 9$, $K_{inner} = 5$, and $K_{outer} = 5$, that is $5 \times 46 = 230$ fits — which is why nested CV is reserved for small datasets and publication-grade estimates.

Hyperopt is budgeted, not enumerated

fmin(..., max_evals=40) trains exactly 40 models regardless of how large the search space is, because Bayesian and random search sample rather than enumerate. If each trial internally runs its own $K$-fold cross-validation, the total becomes $40 \times K$. This contrast — a grid whose cost is set by the space, versus a Hyperopt run whose cost is set by the budget — is the reason a large space pushes practitioners toward fmin.