3.14 Model Complexity and the Bias-Variance Tradeoff
Key Takeaways
- Expected generalisation error decomposes into bias squared, variance, and irreducible noise; complexity trades the first against the second.
- Underfitting (high bias) shows high training error, high validation error, and a small gap between them; overfitting (high variance) shows very low training error, high validation error, and a wide gap.
- Increase capacity, add features, or reduce regularisation for high bias; add data, regularise, constrain depth, or apply early stopping for high variance.
- L1 (Lasso) drives redundant coefficients to exactly zero; L2 (Ridge) shrinks all coefficients smoothly and stabilises multicollinearity.
- Spark ML sets strength with `regParam` and the L1/L2 mix with `elasticNetParam` (0.0 = ridge, 1.0 = lasso); scikit-learn's `C` is inverted, so a smaller `C` means stronger regularisation.
3.14 Model Complexity and the Bias-Variance Tradeoff
Bias-Variance Tradeoff & Learning Curve Diagnostics
The expected generalization error of any supervised model decomposes mathematically into three distinct components:
+-----------------------------------------------------------------------------+
| BIAS VS. VARIANCE LEARNING CURVES |
| |
| UNDERFITTING (HIGH BIAS) OVERFITTING (HIGH VARIANCE) |
| Error Error |
| ^ ^ |
| | --- Validation Error | --- Validation Error |
| | === Training Error | |
| | | <--- LARGE GAP ---> |
| | ------------------- (High Error) | ------------------- |
| | =================== (High Error) | |
| | <-- SMALL GAP --> | =================== (Low Error)|
| +----------------------------> +----------------------------> |
| 0 Training Set Size (N) 0 Training Set Size (N) |
| |
| Symptoms: Symptoms: |
| - Training error is high. - Training error is near zero. |
| - Validation error is high. - Validation error is high. |
| - Small gap between curves. - Massive gap between curves. |
+-----------------------------------------------------------------------------+
Comprehensive Diagnostic and Remediation Matrix
| Diagnostic State | Primary Cause | Observed Symptoms | Remediation Strategies |
|---|---|---|---|
| Underfitting (High Bias) | Model is too simple or constrained to capture underlying data structure. | High training error, high validation error, small gap between train and validation performance. | 1. Increase model capacity (deeper trees, polynomial features, neural layers).<br>2. Engineer informative interaction features.<br>3. Reduce regularization penalties (decrease $\lambda$, reduce reg_alpha/reg_lambda).<br>4. Switch to more expressive algorithm (e.g. from linear model to XGBoost). |
| Overfitting (High Variance) | Model memorizes training sample noise and idiosyncrasies rather than true population signals. | Very low training error, high validation error, large generalization gap between curves. | 1. Acquire more training data.<br>2. Apply feature selection / dimensionality reduction.<br>3. Increase regularization penalties (increase L1/L2 penalties).<br>4. Constrain tree complexity (max_depth, min_samples_leaf, subsample).<br>5. Apply early stopping during boosting iterations. |
| Optimal Fit | Balanced model complexity. | Low training error, low validation error, narrow generalization gap. | Maintain configuration; validate on untouched holdout test set. |
Reading the Diagnosis from Two Numbers
Almost every bias-variance question reduces to comparing training error against validation error:
| Training error | Validation error | Gap | Diagnosis | First move |
|---|---|---|---|---|
| High | High | Small | Underfitting / high bias | More capacity, better features, less regularisation |
| Very low | High | Large | Overfitting / high variance | More data, more regularisation, less capacity, early stopping |
| Low | Low | Small | Good fit | Confirm on the untouched test set |
| Low | Much lower than training | Negative | Leakage or a broken split | Investigate before anything else |
That last row is worth remembering: validation error meaningfully below training error is not a triumph, it is a bug — usually a target leak or an accidental overlap between the splits.
What more data does, and does not, fix
Adding training rows reduces variance and therefore narrows the gap. It does essentially nothing for bias: a linear model on a genuinely non-linear relationship will fit that relationship no better with ten times the data. This is why the correct answer to "training and validation error are both high" is never "collect more data".
Complexity knobs by model family
| Family | Increase capacity (less bias) | Decrease capacity (less variance) |
|---|---|---|
| Linear models | Add polynomial or interaction features; lower regParam | Raise regParam; drop features |
| Decision trees | Raise maxDepth; lower minInstancesPerNode | Lower maxDepth; prune; raise the minimum leaf size |
| Random forest | Deeper trees; more features per split | More trees (also stabilises); shallower trees |
| Gradient boosting | More rounds; deeper trees; higher learning rate | Early stopping; subsample; colsample_bytree; reg_alpha/reg_lambda |
| Neural networks | More layers or units | Dropout; weight decay; early stopping |
Random forests are the instructive case: adding trees reduces variance without adding bias, because predictions are averaged over independent bagged learners. Adding boosting rounds does the opposite — it reduces bias and eventually increases variance, which is why early stopping exists for boosting but not for bagging.
Regularization Mechanics: L1 (Lasso) vs. L2 (Ridge) vs. ElasticNet
Regularization injects a penalty term $\Omega(\mathbf{w})$ into the loss function $\mathcal{L}(\mathbf{w})$ to constrain coefficient magnitudes:
| Property | L1 (Lasso) | L2 (Ridge) |
|---|---|---|
| Penalty term | Sum of absolute weights | Sum of squared weights |
| Constraint geometry | Diamond, with corners sitting on the axes | Sphere, smooth everywhere |
| Effect on coefficients | Drives redundant weights to exactly zero | Shrinks every weight smoothly toward zero |
| Side effect | Embedded feature selection; a sparse model | Stable behaviour under multicollinearity |
| Prefer when | Many features, most believed irrelevant | Correlated features you want to keep but damp |
ElasticNet mixes the two penalties. It is the usual choice when features are both numerous and correlated: the L1 component removes the clearly useless ones while the L2 component stops the correlated survivors from thrashing between fits.
Setting the Knobs in Code
The parameter names differ between the two libraries this exam uses, and one of them runs in the opposite direction:
| Library and class | Strength parameter | Mix parameter | Direction |
|---|---|---|---|
Spark ML LinearRegression, LogisticRegression | regParam | elasticNetParam | Higher regParam means more regularisation |
scikit-learn Ridge, Lasso, ElasticNet | alpha | l1_ratio | Higher alpha means more regularisation |
scikit-learn LogisticRegression, SVC | C | penalty | Higher C means less regularisation |
In Spark ML, elasticNetParam=0.0 is pure L2 (ridge), elasticNetParam=1.0 is pure L1
(lasso), and anything between is ElasticNet. In scikit-learn's LogisticRegression, C
is the inverse of regularisation strength, so C=0.01 is a heavily regularised model
and C=100 is very nearly unregularised. A scenario stating that an analyst raised C
to cure overfitting is describing a change that makes the overfitting worse.
A Worked Diagnosis
A random forest is evaluated on a churn dataset at three depths:
| Configuration | Train AUC | Validation AUC | Gap |
|---|---|---|---|
maxDepth=3 | 0.71 | 0.70 | 0.01 |
maxDepth=8 | 0.86 | 0.84 | 0.02 |
maxDepth=20 | 0.99 | 0.79 | 0.20 |
Depth 3 underfits: both numbers are low and nearly identical, so the model is capacity-limited and additional rows would not move it. Depth 20 overfits: training AUC is almost perfect while validation falls back below the depth-8 result, and the 0.20 gap is the variance made visible. Depth 8 is the configuration to select, and the honest estimate of its performance comes from a test set that played no part in this comparison — reusing the validation set to both select and report inflates the number you publish.
A caution on the classic U-curve. The conceptual chart below shows total error falling and then rising as complexity grows, which is the behaviour this exam tests. Very heavily over-parameterised models can show a second descent past the interpolation point, but that phenomenon sits outside this blueprint — answer the classic tradeoff.
During training of a deep neural network, the training loss steadily declines to near zero (0.02), while the validation loss plateaus at a high value (0.75) with a massive widening gap between the two curves. What condition does this indicate and what is the appropriate remediation?
Which statement correctly describes the key difference in behavior between L1 Regularization (Lasso) and L2 Regularization (Ridge)?
A gradient boosted model shows a training RMSE of 41.2 and a validation RMSE of 43.0. A colleague proposes collecting ten times more training data to improve results. What is the correct assessment?
A scikit-learn LogisticRegression with C=1.0 overfits badly: training accuracy is 0.99 and validation accuracy is 0.74. A colleague proposes setting C=100. What will that do?