3.14 Model Complexity and the Bias-Variance Tradeoff
Key Takeaways
Expected generalisation error decomposes into bias squared, variance, and irreducible noise; complexity trades the first against the second.
Underfitting (high bias) shows high training error, high validation error, and a small gap between them; overfitting (high variance) shows very low training error, high validation error, and a wide gap.
Increase capacity, add features, or reduce regularisation for high bias; add data, regularise, constrain depth, or apply early stopping for high variance.
L1 (Lasso) drives redundant coefficients to exactly zero; L2 (Ridge) shrinks all coefficients smoothly and stabilises multicollinearity.
Spark ML sets strength with
regParamand the L1/L2 mix withelasticNetParam(0.0 = ridge, 1.0 = lasso); scikit-learn'sCis inverted, so a smallerCmeans stronger regularisation.
3.14 Model Complexity and the Bias-Variance Tradeoff
Bias-Variance Tradeoff & Learning Curve Diagnostics
The expected generalization error of any supervised model decomposes mathematically into three distinct components:
+-----------------------------------------------------------------------------+
| BIAS VS. VARIANCE LEARNING CURVES |
| |
| UNDERFITTING (HIGH BIAS) OVERFITTING (HIGH VARIANCE) |
| Error Error |
| ^ ^ |
| | --- Validation Error | --- Validation Error |
| | === Training Error | |
| | | <--- LARGE GAP ---> |
| | ------------------- (High Error) | ------------------- |
| | =================== (High Error) | |
| | <-- SMALL GAP --> | =================== (Low Error)|
| +----------------------------> +----------------------------> |
| 0 Training Set Size (N) 0 Training Set Size (N) |
| |
| Symptoms: Symptoms: |
| - Training error is high. - Training error is near zero. |
| - Validation error is high. - Validation error is high. |
| - Small gap between curves. - Massive gap between curves. |
+-----------------------------------------------------------------------------+
Comprehensive Diagnostic and Remediation Matrix
| Diagnostic State | Primary Cause | Observed Symptoms | Remediation Strategies |
|---|---|---|---|
| Underfitting (High Bias) | Model is too simple or constrained to capture underlying data structure. | High training error, high validation error, small gap between train and validation performance. | 1. Increase model capacity (deeper trees, polynomial features, neural layers).; 2. Engineer informative interaction features.; 3. Reduce regularization penalties (decrease , reduce reg_alpha/reg_lambda).; 4. Switch to more expressive algorithm (e.g. from linear model to XGBoost). |
| Overfitting (High Variance) | Model memorizes training sample noise and idiosyncrasies rather than true population signals. | Very low training error, high validation error, large generalization gap between curves. | 1. Acquire more training data.; 2. Apply feature selection / dimensionality reduction.; 3. Increase regularization penalties (increase L1/L2 penalties).; 4. Constrain tree complexity (max_depth, min_samples_leaf, subsample).; 5. Apply early stopping during boosting iterations. |
| Optimal Fit | Balanced model complexity. | Low training error, low validation error, narrow generalization gap. | Maintain configuration; validate on untouched holdout test set. |
Reading the Diagnosis from Two Numbers
Almost every bias-variance question reduces to comparing training error against validation error:
| Training error | Validation error | Gap | Diagnosis | First move |
|---|---|---|---|---|
| High | High | Small | Underfitting / high bias | More capacity, better features, less regularisation |
| Very low | High | Large | Overfitting / high variance | More data, more regularisation, less capacity, early stopping |
| Low | Low | Small | Good fit | Confirm on the untouched test set |
| Low | Much lower than training | Negative | Leakage or a broken split | Investigate before anything else |
That last row is worth remembering: validation error meaningfully below training error is not a triumph, it is a bug — usually a target leak or an accidental overlap between the splits.
What more data does, and does not, fix
Adding training rows reduces variance and therefore narrows the gap. It does essentially nothing for bias: a linear model on a genuinely non-linear relationship will fit that relationship no better with ten times the data. This is why the correct answer to "training and validation error are both high" is never "collect more data".
Complexity knobs by model family
| Family | Increase capacity (less bias) | Decrease capacity (less variance) |
|---|---|---|
| Linear models | Add polynomial or interaction features; lower regParam | Raise regParam; drop features |
| Decision trees | Raise maxDepth; lower minInstancesPerNode | Lower maxDepth; prune; raise the minimum leaf size |
| Random forest | Deeper trees; more features per split | More trees (also stabilises); shallower trees |
| Gradient boosting | More rounds; deeper trees; higher learning rate | Early stopping; subsample; colsample_bytree; reg_alpha/reg_lambda |
| Neural networks | More layers or units | Dropout; weight decay; early stopping |
Random forests are the instructive case: adding trees reduces variance without adding bias, because predictions are averaged over independent bagged learners. Adding boosting rounds does the opposite — it reduces bias and eventually increases variance, which is why early stopping exists for boosting but not for bagging.
Regularization Mechanics: L1 (Lasso) vs. L2 (Ridge) vs. ElasticNet
Regularization injects a penalty term into the loss function to constrain coefficient magnitudes:
| Property | L1 (Lasso) | L2 (Ridge) |
|---|---|---|
| Penalty term | Sum of absolute weights | Sum of squared weights |
| Constraint geometry | Diamond, with corners sitting on the axes | Sphere, smooth everywhere |
| Effect on coefficients | Drives redundant weights to exactly zero | Shrinks every weight smoothly toward zero |
| Side effect | Embedded feature selection; a sparse model | Stable behaviour under multicollinearity |
| Prefer when | Many features, most believed irrelevant | Correlated features you want to keep but damp |
ElasticNet mixes the two penalties. It is the usual choice when features are both numerous and correlated: the L1 component removes the clearly useless ones while the L2 component stops the correlated survivors from thrashing between fits.
Setting the Knobs in Code
The parameter names differ between the two libraries this exam uses, and one of them runs in the opposite direction:
| Library and class | Strength parameter | Mix parameter | Direction |
|---|---|---|---|
Spark ML LinearRegression, LogisticRegression | regParam | elasticNetParam | Higher regParam means more regularisation |
scikit-learn Ridge, Lasso, ElasticNet | alpha | l1_ratio | Higher alpha means more regularisation |
scikit-learn LogisticRegression, SVC | C | penalty | Higher C means less regularisation |
In Spark ML, elasticNetParam=0.0 is pure L2 (ridge), elasticNetParam=1.0 is pure L1
(lasso), and anything between is ElasticNet. In scikit-learn's LogisticRegression, C
is the inverse of regularisation strength, so C=0.01 is a heavily regularised model
and C=100 is very nearly unregularised. A scenario stating that an analyst raised C
to cure overfitting is describing a change that makes the overfitting worse.
A Worked Diagnosis
A random forest is evaluated on a churn dataset at three depths:
| Configuration | Train AUC | Validation AUC | Gap |
|---|---|---|---|
maxDepth=3 | 0.71 | 0.70 | 0.01 |
maxDepth=8 | 0.86 | 0.84 | 0.02 |
maxDepth=20 | 0.99 | 0.79 | 0.20 |
Depth 3 underfits: both numbers are low and nearly identical, so the model is capacity-limited and additional rows would not move it. Depth 20 overfits: training AUC is almost perfect while validation falls back below the depth-8 result, and the 0.20 gap is the variance made visible. Depth 8 is the configuration to select, and the honest estimate of its performance comes from a test set that played no part in this comparison — reusing the validation set to both select and report inflates the number you publish.
A caution on the classic U-curve. The conceptual chart below shows total error falling and then rising as complexity grows, which is the behaviour this exam tests. Very heavily over-parameterised models can show a second descent past the interpolation point, but that phenomenon sits outside this blueprint — answer the classic tradeoff.
During training of a deep neural network, the training loss steadily declines to near zero (0.02), while the validation loss plateaus at a high value (0.75) with a massive widening gap between the two curves. What condition does this indicate and what is the appropriate remediation?
Underfitting (High Bias); remediate by adding more layers and decreasing L2 regularization.
Data leakage; remediate by removing cross-validation folds.
Optimal convergence; deploy model immediately to production.
Overfitting (High Variance); remediate by increasing regularization, applying early stopping, or constraining model capacity.
Which statement correctly describes the key difference in behavior between L1 Regularization (Lasso) and L2 Regularization (Ridge)?
L1 regularization adds an absolute weight penalty that produces sparse models by driving redundant feature coefficients to exactly zero, whereas L2 regularization adds a squared weight penalty that smoothly shrinks coefficients toward zero without setting them to exact zero.
L1 regularization is only applicable to decision trees, while L2 regularization is only applicable to linear regression.
L2 regularization performs automatic feature selection, while L1 regularization prevents multicollinearity without eliminating features.
L1 regularization requires no feature scaling, whereas L2 regularization requires all features to be categorical.
A gradient boosted model shows a training RMSE of 41.2 and a validation RMSE of 43.0. A colleague proposes collecting ten times more training data to improve results. What is the correct assessment?
Correct — more data always improves generalisation
The small gap indicates high bias (underfitting), not high variance, so more data will not help; increasing capacity, engineering better features, or reducing regularisation is what addresses it
The model is overfitting and more data is the standard remedy
The RMSE values cannot be compared unless both are converted to
A scikit-learn LogisticRegression with C=1.0 overfits badly: training accuracy is 0.99 and validation accuracy is 0.74. A colleague proposes setting C=100. What will that do?
Weaken the regularisation further and make the overfitting worse, because C is the inverse of regularisation strength; a smaller value such as C=0.01 is what strengthens the penalty
Strengthen the penalty and close the generalisation gap, because C scales the penalty term directly
Leave the fit unchanged, because C only affects the solver's convergence tolerance
Switch the penalty from L2 to L1 and perform feature selection
Sections you finish are checked off in the contents.