7.1 Regression Diagnostics

Regression Diagnostics

After fitting a linear model, FRM candidates must ask whether the Gauss–Markov world still holds. Regression diagnostics detect heteroskedasticity, collinearity, misspecification, and influential points so that coefficient estimates and standard errors are interpreted correctly.

Heteroskedasticity: Problem, Tests, and Fixes

Homoskedasticity means Var(εᵢ | X) = σ² for all i. Heteroskedasticity means the conditional error variance changes with regressors or with the level of the fitted value—common in cross-section risk data (large banks more volatile than small ones) and in some return regressions.

Under classical assumptions with E[ε|X] = 0, OLS slope estimates remain unbiased and consistent even if variances differ. What breaks is the usual variance formula Var(β̂) = σ²(X′X)⁻¹. Reported t- and F-statistics become unreliable: often too optimistic when large residuals coincide with high-leverage x-values.

Worked sketch: White-style intuition

Suppose ê² rises with firm size S. A Breusch–Pagan regression ê² = a + b S + u with a large t-stat on b rejects constant variance. Reporting OLS with ordinary SEs is then unsafe; use heteroskedasticity-consistent (HC) standard errors or model the variance and use weighted least squares.

Exam trap: “heteroskedasticity biases OLS slopes.” Under mean-zero errors, slopes stay unbiased; inference is what fails unless you correct SEs or the variance model.

Multicollinearity Versus Perfect Collinearity

Multicollinearity means two or more regressors are highly linearly related. Coefficients remain unbiased if the model is otherwise correct, but Var(β̂ⱼ) inflates because (X′X) is nearly singular. Individual t-stats look weak even when the group of variables jointly matters (significant F, insignificant individuals).

Diagnostics: variance inflation factors (VIFⱼ = 1 / (1 − Rⱼ²) from regressing xⱼ on the other x’s); pairwise correlations; unstable signs when one correlated variable is dropped.

Perfect collinearity is the extreme case: one column of X is an exact linear combination of others (e.g., including both a dummy for every category and an intercept). Then X′X is singular and OLS cannot be computed—software drops a column. Perfect collinearity is a design error; multicollinearity is a precision problem.

Worked example: two correlated factors

Regress excess return on market and industry factors with sample correlation 0.95. Both may be “insignificant” individually while a joint test rejects zero. The model still estimates an overall fit; you simply cannot cleanly separate the two slopes. Solutions: drop or combine variables, use a principal component, or accept the joint story and stop over-interpreting single coefficients.

Omitted Variables Versus Irrelevant Variables

Omitted variable bias arises when a relevant regressor z that belongs in the true model is left out and Corr(x, z) ≠ 0. The OLS slope on the included x absorbs part of z’s effect. Direction of bias follows the product of (effect of z on y) and (correlation of z with x).

Irrelevant variables are regressors with true coefficient zero. Including them does not bias the other coefficients if the rest of the model is correct, but it increases the variance of the estimators (degrees of freedom and multicollinearity costs). So: omission can create bias; kitchen-sink inclusion mainly hurts precision.

Worked bias direction

True model: credit spread = β₀ + β₁ Leverage + β₂ Volatility + ε, with β₂ > 0 and Leverage positively correlated with Volatility. If you omit Volatility, the OLS slope on Leverage is typically biased upward—leverage “steals” part of volatility’s positive effect.

Model Selection and the Bias–Variance Trade-Off

Richer models reduce bias (better approximation to E[y|X]) but raise variance (more parameters estimated from the same n). Underfitting leaves systematic residual structure; overfitting fits noise and forecasts poorly out of sample.

Practical tools:

• Adjusted R² penalizes extra regressors relative to raw R².
• AIC and BIC (BIC penalizes complexity more heavily) compare likelihood versus parameter count.
• Cross-validation / out-of-sample MSE for predictive risk models.

FRM emphasis: statistical fit on the estimation sample is not the same as economic validity or stable forecast performance. A high in-sample R² with fragile coefficients is a red flag in risk factor models.

Residual Plots, Outliers, and Influence

Plot residuals êᵢ against fitted ŷᵢ and against each xⱼ:

• Random scatter around zero → comfort with linearity and constant variance.
• Curved pattern → missing nonlinear term or transformation.
• Fan shape → heteroskedasticity.
• Clump of large |êᵢ| → outliers or heavy tails.

An outlier is an unusual y given X; a leverage point has unusual X; an influential point moves β̂ a lot (Cook’s distance, DFBETAS). Always ask whether the point is a data error, a regime the model should include (crisis dummy), or a genuine rare event.

Worked residual check

n = 5 toy residuals after a line fit: −0.1, 0.2, −0.05, 0.1, 3.5. The last point dominates SSE. Refitting without it may change the slope sharply—report both fits and investigate the observation before “deleting inconvenient crises.”

When Is OLS BLUE?

BLUE = Best Linear Unbiased Estimator. Under the Gauss–Markov conditions—linear correctly specified conditional mean, E[ε|X] = 0 (or at least E[ε] = 0 and exogenous regressors in the classical finite-sample statement), spherical errors (homoskedastic and uncorrelated), and fixed regressors / appropriate sampling—the OLS estimator has the smallest variance among all linear unbiased estimators of β.

OLS is not BLUE if errors are heteroskedastic or serially correlated (GLS can dominate), if the linear mean is wrong, or if you leave the class of linear unbiased rules (nonlinear or biased estimators may win on MSE). Normality is not required for BLUE; normality underwrites exact t/F finite-sample distributions.

Diagnostic Workflow for FRM

1. Fit OLS; inspect residual plots and summary fit.
2. Test / robustify against heteroskedasticity; check serial correlation in time-series regressions (next sections).
3. Screen VIFs and perfect collinearity.
4. Ask what was omitted; avoid stuffing irrelevant controls without a reason.
5. Use information criteria or out-of-sample tests for competing specifications.
6. Document outliers and influence; justify any exclusions.

Master these checks and you will know when a published beta, hedge ratio, or factor loading is statistically trustworthy—and when only the point estimate (not the starred t-stat) deserves attention.