3.2 Regression Diagnostics: Multicollinearity, Heteroskedasticity & Serial Correlation

Econometric Violations & Diagnostic Testing

When applying multiple regression to financial datasets, the Classical Linear Regression Model (CLRM) assumptions are frequently violated. Financial analysts must identify the nature of each violation, its statistical consequences on Ordinary Least Squares (OLS) estimates, the appropriate diagnostic tests, and the correct econometric remedies.

                     ┌──────────────────────────────────────────────────────────┐
                     │          The "Big Three" Econometric Violations          │
                     └────────────────────────────┬─────────────────────────────┘
                                                  │
         ┌────────────────────────────────────────┼────────────────────────────────────────┐
         ▼                                        ▼                                        ▼
┌────────────────────────┐              ┌────────────────────────┐               ┌────────────────────────┐
│   Heteroskedasticity   │              │   Serial Correlation   │               │   Multicollinearity    │
│ - Non-constant Var(e)  │              │ - Correlated errors    │               │ - High corr among Xs   │
│ - Breusch-Pagan test   │              │ - Durbin-Watson test   │               │ - High R², low t-stats │
│ - White robust SEs     │              │ - Newey-West HAC SEs   │               │ - VIF > 5 or 10        │
└────────────────────────┘              └────────────────────────┘               └────────────────────────┘

---

1. Heteroskedasticity: Non-Constant Error Variance

Heteroskedasticity occurs when the variance of the error terms is not constant across all observations: $Var(\epsilon_i \mid X) = \sigma_i^2$.

Unconditional vs. Conditional Heteroskedasticity

• Unconditional Heteroskedasticity: Error variance varies across observations but is uncorrelated with the independent variables. While technically a violation, it creates no major problems for statistical inference.
• Conditional Heteroskedasticity: Error variance is systematically correlated with one or more independent variables (e.g., error variance increases as firm market cap increases). This is a severe violation that distorts hypothesis testing.

Consequences of Conditional Heteroskedasticity on OLS:

1. OLS parameter estimates ($b_0, b_1, \dots, b_k$) remain unbiased and consistent.
2. OLS standard errors ($s_{b_j}$) are biased and inconsistent (typically severely understated in financial applications).
3. Understated standard errors lead to artificially inflated $t$-statistics, causing false rejections of the null hypothesis (Type I errors).
4. The overall regression $F$-statistic is unreliable.

Detection Methods:

• Residual Scatter Plots: Plotting squared residuals ($e_i^2$) against independent variables $X_j$ or fitted values $\hat{Y}_i$ reveals funneling or fan-shaped patterns.
• Breusch-Pagan (BP) Test: Regresses the squared OLS residuals on the $k$ independent variables: $e_i^2 = a_0 + a_1 X_{1i} + \dots + a_k X_{ki} + u_i$. The test statistic is calculated as:

$$LM = n \times R^2_{resid} \sim \chi^2_k$$

Where $R^2_{resid}$ is the $R^2$ from the residual regression and $k$ is the degrees of freedom. If $LM > \chi^2_{crit, \alpha, k}$, reject $H_0$ of homoskedasticity.

• White's Test: A generalized test regressing $e_i^2$ on all independent variables, their squares ($X_j^2$), and their cross-products ($X_j X_m$).

Remedies:

• Hansen-White Robust Standard Errors (also known as White-corrected or heteroskedasticity-consistent standard errors): Re-estimates standard errors without altering point estimates $b_j$. This restores valid $t$-tests and $F$-tests.
• Generalized Least Squares (GLS / Weighted Least Squares): Transforms variables by dividing by the conditional standard deviation $\sigma_i$ when the error variance structure is known.

---

2. Serial Correlation (Autocorrelation)

Serial correlation occurs in time-series data when regression error terms are correlated across consecutive time periods:

$$\epsilon_t = \rho \epsilon_{t-1} + u_t, \quad |\rho| < 1$$

Where $\rho$ is the first-order serial correlation parameter.

Positive vs. Negative Serial Correlation

• Positive Serial Correlation (most common in finance): Positive errors follow positive errors, and negative errors follow negative errors ($\rho > 0$).
• Negative Serial Correlation: Errors alternate systematically in sign across adjacent periods ($\rho < 0$).

Consequences of Serial Correlation on OLS:

1. OLS parameter estimates remain unbiased and consistent, UNLESS the regression includes a lagged dependent variable ($Y_{t-1}$) as a regressor. If a lagged dependent variable is present alongside serial correlation, OLS estimates are biased and inconsistent.
2. For positive serial correlation, OLS standard errors are understated, leading to artificially inflated $t$-statistics and excessive Type I errors.
3. The Mean Square Error (MSE) is understated, causing $R^2$ to be artificially inflated.
4. The $F$-statistic is invalid and overstated.

Detection Methods:

• Durbin-Watson ($DW$) Statistic: Measures first-order serial correlation:

$$DW = \frac{\sum_{t=2}^n (e_t - e_{t-1})^2}{\sum_{t=1}^n e_t^2} \approx 2(1 - r)$$

Where $r$ is the sample correlation between adjacent residuals $e_t$ and $e_{t-1}$.

• If $r = 0$ (no autocorrelation), $DW \approx 2.0$.
• If $r = +1$ (perfect positive autocorrelation), $DW \approx 0.0$.
• If $r = -1$ (perfect negative autocorrelation), $DW \approx 4.0$.

   0                d_L              d_U              4 - d_U          4 - d_L            4
   ├─── Positive ───┼── Inconclusive ──┼── No Serial ───┼── Inconclusive ──┼─── Negative ───┤
   │  Autocorr (H₁) │                  │Autocorr (H₀)   │                  │  Autocorr (H₁) │

• Decision Rules for Positive Autocorrelation ($H_0: \rho = 0$ vs. $H_a: \rho > 0$):
  • If $DW < d_L$: Reject $H_0$ (positive serial correlation confirmed).
  • If $d_L \le DW \le d_U$: Inconclusive.
  • If $DW > d_U$: Fail to reject $H_0$ (no evidence of positive serial correlation).

Important Limitation of DW: The Durbin-Watson test is invalid if the regression contains a lagged dependent variable as an independent regressor. In such cases, use the Breusch-Godfrey Lagrange Multiplier test.

Remedies:

• Newey-West Standard Errors (HAC SEs): Computes Heteroskedasticity and Autocorrelation Consistent standard errors, correcting $t$-statistics for both violations without changing OLS slope estimates.
• Autoregressive Transformation: Re-estimates the equation via Generalized Least Squares (e.g., Cochrane-Orcutt or Prais-Winsten procedure).

---

3. Multicollinearity

Multicollinearity occurs when two or more independent variables in a multiple regression are highly linearly correlated with each other.

Consequences of Multicollinearity on OLS:

1. OLS parameter estimates remain unbiased and consistent.
2. The standard errors of individual slope coefficients ($s_{b_j}$) are artificially inflated (greatly enlarged).
3. Individual $t$-statistics are artificially low, resulting in failure to reject $H_0$ even when variables are economically critical (Type II errors).
4. Classic Symptom: High overall $R^2$ and a statistically significant overall $F$-test, yet none (or very few) of the individual slope coefficients have statistically significant $t$-statistics.
5. Estimated coefficients become highly sensitive to minor changes in data or specification.

Detection Methods:

• Pairwise Correlations: Regressors with $|r| > 0.70$ suggest potential collinearity (though collinearity among three or more variables can exist even if pairwise correlations are moderate).
• Variance Inflation Factor (VIF): For each independent variable $X_j$, regressing $X_j$ on all remaining $k - 1$ regressors yields $R_j^2$:

$$VIF_j = \frac{1}{1 - R_j^2}$$

• Interpretation Thresholds:
  • $VIF = 1.0$: No correlation whatsoever with other regressors.
  • $VIF > 5.0$: Moderate to severe multicollinearity requiring scrutiny.
  • $VIF > 10.0$: Severe multicollinearity that severely degrades regression precision.

Remedies:

• Drop one or more collinear variables: Remove redundant predictors that share economic information.
• Transform collinear variables: Use first differences, ratios, or construct orthogonal factors via Principal Component Analysis (PCA).
• Increase sample size: Gathering additional data points can reduce coefficient standard errors.

---

Summary Matrix: Diagnostic Comparison

---

Model Misspecification: Types and Impacts

Model misspecification occurs when the regression equation fails to reflect the true data generating process:

1. Omission of a Relevant Variable: If a true determinant $Z$ is omitted and correlated with included $X$, OLS estimates suffer from omitted variable bias: $$E(\hat{b}_1) = \beta_1 + \beta_z \frac{Cov(X, Z)}{Var(X)}$$ Coefficients are biased and inconsistent, and error variance estimates are distorted.
2. Inclusion of an Irrelevant Variable: If an unnecessary variable is included, OLS estimates remain unbiased and consistent, but degrees of freedom are lost and coefficient standard errors become inefficient (inflated).
3. Incorrect Functional Form: Modeling a non-linear relationship (e.g., quadratic, logarithmic) as strictly linear causes biased residuals and inconsistent predictions (detectable via Ramsey RESET test).
4. Inappropriate Pooling / Structural Breaks: Combining data from distinct structural regimes (e.g., pre-crisis and post-crisis) biases coefficients. Evaluated via the Chow test.

---

Qualitative Dependent Variable Models

When the dependent variable $Y$ is discrete/binary (e.g., corporate default $Y \in \{0, 1\}$, loan approval, merger success), linear OLS (the Linear Probability Model) fails because:

• Predicted probabilities $\hat{Y}$ can fall outside the logical range $[0, 1]$.
• Residuals are inherently heteroskedastic ($Var(\epsilon_i) = P_i(1 - P_i)$).
• Error terms cannot be normally distributed.

Non-Linear Binary Response Models