3.1 Multiple Linear Regression, ANOVA & Hypothesis Testing

Multiple Linear Regression in Investment Analysis

In financial valuation and quantitative asset management, single-factor models rarely explain asset pricing or financial performance adequately. Multiple linear regression extends simple linear regression by estimating the relationship between a single continuous dependent (target) variable $Y$ and two or more independent (explanatory) variables $X_1, X_2, \dots, X_k$:

$$Y_i = b_0 + b_1 X_{1i} + b_2 X_{2i} + \dots + b_k X_{ki} + \epsilon_i, \quad i = 1, 2, \dots, n$$

Where:

• $Y_i$ is the $i$-th observation of the dependent variable.
• $b_0$ is the intercept term (the expected value of $Y$ when all $X_j = 0$).
• $b_j$ is the partial slope coefficient for independent variable $X_j$, measuring the expected change in $Y$ for a one-unit change in $X_j$, holding all other independent variables constant (ceteris paribus).
• $\epsilon_i$ is the unobservable random error (disturbance) term for observation $i$.
• $k$ is the number of independent variables, and $n$ is the total sample size.

                  ┌──────────────────────────────────────────────────────────┐
                  │             Multiple Linear Regression Model             │
                  │      Y_i = b_0 + b_1*X_{1i} + ... + b_k*X_{ki} + e_i     │
                  └────────────────────────────┬─────────────────────────────┘
                                               │
         ┌─────────────────────────────────────┼─────────────────────────────────────┐
         ▼                                     ▼                                     ▼
┌─────────────────┐                   ┌─────────────────┐                   ┌─────────────────┐
│ CLRM Assumptions│                   │   ANOVA Table   │                   │Hypothesis Tests │
│ - Linearity     │                   │ - SSR, SSE, SST │                   │ - Individual t  │
│ - E(e|X) = 0    │                   │ - MSR, MSE      │                   │ - Overall F     │
│ - Homoskedastic │                   │ - s_e = √MSE    │                   │ - p-values & CIs│
│ - No autocorr   │                   │ - R² vs Adj R²  │                   └─────────────────┘
│ - No multicol   │                   └─────────────────┘
│ - Normality     │
└─────────────────┘

---

The Six Core Assumptions of the Classical Linear Regression Model (CLRM)

For Ordinary Least Squares (OLS) estimators to be BLUE (Best Linear Unbiased Estimators) and for statistical inference ($t$-tests, $F$-tests, confidence intervals) to be valid, six assumptions must hold:

1. Linearity in Parameters: The relationship between the dependent variable and the independent variables is linear in the parameters $b_0, b_1, \dots, b_k$. Independent variables themselves can be non-linear transformations (e.g., $X^2$ or $\ln(X)$).
2. Zero Conditional Mean of Errors: The expected value of the error term conditional on the independent variables is zero: $E(\epsilon_i \mid X_{1i}, X_{2i}, \dots, X_{ki}) = 0$. This implies no omitted variable bias and that regressors are strictly exogenous ($Cov(X_{ji}, \epsilon_i) = 0$).
3. Homoskedasticity (Constant Error Variance): The variance of the error term is constant across all observations conditional on the independent variables: $Var(\epsilon_i \mid X) = \sigma^2$ for all $i$.
4. No Serial Correlation (Independence of Errors): The error terms for any two distinct observations are uncorrelated: $Cov(\epsilon_i, \epsilon_j) = 0$ for all $i \neq j$.
5. No Perfect Multicollinearity: No independent variable is an exact linear combination of other independent variables. If exact collinearity exists, the OLS parameter matrix cannot be inverted.
6. Normality of Error Distribution: The error terms are normally distributed: $\epsilon_i \sim N(0, \sigma^2)$. This assumption is essential for conducting exact hypothesis tests in small samples (though the Central Limit Theorem relaxes this in large samples).

---

ANOVA Table Breakdown & Derivations

Analysis of Variance (ANOVA) partitions the total variability of the dependent variable into explained (regression) and unexplained (residual) components.

Sum of Squares Relationships

$$\text{Total Sum of Squares (SST)} = \sum_{i=1}^n (Y_i - \bar{Y})^2$$ $$\text{Regression Sum of Squares (SSR)} = \sum_{i=1}^n (\hat{Y}_i - \bar{Y})^2$$ $$\text{Sum of Squared Errors (SSE)} = \sum_{i=1}^n (Y_i - \hat{Y}_i)^2 = \sum_{i=1}^n e_i^2$$ $$\mathbf{SST = SSR + SSE}$$

Degrees of Freedom and Mean Squares

• Regression Degrees of Freedom: $k$ (number of slope parameters).
• Residual (Error) Degrees of Freedom: $n - k - 1$ (sample size minus total estimated parameters).
• Total Degrees of Freedom: $n - 1$.
• Mean Square Regression (MSR): $\text{MSR} = \frac{\text{SSR}}{k}$
• Mean Square Error (MSE): $\text{MSE} = \frac{\text{SSE}}{n - k - 1}$
• Standard Error of the Estimate ($s_e$): The sample estimate of the standard deviation of regression residuals:

$$s_e = \sqrt{\text{MSE}} = \sqrt{\frac{\text{SSE}}{n - k - 1}}$$

Standard ANOVA Table Format

---

Goodness-of-Fit: $R^2$ versus Adjusted $R^2$

The Coefficient of Determination ($R^2$) measures the proportion of total variation in $Y$ explained by the regression model:

$$R^2 = \frac{\text{SSR}}{\text{SST}} = 1 - \frac{\text{SSE}}{\text{SST}}$$

A fundamental limitation of $R^2$ is that adding any new independent variable will mathematically never decrease $R^2$ and almost always increases it, even if the new variable is economically irrelevant. To address this, Adjusted $R^2$ ($R^2_{adj}$ or $\bar{R}^2$) introduces a degrees-of-freedom penalty:

$$R^2_{adj} = 1 - \left[\frac{n - 1}{n - k - 1}\right](1 - R^2) = 1 - \left[\frac{n - 1}{n - k - 1}\right]\left(\frac{\text{SSE}}{\text{SST}}\right) = 1 - \frac{\text{MSE}}{\text{SST} / (n - 1)}$$

Key Properties of Adjusted $R^2$:

1. $R^2_{adj} \le R^2$ always. The equality holds only when $R^2 = 1.0$ or $k = 0$.
2. $R^2_{adj}$ can decrease when a new variable is added if the reduction in $\text{SSE}$ is insufficient to offset the loss of one degree of freedom.
3. Decision Rule: Adding an independent variable will increase $R^2_{adj}$ if and only if the absolute value of the $t$-statistic on that added variable exceeds 1.0 (or equivalently, its partial $F > 1.0$).
4. $R^2_{adj}$ can become negative when the model explains almost no variation and $n - k - 1$ is small.

---

Hypothesis Testing in Multiple Regression

1. Individual Slope Significance: $t$-Test

To test whether a specific independent variable $X_j$ has a statistically significant linear relationship with $Y$, we formulate hypotheses:

$$H_0: b_j = B_{j0} \quad \text{vs.} \quad H_a: b_j \neq B_{j0} \quad (\text{usually } B_{j0} = 0)$$

The test statistic is calculated as:

$$t = \frac{b_j - B_{j0}}{s_{b_j}}$$

Where $s_{b_j}$ is the estimated standard error of coefficient $b_j$. The test statistic follows a Student's $t$-distribution with $df = n - k - 1$ degrees of freedom.

• Decision Rule: Reject $H_0$ at significance level $\alpha$ if $|t| > t_{critical, \alpha/2, n-k-1}$ or if the $p$-value $< \alpha$.
• Confidence Interval: The $(1 - \alpha)\%$ confidence interval for the true population coefficient $\beta_j$ is:

$$b_j \pm \left(t_{critical, \alpha/2, n-k-1} \times s_{b_j}\right)$$

2. Overall Regression Significance: $F$-Test

The $F$-test evaluates whether at least one of the $k$ independent variables explains a significant portion of the variance in $Y$:

$$H_0: b_1 = b_2 = \dots = b_k = 0 \quad \text{vs.} \quad H_a: \text{At least one } b_j \neq 0$$

The test statistic is:

$$F = \frac{\text{MSR}}{\text{MSE}} = \frac{\text{SSR} / k}{\text{SSE} / (n - k - 1)} = \frac{R^2 / k}{(1 - R^2) / (n - k - 1)}$$

This follows an $F$-distribution with numerator degrees of freedom $df_1 = k$ and denominator degrees of freedom $df_2 = n - k - 1$. The $F$-test is always a one-tailed test in the upper tail.

• Decision Rule: Reject $H_0$ if $F > F_{critical, \alpha, k, n-k-1}$. Rejecting $H_0$ means the set of independent variables collectively explains a statistically significant proportion of the variance in $Y$.

---

Dummy Variables & Slope Interaction Terms

Qualitative Predictors (Intercept Dummies)

Qualitative attributes (e.g., sector, recession vs. expansion, January effect) are captured using dummy variables $D_i \in \{0, 1\}$:

$$Y_i = b_0 + b_1 X_{1i} + d_1 D_i + \epsilon_i$$

• When $D_i = 0$ (the baseline/reference group): $E(Y_i) = b_0 + b_1 X_{1i}$.
• When $D_i = 1$: $E(Y_i) = (b_0 + d_1) + b_1 X_{1i}$.
• The coefficient $d_1$ represents the differential intercept shift relative to the reference category.

The Dummy Variable Trap: If a qualitative factor has $m$ distinct categories, exactly $m - 1$ dummy variables must be included in a regression with an intercept. Including $m$ dummy variables alongside an intercept creates perfect collinearity (the sum of all dummies equals 1, matching the constant column), preventing matrix inversion.

Slope Interaction Terms

To test whether the slope coefficient of an independent variable differs across qualitative groups, an interaction term is included:

$$Y_i = b_0 + b_1 X_{1i} + d_1 D_i + d_2 (D_i \times X_{1i}) + \epsilon_i$$

• When $D_i = 0$: Intercept is $b_0$, Slope on $X_1$ is $b_1$.
• When $D_i = 1$: Intercept is $(b_0 + d_1)$, Slope on $X_1$ is $(b_1 + d_2)$.
• The parameter $d_2$ measures the differential slope between the two groups.

---

Functional Forms & Log Transformations

Non-linear relationships can frequently be estimated using OLS via logarithmic transformations of $Y$, $X$, or both:

---

Worked Numerical Example: Multi-Factor Equity Return Model

An equity analyst regresses the monthly excess returns ($R_{fund} - R_f$) of a hedge fund against $k = 3$ risk factors (Market excess return $MKT$, Small-minus-Big size factor $SMB$, and High-minus-Low value factor $HML$) over $n = 64$ monthly observations.

Given Regression Output:

• Total Sum of Squares (SST) = $1,250.00$
• Sum of Squared Errors (SSE) = $375.00$
• Intercept $b_0 = 0.45\%, s_{b0} = 0.18\%$
• Slope $b_{MKT} = 1.15, s_{b1} = 0.12$
• Slope $b_{SMB} = 0.40, s_{b2} = 0.16$
• Slope $b_{HML} = -0.30, s_{b3} = 0.20$

Step-by-Step Calculations:

1. Regression Sum of Squares (SSR):

   $$\text{SSR} = \text{SST} - \text{SSE} = 1,250.00 - 375.00 = 875.00$$

2. Degrees of Freedom:

   $$df_{\text{regression}} = k = 3$$ $$df_{\text{residual}} = n - k - 1 = 64 - 3 - 1 = 60$$ $$df_{\text{total}} = n - 1 = 64 - 1 = 63$$

3. Mean Squares and Standard Error of Estimate:

   $$\text{MSR} = \frac{\text{SSR}}{k} = \frac{875.00}{3} = 291.67$$ $$\text{MSE} = \frac{\text{SSE}}{n - k - 1} = \frac{375.00}{60} = 6.25$$ $$s_e = \sqrt{\text{MSE}} = \sqrt{6.25} = 2.50\%$$

4. Goodness of Fit ($R^2$ and $R^2_{adj}$):

   $$R^2 = \frac{\text{SSR}}{\text{SST}} = \frac{875.00}{1,250.00} = 0.7000 \quad (70.00\%$$ $$R^2_{adj} = 1 - \left[\frac{64 - 1}{64 - 3 - 1}\right](1 - 0.7000) = 1 - \left[\frac{63}{60}\right](0.3000) = 1 - (1.05)(0.3000) = 1 - 0.3150 = 0.6850 \quad (68.50\%$$

5. Overall Model Significance ($F$-Test):

   $$F = \frac{\text{MSR}}{\text{MSE}} = \frac{291.67}{6.25} = 46.67$$

   With $df_1 = 3, df_2 = 60$, the critical $F$ at $\alpha = 0.01$ is $4.13$. Since $46.67 > 4.13$, we reject $H_0: b_1 = b_2 = b_3 = 0$ at the $1\%$ significance level.

6. Individual $t$-Tests at $\alpha = 0.05$ ($df = 60, t_{crit} = 2.000$):

   • Alpha ($b_0$): $t = \frac{0.45 - 0}{0.18} = 2.50 > 2.000 \implies$ Statistically significant positive alpha.
   • Market ($b_{MKT}$): $t = \frac{1.15 - 0}{0.12} = 9.58 > 2.000 \implies$ Statistically significant.
   • Size ($b_{SMB}$): $t = \frac{0.40 - 0}{0.16} = 2.50 > 2.000 \implies$ Statistically significant.
   • Value ($b_{HML}$): $t = \frac{-0.30 - 0}{0.20} = -1.50$. Since $|-1.50| < 2.000$, we fail to reject $H_0$; the value factor is not statistically significant.