6.3 Linear Regression

Linear Regression

Simple linear regression models a dependent variable as a straight-line function of one regressor plus an error:

Y_i = α + β X_i + ε_i

In risk applications, Y might be a stock’s excess return and X the market excess return (CAPM-style), or Y a loss rate and X a macro factor. FRM Part I expects you to estimate, interpret, and test β.

OLS With a Single Regressor

Ordinary least squares (OLS) picks α̂ and β̂ to minimize Σ ε̂_i² where ε̂_i = Y_i − α̂ − β̂ X_i.

Closed forms:

β̂ = Σ (x_i − x̄)(y_i − ȳ) / Σ (x_i − x̄)² = Cov̂(X, Y) / Var̂(X)

α̂ = ȳ − β̂ x̄

Fitted values: ŷ_i = α̂ + β̂ x_i. Residuals are orthogonal to the constant and to X in the sample (normal equations).

Worked example

X (factor): 1, 2, 3, 4 Y (return): 2, 2, 4, 6

x̄ = 2.5, ȳ = 3.5 Σ(x − x̄)(y − ȳ) = (−1.5)(−1.5) + (−0.5)(−1.5) + (0.5)(0.5) + (1.5)(2.5) = 2.25 + 0.75 + 0.25 + 3.75 = 7 Σ(x − x̄)² = 2.25 + 0.25 + 0.25 + 2.25 = 5 β̂ = 7/5 = 1.4 α̂ = 3.5 − 1.4 × 2.5 = 0

So Ŷ = 0 + 1.4 X. At X = 3, ŷ = 4.2; residual = 4 − 4.2 = −0.2.

Classical Assumptions (Simple Regression)

A standard teaching set (Gauss–Markov plus extras for exact inference):

1. Linearity in parameters: true CEF is α + βX (or model is otherwise correctly specified as linear in α, β).
2. Strict exogeneity / zero conditional mean: E[ε | X] = 0 (no omitted confounding correlated with X; regressor variation is “as good as exogenous” for the effect of interest).
3. Homoskedasticity: Var(ε | X) = σ² constant.
4. No serial correlation (in cross-section: independent errors; in time series: uncorrelated errors across i).
5. No perfect collinearity: Var(X) > 0 (X is not constant).
6. Often for exact normal t/F tests: ε | X ~ Normal(0, σ²). Asymptotically, CLT delivers approximate normality of β̂ without normal errors if other conditions hold and n is large.

Under 1–5, OLS is BLUE. Normality (6) justifies exact small-sample t tests; without it, use large-sample (robust) reasoning.

Sampling Distribution of the Slope

Under classical assumptions with fixed regressors (or conditioning on X):

E[β̂] = β

Var(β̂) = σ² / Σ(x_i − x̄)²

More spread-out X → smaller variance of β̂ (stronger experimental design). Estimate σ² with

s² = Σ ε̂_i² / (n − 2)

(two parameters estimated: α and β). Then

SE(β̂) = s / √[Σ(x_i − x̄)²]

t = (β̂ − β₀) / SE(β̂) ~ t_{n−2} under H₀ when errors are normal.

Worked example: SE and t

Continue the toy fit Ŷ = 1.4X, n = 4 (tiny—for method only). Fitted values are 1.4, 2.8, 4.2, 5.6, so residuals are +0.6, −0.8, −0.2, +0.4 and Σ ε̂_i² = 0.36 + 0.64 + 0.04 + 0.16 = 1.2. Then s² = 1.2/(4 − 2) = 0.6, s ≈ 0.775. Σ(x − x̄)² = 5, SE(β̂) = 0.775/√5 ≈ 0.346. Test H₀: β = 0: t = 1.4/0.346 ≈ 4.04. Even with n − 2 = 2 df, this is large—but never trust inference from n = 4 in practice; FRM wants the mechanics.

Realistic sketch: n = 60, β̂ = 1.10, SE = 0.20. t = 1.10/0.20 = 5.5 → decisively reject β = 0 at standard levels.

Coefficient Tests, p-Values, and Confidence Intervals

For H₀: β = β₀ (often β₀ = 0):

• Compute t = (β̂ − β₀)/SE(β̂).
• Two-sided p-value from t_{n−2} (or normal approximation for large n).
• (1 − α) CI: β̂ ± t_{α/2, n−2} · SE(β̂).

Trinity of equivalence (two-sided): reject H₀: β = β₀ at level α ⟺ p ≤ α ⟺ β₀ outside the (1 − α) CI.

Worked CI

β̂ = 0.85, SE = 0.25, n large so use z ≈ 1.96 for 95%. CI: 0.85 ± 1.96 × 0.25 = 0.85 ± 0.49 = [0.36, 1.34]. H₀: β = 0 rejected; H₀: β = 1 not rejected (1 is inside the CI). Same story from t for β = 1: (0.85 − 1)/0.25 = −0.6.

R² and Correlation

Total sum of squares TSS = Σ (y_i − ȳ)² Explained sum of squares ESS = Σ (ŷ_i − ȳ)² Residual sum of squares RSS = Σ (y_i − ŷ_i)²

TSS = ESS + RSS (with intercept).

R² = ESS/TSS = 1 − RSS/TSS

In simple linear regression with an intercept:

R² = r_{X,Y}²

where r_{X,Y} is the sample correlation between X and Y. If r = −0.6, R² = 0.36: 36% of sample variation in Y is linearly associated with X. Sign of dependence lives in β̂ (and r), not in R².

High R² is not the same as a causal story, nor does it guarantee good forecasting out of sample. Low R² is common in daily return regressions even when β is estimated precisely (because residual variance is large).

Interpreting β in Risk Contexts

If Y is asset excess return and X is market excess return, β̂ ≈ 1.2 means the asset moved about 1.2% for a 1% market move on average in the sample, under the linear model. That is a statistical association from OLS—not automatic proof of equilibrium CAPM pricing. Still, testing H₀: β = 1 (market-like) versus H₀: β = 0 (no linear market link) uses exactly the t/CI tools above.

Common Traps

• Confusing statistical significance of β with economic size of β.
• Using R² to “validate” a trading signal without out-of-sample tests.
• Ignoring that heteroskedasticity/autocorrelation invalidate textbook SEs.
• Forgetting df = n − 2 in simple regression.
• Claiming R² = r when r is negative—R² = r² always in the simple intercept case.

Simple OLS is the backbone for multiple regression next: same least-squares idea, richer interpretation and joint tests when several X’s enter together.