6.2 Hypothesis Testing

Hypothesis Testing

Risk managers and FRM candidates continually ask: is this mean return different from zero? Did two desks’ loss rates change? Is a hedge ratio equal to one? Hypothesis testing turns those questions into a controlled error framework.

Null and Alternative Hypotheses

• Null hypothesis H₀: the baseline claim (often “no effect,” “no difference,” or a regulatory threshold). You either reject H₀ or fail to reject H₀—you do not “prove” H₀ true.
• Alternative H₁ (or H_a): what you entertain when H₀ looks untenable.

Examples:

• H₀: μ = 0 versus H₁: μ ≠ 0 (two-sided alpha test for mean excess return).
• H₀: μ ≤ 0 versus H₁: μ > 0 (one-sided; only large positive t-stats reject).

The alternative determines the rejection region. Mixing up one-sided versus two-sided critical values is a frequent exam error.

One-Sided Versus Two-Sided Tests

Two-sided (H₁: θ ≠ θ₀): reject for very small or very large test statistics. For a normal/z test at α = 0.05, critical values are about ±1.96.

One-sided (H₁: θ > θ₀ or θ < θ₀): all of α sits in one tail. At α = 0.05, the critical z is about +1.645 (upper) or −1.645 (lower).

Use one-sided only when the research or risk question truly rules out the other direction before seeing the data. Choosing the side after peeking at the sign of the estimate inflates Type I error.

Type I, Type II, Size, and Power

• Type I error: reject H₀ when H₀ is true. Size (significance level) α = P(Type I).
• Type II error: fail to reject H₀ when H₁ is true. β = P(Type II) depends on the specific alternative, n, and α.
• Power = 1 − β = P(reject H₀ | specific H₁ is true).

Trade-off: lowering α (fewer false alarms) typically raises β (more misses) unless you increase sample size. Larger |μ − μ₀|, larger n, and smaller σ all raise power.

Worked example: size versus power intuition

Test H₀: μ = 0 vs H₁: μ > 0 with z = X̄ / (σ/√n), σ = 10, n = 25, α = 0.05. Critical value: X̄ ≥ 1.645 × (10/5) = 3.29.

If the true mean is μ = 5, then X̄ ~ approx N(5, 2²). Power ≈ P(X̄ ≥ 3.29 | μ = 5) = P(Z ≥ (3.29 − 5)/2) = P(Z ≥ −0.855) ≈ 0.80. If instead true μ = 2, power falls because 2 is closer to the null—same α, harder to detect.

Confidence Intervals and Tests

For the usual two-sided test of H₀: μ = μ₀ with known σ (z-test):

(1 − α) CI: x̄ ± z_{α/2} · (σ/√n)

Equivalence: reject H₀ at level α ⟺ μ₀ lies outside the (1 − α) CI.

If a 95% CI for mean daily P&L is [0.4, 1.2], then H₀: μ = 0 is rejected at 5% two-sided; H₀: μ = 0.7 is not. This link is a high-yield FRM fact for mean tests and coefficient tests in regression.

Worked calculation

n = 36, x̄ = 2.0, σ = 6, α = 0.05 two-sided.

SE = 6/6 = 1. CI: 2 ± 1.96 × 1 = [0.04, 3.96].

z for H₀: μ = 0 is (2 − 0)/1 = 2.0 > 1.96 → reject. Indeed 0 is outside the CI. For H₀: μ = 1, z = 1.0 < 1.96 → do not reject; 1 ∈ [0.04, 3.96].

p-Values

The p-value is the probability, computed under H₀, of obtaining a test statistic at least as extreme (toward H₁) as the one observed.

• Small p-value → data are surprising under H₀ → reject at common α levels if p ≤ α.
• p-value is not P(H₀ true | data).
• p-value is not the probability of Type I error for this specific experiment after seeing the data; α is chosen before the test as the long-run Type I rate under H₀.

For two-sided z = 2.0, p ≈ 2 × P(Z ≥ 2) ≈ 2 × 0.0228 = 0.0456 < 0.05 → reject at 5%. For one-sided upper test with the same z = 2.0, p ≈ 0.0228.

Tests for Two Means

Independent samples, common known variances (or large-n with estimated SEs):

H₀: μ_X − μ_Y = 0, z = (x̄ − ȳ) / √(σ_X²/n_X + σ_Y²/n_Y)

If variances unknown but assumed equal, use a pooled-variance t-statistic; if unequal, use Welch-style SE. FRM questions often give SEs or variances explicitly—plug in carefully.

Paired samples (same firms before/after, matched books): work with differences D_i = X_i − Y_i and run a one-sample test on μ_D. Pairing removes shared noise and can raise power.

Worked example: two independent means

Desk A: n = 40, x̄ = 1.5, s = 2. Desk B: n = 40, ȳ = 0.5, s = 2. Treat s as σ for a large-sample z.

SE(diff) = √(4/40 + 4/40) = √0.2 ≈ 0.447. z = (1.5 − 0.5)/0.447 ≈ 2.24.

Two-sided p ≈ 0.025 → reject equal means at 5%. A 95% CI for μ_A − μ_B: 1.0 ± 1.96 × 0.447 ≈ [0.12, 1.88], which excludes 0—same decision.

Multiple Testing Bias

If you run m independent tests each at α = 0.05, and all nulls are true, the probability of at least one false rejection is 1 − (1 − α)^m ≈ mα for small α (union bound / Bonferroni intuition).

Multiple testing bias (data mining / p-hacking): scanning many desks, factors, or cutoffs and reporting only “significant” hits produces a flood of Type I errors. Mitigations include Bonferroni (use α/m), FDR control, pre-registration of hypotheses, and out-of-sample validation—central in quant research and model validation culture.

Worked sketch

A researcher tests 40 factor premia at α = 0.05 with no adjustment. Expected false positives under global null ≈ 40 × 0.05 = 2. Finding “two significant factors” is unsurprising noise. Bonferroni would require p ≤ 0.05/40 = 0.00125.

Practical FRM Checklist

1. State H₀/H₁ and whether the test is one- or two-sided before computing the statistic.
2. Choose α (size) knowing the Type I / Type II trade-off.
3. Compute the statistic and p-value; cross-check with the CI when testing a point null.
4. For two groups, pick independent vs paired formulas deliberately.
5. If many hypotheses were tried, discount unadjusted significance.

Hypothesis testing does not replace economic judgment: a statistically significant 0.1 bp mean difference may be irrelevant, and a noisy but large risk difference may fail to reject with short samples. Still, size, power, p-values, and CI equivalence are the grammar of quantitative FRM inference.