24.3 Hypothesis Testing

The Logic of Hypothesis Testing

Hypothesis testing is the formal framework for deciding whether observed sample evidence is compatible with a default claim about a population. The PA-CAT expects you to set up hypotheses, identify the correct test statistic, interpret a p-value, and distinguish Type I from Type II errors (PA-CAT Bulletin of Information, rev. 20240815, Table 11).

Null and Alternative Hypotheses

The null hypothesis (H₀) is the default claim, typically "no effect" or "no difference": μ = μ₀, p = p₀, μ₁ = μ₂. The alternative hypothesis (H₁ or Hₐ) is what the investigator hopes to find evidence for. H₁ may be:

• One-tailed (directional): μ > μ₀ or μ < μ₀. Used only when a directional claim is pre-specified and justified.
• Two-tailed (non-directional): μ ≠ μ₀. The default in most clinical research because an effect in either direction is scientifically meaningful.

The burden of proof falls on H₁. We either reject H₀ (evidence favors H₁) or fail to reject H₀ (insufficient evidence). We never "accept H₀"—absence of evidence is not evidence of absence.

Test Statistics

A test statistic measures how far the sample estimate falls from the value claimed by H₀, in standard-error units:

• z-statistic (mean, σ known): z = (x̄ − μ₀)/(σ/√n)
• t-statistic (mean, σ unknown): t = (x̄ − μ₀)/(s/√n), df = n − 1
• z-statistic (proportion): z = (p̂ − p₀)/√(p₀(1−p₀)/n)

Larger absolute values of the statistic indicate stronger evidence against H₀.

p-value and Significance Level

The p-value is the probability of obtaining a test statistic at least as extreme as the observed value, assuming H₀ is true. A small p-value means the observed data would be unusual under H₀. The significance level α is a pre-chosen cutoff (commonly 0.05) defining how unusual is "too unusual." If p ≤ α, reject H₀; if p > α, fail to reject. The p-value is not the probability that H₀ is true, nor the probability that the result is due to chance alone, nor the probability of a Type I error.

Type I and Type II Errors, Power

Because we decide on incomplete data, two errors are possible:

• α = P(Type I error) — the probability of declaring an effect when none exists. We set α; it is not computed from data.
• β = P(Type II error) — the probability of missing a real effect. β depends on sample size, effect size, variability, and α.
• Power = 1 − β — the probability of detecting a real effect of a given size. Conventionally, studies are designed for 80% or 90% power.

Power increases with larger sample size, larger true effect size, lower variability, and higher α. A nonsignificant result from an underpowered study does not confirm H₀; it reflects inadequate evidence.

Steps of Hypothesis Testing

1. State H₀ and H₁, choose α (typically 0.05) and one- or two-tailed.
2. Check assumptions (SRS, normality or large n, independence).
3. Compute the test statistic.
4. Find the p-value (or compare the statistic to the critical value).
5. Decide: reject H₀ if p ≤ α; otherwise fail to reject.
6. Interpret in context: "At the 0.05 level, there is sufficient evidence that the mean HbA1c reduction differs from 0."

Tests for a Mean and a Proportion

• One-sample t-test: tests H₀: μ = μ₀ against a sample mean; df = n − 1.
• One-proportion z-test: tests H₀: p = p₀; requires np₀ ≥ 10 and n(1−p₀) ≥ 10.

Example: A new lipid-lowering agent is tested in 36 patients. Mean LDL change is −18 mg/dL, s = 24. H₀: μ = 0; H₁: μ ≠ 0. t = −18/(24/√36) = −4.5, df = 35. The two-tailed p-value is < 0.001; reject H₀. There is strong evidence the drug lowers LDL.

Paired vs Independent Samples

Paired (dependent) samples arise when two measurements are taken on the same subject (before/after, two ears) or on matched pairs. The analysis uses the differences dᵢ = xᵢ − yᵢ and a one-sample t-test on the mean difference: t = d̄/(s_d/√n), df = n − 1. Example: diastolic blood pressure before and after a DASH diet in the same 20 patients.

Independent samples come from two separate groups. The two-sample t-test statistic is t = (x̄₁ − x̄₂)/√(s₁²/n₁ + s₂²/n₂). The conservative Welch (Satterthwaite) approximation for df is preferred on the PA-CAT when the two group variances may differ; the pooled version assumes equal variances.

Statistical Significance vs Practical Significance

A very large sample can produce a tiny p-value for a trivial effect. Always pair the hypothesis test with a confidence interval to judge practical importance: a mean reduction of 0.5 mm Hg may be statistically significant yet clinically irrelevant. The PA-CAT may frame items around exactly this distinction.