7.9 Analysis of Variance

Why not repeated t-tests

With $k$ groups there are $k(k-1)/2$ pairwise comparisons. For five groups that is ten tests, and at $\alpha = 0.05$ per test the family-wise error rate is $1 - 0.95^{10} = 0.40$. ANOVA answers the question with a single test at the stated $\alpha$.

The partition

$$SS_{total} = SS_{treatment} + SS_{error}$$ $$SS_{total} = \sum_{i}\sum_{j}(y_{ij} - \bar{y})^2, \qquad SS_{treat} = \sum_{i} n_i(\bar{y}_i - \bar{y})^2, \qquad SS_{error} = \sum_{i}\sum_{j}(y_{ij} - \bar{y}_i)^2$$

The logic: if the group means were all equal, the only reason group averages would differ is random error, so between-group variation would be about the same size as within-group variation. If treatment means genuinely differ, between-group variation is inflated.

The ANOVA table

where $k$ is the number of groups and $N$ the total number of observations. Reject $H_0: \mu_1 = \mu_2 = \cdots = \mu_k$ when $F > F_{\alpha,\,k-1,\,N-k}$.

Note that $MS_{error}$ is the pooled estimate of the common within-group variance $\sigma^2$, and its square root is the standard deviation used for post-hoc comparisons.

Worked example

Four suppliers, five specimens each, tensile strength in MPa.

Grand mean $\bar{y} = 419.75$, $N = 20$, $k = 4$.

$$SS_{treat} = 5\left[(416.0 - 419.75)^2 + (429.0 - 419.75)^2 + (411.0 - 419.75)^2 + (423.0 - 419.75)^2\right]$$ $$= 5\left[14.06 + 85.56 + 76.56 + 10.56\right] = 5 \times 186.75 = 933.75$$

Within-group sums of squares are 50.0, 38.0, 38.0, and 34.0, giving $SS_{error} = 160.0$.

$F_{0.05,3,16} = 3.24$, and $31.13 \gg 3.24$, so at least one supplier mean differs. The effect size is

$$\eta^2 = \frac{SS_{treat}}{SS_{total}} = \frac{933.75}{1093.75} = 0.854$$

so supplier accounts for 85.4% of the variation in tensile strength.

After a significant F

A significant $F$ says at least one mean differs; it does not say which. Post-hoc procedures control the family-wise error rate while making the pairwise comparisons:

Running post-hoc tests without a significant omnibus F reintroduces the error-rate inflation that ANOVA exists to prevent.

Assumptions

ANOVA is robust to moderate non-normality when the design is balanced, but it is not robust to dependence. Time-ordered data with autocorrelation, or observations clustered within batches, violate the assumption in a way no transformation fixes; the design must change.

Bartlett's test for equal variances is itself very sensitive to non-normality, so Levene's test is preferred in practice. Where variances differ materially, use Welch's ANOVA or transform the response. When normality fails badly, the Kruskal-Wallis test is the non-parametric alternative.

Residual analysis is the practical way to check all three at once: plot residuals against fitted values (equal variance), on a normal probability plot (normality), and against run order (independence).

Beyond one-way

The two-way case introduces the interaction term, which tests whether the effect of one factor depends on the level of the other. Interactions are frequently the most important finding in a Six Sigma analysis, because they are invisible to one-factor-at-a-time experimentation. This is the direct link between ANOVA and the Improve phase: the analysis of a factorial experiment is an ANOVA.