7.6 Sample Size for Hypothesis Tests

The four inputs

Every sample size calculation needs the same four quantities, and three of them are business decisions rather than statistical ones.

The most common failure is treating $\delta$ as a statistical parameter. It is not: it is the smallest difference worth detecting, and it comes from the economics of the process. If a 2% yield improvement is the smallest change that would justify implementation, size the study to detect 2%.

Standard normal values needed:

Comparing two means

For a two-sided test of $H_0: \mu_1 = \mu_2$ with equal group sizes:

$$n = \frac{2(z_{\alpha/2} + z_{\beta})^2 \sigma^2}{\delta^2} \quad \text{per group}$$

At $\alpha = 0.05$ two-sided and 80% power, $(1.960 + 0.842)^2 = 7.85$, so a useful shortcut is

$$n \approx \frac{15.7\,\sigma^2}{\delta^2} \quad \text{per group}$$

Worked example. A team wants to detect a 1.5-unit shift in mean cycle time. Baseline data gives $\sigma = 3.0$ units. At $\alpha = 0.05$ and 80% power:

$$n = \frac{2(1.960 + 0.842)^2 (3.0)^2}{(1.5)^2} = \frac{2 \times 7.851 \times 9}{2.25} = \frac{141.3}{2.25} = 62.8 \rightarrow 63 \text{ per group}$$

Always round up. Raising power to 90% changes the multiplier to $(1.960 + 1.282)^2 = 10.51$ and the requirement to 85 per group -- a 35% increase in data for a 10 percentage point gain in power.

One sample against a target

$$n = \frac{(z_{\alpha/2} + z_{\beta})^2 \sigma^2}{\delta^2}$$

Exactly half the two-sample requirement, because only one mean is being estimated.

When sigma is unknown and n is small

These formulas use $z$ values, which assumes $\sigma$ is known. When $\sigma$ is estimated from the sample, the $t$ distribution applies and the requirement is slightly larger. The standard practical fix is to compute with $z$, then add two to four observations per group, or iterate using $t$ values at the resulting degrees of freedom. For $n$ above about 30 per group the difference is negligible.

Comparing two proportions

$$n = \frac{(z_{\alpha/2} + z_{\beta})^2 \left[p_1(1-p_1) + p_2(1-p_2)\right]}{(p_1 - p_2)^2} \quad \text{per group}$$

Worked example. Current defect rate is 5%, and the team wants to detect a reduction to 3%. At $\alpha = 0.05$ two-sided and 80% power:

• $p_1(1-p_1) = 0.05 \times 0.95 = 0.0475$
• $p_2(1-p_2) = 0.03 \times 0.97 = 0.0291$
• Sum $= 0.0766$; $(p_1 - p_2)^2 = (0.02)^2 = 0.0004$

$$n = \frac{7.851 \times 0.0766}{0.0004} = \frac{0.6014}{0.0004} = 1{,}503 \text{ per group}$$

Just over 1,500 units per group -- 3,000 units in total -- to detect a 2 percentage point change. This single result explains why continuous data is preferred wherever it is available: a comparable continuous study typically needs a few dozen observations per group.

One sample against a target proportion

$$n = \frac{(z_{\alpha/2}\sqrt{p_0(1-p_0)} + z_{\beta}\sqrt{p_1(1-p_1)})^2}{(p_1 - p_0)^2}$$

The inverse-square relationship

Because $\delta$ appears squared in the denominator, sensitivity is expensive:

The same relationship works in your favour on the other side: halving $\sigma$ quarters the required sample. Improving the measurement system is therefore often cheaper than collecting more data. If gage R&R contributes a large share of the observed variance, fixing the gauge reduces $\sigma$ and shrinks every subsequent sample size requirement.

$$\sigma^2_{observed} = \sigma^2_{process} + \sigma^2_{measurement}$$

Power analysis after the fact

If a study has already been run and returned a non-significant result, compute the difference it was actually able to detect:

$$\delta_{detectable} = (z_{\alpha/2} + z_{\beta})\,\sigma\sqrt{\frac{2}{n}}$$

With $n = 12$ per group and $\sigma = 3.0$:

$$\delta = 2.802 \times 3.0 \times \sqrt{2/12} = 2.802 \times 3.0 \times 0.408 = 3.43$$

The study could only detect a shift of 3.4 units. Reporting "no significant difference" when the practically important difference was 1.5 units is misleading; the honest statement is that the study was inconclusive.

Practical adjustments

Sizing checklist

1. Is $\delta$ the smallest practically important difference, agreed with the process owner?
2. Is $\sigma$ from real baseline data on this process, not a textbook value?
3. Does $\sigma$ include measurement variation, and would improving the gauge be cheaper than more data?
4. Are $\alpha$ and power chosen from the relative cost of the two errors?
5. Has $n$ been rounded up and inflated for expected loss?
6. Is collecting that many observations feasible in the project timeline? If not, renegotiate $\delta$ or power before collecting, not after.