9.5 Sample Size Determination for Hypothesis Tests

9.5 Sample Size Determination for Hypothesis Tests

Quick Summary: Sections 9.1–9.4 asked "given this sample, what can I conclude?" This section asks the question that must be answered first: "how large a sample do I need before the test is worth running?" Too small and a real improvement disappears into noise. Too large and the team burns weeks collecting data to detect differences no customer would notice. CSSC devotes an entire Body of Knowledge chapter to it, and the Measure tollgate requires that "the team has conducted appropriate sampling to allow for statistically valid conclusions in the next phase."

---

The Four Inputs

Every sample size calculation needs the same four pieces of information. Three are risk decisions the team makes; one is a property of the process.

Choosing alpha

Ask what the cost is of concluding that a statistical change occurred when it did not. In a non-manufacturing setting: are there dangers or costs associated with concluding a change occurred, and what are they?

Choosing beta

Ask what the potential cost of a Type II error is — of failing to reject a false null. CSSC's own framing is memorable: if a defective spoon is passed to the customer, the ultimate cost is minimal compared with a defective piece of medical equipment or a car engine. Also ask whether there are costs in lost time or resources from correcting a problem that arises out of a Type II error.

Choosing delta

This is where candidates lose points, because delta is a business decision dressed in statistical clothing. The manual's questions:

• How small does the difference have to be before it becomes practically insignificant to the customer? The illustration is a pizza restaurant: a customer who orders a 14-inch pizza will care about receiving a 12-inch one, but is unlikely to care about 13.9 inches.
• What is the smallest delta that exposes all benefits or information but is not so small as to be unfeasible? Smaller deltas mean larger sample sizes or tighter measurement requirements, and both take time to gather.
• Ultimately: what margin of error is tolerable in the results? The test type, the decisions being made, and the process itself determine that.

CSSC states the rule directly: the delta, or critical difference, should always be based on business needs. It also notes that in Minitab it is helpful to express delta in terms of sigma — "we want to detect differences of $0.5\sigma$ or $2\sigma$."

[!IMPORTANT] Statistical significance is not practical significance. A large enough sample will declare a $0.1$-inch pizza difference "significant." That is the machinery working correctly and the analyst asking the wrong question. Delta is the guardrail that keeps the two apart.

---

CSSC's Default Guidelines

Where business needs give no specific guidance, the manual publishes baseline values. Note the pattern: alpha and beta are stable across data types; only delta changes.

A beta of 0.10 yields power 0.90; a beta of 0.20 yields power 0.80. CSSC's note on the trade-off is exact: a power of 0.90 is less likely to produce a Type II error than a power of 0.80, but it will also require a much larger sample size in most cases.

The survey-sampling row is the odd one out and is worth memorizing precisely. General sampling on an unknown population requires a different approach: the beta of 0.5 and unit standard deviation produce an initial sample size that lets you understand the population, and additional sampling may be required based on the statistics calculated from that first sample. It is a two-stage procedure, not a one-shot calculation.

---

The Nine Calculation Methods

Just as there are numerous hypothesis tests for different data types and questions, there are numerous sample size calculations. CSSC lists them with selection criteria; all can be performed in Minitab under Stat > Power and Sample Size.

The selection logic reduces to three questions, in order: (1) Am I testing means, variance, or proportions? (2) How many data sets — one against a target, two, or more than two? (3) If more than two, how many factors act on x? One factor sends you to ANOVA; more than one sends you to DOE.

[!NOTE] A convention watch-out on 1-Sample T versus 1-Sample Z. CSSC's stated cue is reproduced above exactly as the manual gives it. Be aware that in general statistical practice the convention runs the other way: the Z test is used when the population standard deviation is known, and the t test when it is not and must be estimated from the sample. On an open-book CSSC item, answer from the manual's own wording. On the job, and on ASQ or IASSC items, apply the standard convention.

---

Worked Calculation 1: Sizing a Mean Estimate

Scenario. A Green Belt is baselining call-handling time at a service desk. Historical data gives $\sigma = 12$ minutes. Leadership wants the baseline mean estimated to within $\pm 2$ minutes at 95% confidence.

For estimating a mean to a stated margin of error $E$:

$n = \left( \frac{Z_{\alpha/2} \times \sigma}{E} \right)^2$

$n = \left( \frac{1.96 \times 12}{2} \right)^2 = (11.76)^2 = 138.3$

Always round up. The team needs $n = 139$ calls. Note the inverse-square behaviour: halving the margin of error to $\pm 1$ minute would require $(23.52)^2 = 553.2 \rightarrow 554$ calls, a fourfold increase.

---

Worked Calculation 2: Sizing a Two-Sample Comparison

Scenario. The team wants to know whether a revised script changes handling time. It will compare the revised script against the current one and considers a $1\sigma$ shift the smallest business-relevant difference. Following CSSC's guidelines: $\alpha = 0.05$ (so $Z_{\alpha/2} = 1.96$) and $\beta = 0.20$, giving power $0.80$ (so $Z_\beta = 0.84$).

For a two-sample test of means with equal group sizes:

$n_{\text{per group}} = \frac{2(Z_{\alpha/2} + Z_\beta)^2 \sigma^2}{\delta^2}$

With $\delta = 1\sigma$, the $\sigma^2$ terms cancel:

$n_{\text{per group}} = 2(1.96 + 0.84)^2 = 2(2.80)^2 = 2(7.84) = 15.68 \rightarrow \mathbf{16 \text{ per group}}$

Total sample: 32 calls, 16 under each script.

The cost of more power

Rerun the same problem at $\beta = 0.10$ (power 0.90, $Z_\beta = 1.28$):

$n_{\text{per group}} = 2(1.96 + 1.28)^2 = 2(3.24)^2 = 2(10.4976) = 20.99 \rightarrow \mathbf{21 \text{ per group}}$

Raising power from 0.80 to 0.90 costs 5 extra observations per group here — about 31% more data. That is CSSC's "much larger sample size" caution in numbers, and the effect grows sharply as delta shrinks.

The cost of a smaller delta

Hold power at 0.80 and halve delta to $0.5\sigma$:

$n_{\text{per group}} = \frac{2(2.80)^2}{(0.5)^2} = \frac{15.68}{0.25} = 62.72 \rightarrow \mathbf{63 \text{ per group}}$

Halving delta quadrupled the sample. Delta, not alpha, is the dominant cost driver — which is exactly why CSSC insists it be set from business need rather than reflexively made small.

---

Worked Calculation 3: Sizing a Proportion Study

Scenario. A billing department's historical rework rate is about $p = 0.10$. The team wants to estimate the current rate to within $\pm 3$ percentage points at 95% confidence.

$n = \frac{Z_{\alpha/2}^2 \times p(1 - p)}{E^2}$

$n = \frac{(1.96)^2 \times (0.10)(0.90)}{(0.03)^2} = \frac{3.8416 \times 0.09}{0.0009} = \frac{0.345744}{0.0009} = 384.2 \rightarrow \mathbf{385 \text{ invoices}}$

Compare that with the 139 calls needed for the continuous measurement in Worked Calculation 1. This is the same lesson Section 6.1 delivered from the other direction: attribute data is statistically expensive. Whenever a continuous measurement is available, taking it reduces the sample size dramatically.

[!TIP] When $p$ is unknown. Use $p = 0.5$. That maximizes $p(1-p)$ at $0.25$ and therefore produces the largest — most conservative — sample size. Here it would give $n = 3.8416 \times 0.25 / 0.0009 = 1{,}067.1 \rightarrow 1{,}068$.

---

Realistic Exam Scenario

A Green Belt at a packaging plant is asked to confirm whether a new sealing head reduces leak rate. Baseline leak rate is 2%. The plant manager says a reduction below 1.5% would be worth the capital, and the plant produces 4,000 units per shift.

Work the four inputs in order. Proportions, not means, so the 1-Sample or 2-Sample Proportion calculation applies. Two data sets are being compared — old head versus new — so it is 2-Sample Proportion. Delta is not "any detectable reduction"; it is the business-defined 0.5 percentage points. Alpha and beta take the CSSC defaults of 0.05 and 0.20 unless leak escapes carry safety consequences, in which case beta should drop to 0.10 because the Type II error — shipping a leaking package — is expensive.

The instructive part is what the numbers then reveal: detecting a half-point shift in a 2% rate requires thousands of units per arm. The right recommendation may be to run the comparison over several shifts, or to find a continuous surrogate for leak resistance such as seal peel strength, which would collapse the sample size by an order of magnitude.

---

Common Exam Traps

• Trap 1: Rounding sample size down. Always round up to the next whole unit. A calculation of 138.3 means 139.
• Trap 2: Setting delta as small as possible. Halving delta roughly quadruples the sample. Delta comes from business need — the 13.9-inch pizza is not a defect.
• Trap 3: Confusing beta with power. Power $= 1 - \beta$. A beta of 0.10 is power 0.90, and CSSC's guideline offers 0.10 or 0.20.
• Trap 4: Applying the standard formula to survey sampling. General sampling of an unknown population uses $\alpha = 0.05$, $\beta = 0.5$, delta matching $\sigma = 1$, and is a two-stage procedure requiring additional sampling afterward.
• Trap 5: Choosing ANOVA when more than one factor acts on x. More than two data sets with one factor is ANOVA; more than two data sets with more than one factor is DOE.
• Trap 6: Forgetting delta must exceed 1 for variance testing. That is the one delta guideline unique to variance tests.