Hypothesis Testing Basics
Key Takeaways
- A hypothesis test compares sample evidence to a null hypothesis (H₀); you either reject H₀ or fail to reject H₀—you never “prove” H₀ true.
- Type I error (α) is rejecting a true H₀; Type II error (β) is failing to reject a false H₀; power is 1 − β, the probability of detecting a real effect.
- The significance level α is the maximum Type I risk you accept before the study (commonly 0.05); the p-value is the probability of data as extreme as observed (or more) if H₀ were true.
- Statistical significance (p ≤ α) is not the same as practical significance—always judge the size of the effect against process cost, specs, and business goals.
- Larger sample size increases power and can make tiny effects statistically significant; plan n with α, desired power, and the minimum effect you care about detecting.
Hypothesis Testing Basics
Quick Answer: Hypothesis testing is a structured decision method: state H₀ and Hₐ, pick α (Type I risk), collect a valid sample, compute a test statistic and p-value, then reject H₀ if p ≤ α (or the statistic falls in the critical region)—while still asking whether the effect is practically large enough to justify Improve actions.
Why Green Belts Use Hypothesis Tests
In Analyze, sponsors ask: Is line A really different from line B? Did the pilot change cycle time? A hypothesis test answers with a transparent error framework instead of gut feel on one snapshot. CSSGB BoK IV.B.1 is at the Apply level: set up the test correctly, interpret α, power, Type I/II errors, sample size, and statistical-vs-practical significance—not derive every distribution from first principles.
Null, Alternative, and Decision Language
| Symbol | Role | Typical wording |
|---|---|---|
| H₀ (null) | Status quo / no effect / no difference | “Mean cycle time is still 12 min”; “p_A = p_B” |
| Hₐ (alternative) | Claim you hope the data support | “Mean is less than 12”; “means differ” |
Decision outcomes
- Reject H₀ — sample evidence is strong enough (relative to α) against the null.
- Fail to reject H₀ — evidence is not strong enough. This is not proof that H₀ is true; the effect may be small, noise large, or n too small (low power).
Never write “accept H₀ as proven.” Exam language favors reject vs fail to reject.
One-sided vs two-sided Hₐ
- Two-sided: “different from” (μ ≠ μ₀)—common default when either direction matters.
- One-sided: “greater than” or “less than” only when the process story makes the opposite direction irrelevant before seeing the data. Do not flip to one-sided after peeking at results.
Significance Level (α), p-Value, and the Decision Rule
α (significance level) is chosen in advance. It is the long-run Type I error rate you tolerate when H₀ is true. Problems often use α = 0.05; high-cost false alarms may use 0.01; exploratory screening may use 0.10.
p-value (conceptually): probability of a test statistic at least as extreme as observed if H₀ were true. Small p-values mean the data are surprising under H₀.
Rule used throughout CSSGB practice
- If p ≤ α → reject H₀ (statistically significant at level α).
- If p > α → fail to reject H₀.
What p is not: not the probability H₀ is true, not the probability Hₐ is true, and not the size of the improvement. A p of 0.04 does not mean a 4% defect reduction.
Type I Error, Type II Error, and Power
| Error / metric | Definition | Process language |
|---|---|---|
| Type I (α) | Reject H₀ when H₀ is true | False alarm: declare a “real” difference that is only noise |
| Type II (β) | Fail to reject H₀ when Hₐ is true | Missed opportunity: a real shift exists but the test missed it |
| Power (1 − β) | Probability of correctly rejecting a false H₀ | Ability to detect a specified real effect |
Trade-offs:
- Lower α (stricter Type I control) generally raises β unless you increase sample size.
- Power rises when the true effect is larger, variation is smaller, n is larger, or α is larger.
- Plan n so power is adequate for the smallest effect worth finding, not only for convenient sample sizes.
α vs β risk example: A supplier change is expensive. A false alarm (Type I) may force unnecessary requalification; missing a true quality drop (Type II) ships defects. Choose α with stakeholders, then plan n for the shift you care about.
Sample Size and Detectable Effects
For fixed α and assumed σ (or baseline proportion):
- Larger n → smaller standard error → higher power for a given true difference.
- Larger n can also make a tiny difference statistically significant—hence practical significance.
- Small n → often fail to reject even when a useful effect exists (underpowered study).
Green Belt practice: estimate σ from Measure, state the minimum mean or proportion difference that matters economically, pick α and target power (often ~0.80–0.90), then size n. If n is infeasible, reduce measurement noise (MSA) or accept lower power knowingly.
Statistical vs Practical Significance
| Concept | Question answered |
|---|---|
| Statistical significance | Is the sample result unlikely under H₀ at the chosen α? |
| Practical significance | Is the estimated effect large enough for cost, risk, CTQs, or customer impact? |
Example: n = 5,000 fills; mean is 0.02 mL above target; p = 0.001. Statistically significant, but if specs allow ±2 mL and change cost is huge, practical impact may be negligible. Conversely, p = 0.08 with a large estimated yield gain and modest n may justify a better-powered confirmation—not automatic rejection of the idea.
Always report: decision on H₀, estimated effect size, and business recommendation.
Green Belt Decision Framework
- Define the business question in Y and X language.
- Write H₀ / Hₐ and one- vs two-sided before analysis.
- Set α and, when planning, power and n.
- Check assumptions (independence, normality/robustness, equal variance when required, data type).
- Run the test; apply p ≤ α.
- Translate into process action—never stop at “significant.”
- Confirm important claims with control charts, capability, or DOE before full-scale Improve.
Pitfalls and Exam Focus
- Confusing “fail to reject” with “proved equal.”
- Treating α = 0.05 as a law rather than a risk choice.
- Chasing significance by retesting the same data many ways without care.
- Ignoring power when a pilot “shows no difference.”
- Equating a small p-value with a large or causal effect.
CSSGB items at IV.B.1 typically ask you to name Type I/II, link α to false alarms, connect power to sample size and effect size, and choose correctly when statistical and practical significance diverge.
A Green Belt tests whether a new supplier’s mean hardness differs from the current mean. With α = 0.05, the two-sample t-test returns p = 0.12. Which statement is most correct?
A call-center project finds a 2-second reduction in handle time with p = 0.001 (n very large). Leadership asks whether to fund a multi-million-dollar system change for that gain. What should the Green Belt emphasize?