7.5 Statistical Versus Practical Significance
Key Takeaways
- Statistical significance means the observed effect is unlikely under the null hypothesis; practical significance means the effect is large enough to justify action.
- A p-value is a function of both effect size and sample size, so significance can always be manufactured by collecting more data.
- Effect size measures such as Cohen's d and the difference in means with a confidence interval express magnitude independently of sample size.
- A non-significant result from an underpowered study is not evidence that no effect exists.
- The four combinations of statistical and practical significance require four different responses, and only one of them justifies implementation.
Two different questions
| Statistical significance | Practical significance | |
|---|---|---|
| Question | Is the observed effect likely to be real, or could it be sampling variation? | Is the effect large enough to matter? |
| Answered by | p-value against $\alpha$ | Effect size compared with a threshold set by the business |
| Depends on | Effect size and sample size and variability | Effect size and the economics of the process |
| Who decides | The statistical procedure | The process owner, the customer, or finance |
The critical asymmetry: statistical significance is computed, practical significance is judged, and the judgment must be made by someone who knows what the process is worth.
Why sample size drives the p-value
For a two-sample comparison of means the test statistic is roughly
The numerator is the effect. The denominator shrinks as $n$ grows. Hold the effect constant and increase $n$, and $t$ grows without bound -- so any non-zero difference eventually becomes statistically significant.
Worked illustration. Two filling lines differ in mean fill weight by 0.05 g, with $s = 1.2$ g on a product with a tolerance of $\pm 8$ g.
| $n$ per group | Approximate $t$ | p-value | Statistically significant at 0.05? |
|---|---|---|---|
| 30 | 0.16 | 0.87 | No |
| 500 | 0.66 | 0.51 | No |
| 5,000 | 2.08 | 0.037 | Yes |
| 50,000 | 6.59 | < 0.001 | Yes |
At 50,000 units per group the difference is overwhelmingly significant and completely irrelevant: 0.05 g against an 8 g tolerance is 0.6% of the specification width. Nobody should spend a dollar closing it.
Measuring effect size
Report magnitude, not just the p-value.
Difference in means with a confidence interval. The most direct and the most useful. "Mean cycle time fell by 1.8 days, 95% CI 1.1 to 2.5 days" tells the reader the size, the direction, and the precision in the units of the problem.
Cohen's d, the standardized difference:
| $d$ | Conventional label |
|---|---|
| 0.2 | Small |
| 0.5 | Medium |
| 0.8 | Large |
Proportion of variance explained. $r^2$ in regression, or eta-squared in ANOVA: the share of output variation attributable to the factor.
Practical thresholds derived from the process. The most defensible option where they exist:
- A shift smaller than the measurement system's uncertainty cannot be acted on reliably.
- A shift smaller than a set fraction of the tolerance is not worth chasing.
- A saving below the organization's project threshold does not justify the implementation cost.
Set the practical threshold before the analysis, in the charter. Setting it afterwards invites choosing the threshold that makes the result look good.
The four combinations
| Practically significant | Not practically significant | |
|---|---|---|
| Statistically significant | Act. Real and worth doing. | Do not act. Real but trivial; usually a symptom of a very large sample. |
| Not statistically significant | Investigate further. Potentially valuable but not yet demonstrated -- check power and collect more data. | Stop. Neither real nor important. |
The bottom-left cell is the one teams handle worst. A promising effect that fails to reach significance in an underpowered study is frequently reported as "no difference", and the opportunity is discarded. The correct response is to compute the power the study actually had, and if it was low, treat the result as inconclusive rather than negative.
What a p-value is not
| Misinterpretation | Correction |
|---|---|
| "p = 0.03 means there is a 3% chance the null is true" | The p-value is the probability of data this extreme given the null is true, not the probability the null is true |
| "p = 0.001 means the effect is large" | It means the evidence against the null is strong; magnitude is a separate question |
| "p = 0.06 proves there is no effect" | Failure to reject is not acceptance; the effect may be real but undetected |
| "p < 0.05 means the result will replicate" | Replication probability depends on power, not on the p-value |
Reporting practice for a Six Sigma project
A defensible Analyze-phase conclusion states all four of:
- The effect size in process units -- "mean cycle time reduced by 1.8 days".
- The confidence interval -- "95% CI 1.1 to 2.5 days".
- The p-value and the test used -- "two-sample t-test, p = 0.002".
- The practical threshold and the verdict -- "the charter set a 1.0 day threshold; the entire confidence interval exceeds it, so the effect is both real and material".
Point 4 is what distinguishes a Black Belt's analysis from a statistics exercise. The confidence interval lying entirely above the practical threshold is the strongest possible statement, because it resolves both questions at once.
Two filling lines differ in mean fill weight by 0.05 g against a tolerance of plus or minus 8 g. With 50,000 units per group the difference is significant at p < 0.001. What is the correct conclusion?
A pilot with 12 units per group shows a 14% reduction in defect rate with p = 0.11. The charter's practical threshold is a 5% reduction. What is the appropriate next step?
Which statement correctly describes a p-value of 0.03?