Two filling lines differ in mean fill weight by 0.05 g against a tolerance of plus or minus 8 g. With 50,000 units per group the difference is significant at p < 0.001. What is the correct conclusion?

The difference is statistically significant but not practically significant, and no action is warranted. A p-value is a function of effect size, variability, and sample size, so a very large sample makes any non-zero difference statistically detectable. A 0.05 g difference against a 16 g tolerance window is 0.3% of the specification width -- far below any sensible practical threshold. The result belongs in the statistically-significant, practically-insignificant cell, where the correct action is none.

A pilot with 12 units per group shows a 14% reduction in defect rate with p = 0.11. The charter's practical threshold is a 5% reduction. What is the appropriate next step?

Treat the result as inconclusive, compute the power the study had, and collect more data. This is the practically significant but not statistically significant cell. A small pilot is typically underpowered, and failure to reject is not evidence of no effect. The correct response is to evaluate the power the study actually had and, if it was low, extend the pilot to obtain a decisive result. Implementing on an unconfirmed result and adjusting alpha after seeing the data are both indefensible.

Which statement correctly describes a p-value of 0.03?

There is a 3% probability of observing data at least this extreme if the null hypothesis were true. A p-value is the probability of obtaining data at least as extreme as observed, conditional on the null hypothesis being true. It is not the probability that the null hypothesis is true, it is not a measure of effect size or variance explained, and it does not give the probability of replication, which depends on statistical power.

Statistical Versus Practical Significance | Free Guide 2026

Q: Two filling lines differ in mean fill weight by 0.05 g against a tolerance of plus or minus 8 g. With 50,000 units per group the difference is significant at p < 0.001. What is the correct conclusion?

The difference is statistically significant but not practically significant, and no action is warranted. A p-value is a function of effect size, variability, and sample size, so a very large sample makes any non-zero difference statistically detectable. A 0.05 g difference against a 16 g tolerance window is 0.3% of the specification width -- far below any sensible practical threshold. The result belongs in the statistically-significant, practically-insignificant cell, where the correct action is none.

Q: A pilot with 12 units per group shows a 14% reduction in defect rate with p = 0.11. The charter's practical threshold is a 5% reduction. What is the appropriate next step?

Treat the result as inconclusive, compute the power the study had, and collect more data. This is the practically significant but not statistically significant cell. A small pilot is typically underpowered, and failure to reject is not evidence of no effect. The correct response is to evaluate the power the study actually had and, if it was low, extend the pilot to obtain a decisive result. Implementing on an unconfirmed result and adjusting alpha after seeing the data are both indefensible.

Q: Which statement correctly describes a p-value of 0.03?

There is a 3% probability of observing data at least this extreme if the null hypothesis were true. A p-value is the probability of obtaining data at least as extreme as observed, conditional on the null hypothesis being true. It is not the probability that the null hypothesis is true, it is not a measure of effect size or variance explained, and it does not give the probability of replication, which depends on statistical power.

Two different questions

	Statistical significance	Practical significance
Question	Is the observed effect likely to be real, or could it be sampling variation?	Is the effect large enough to matter?
Answered by	p-value against $\alpha$	Effect size compared with a threshold set by the business
Depends on	Effect size and sample size and variability	Effect size and the economics of the process
Who decides	The statistical procedure	The process owner, the customer, or finance

The critical asymmetry: statistical significance is computed, practical significance is judged, and the judgment must be made by someone who knows what the process is worth.

Why sample size drives the p-value

For a two-sample comparison of means the test statistic is roughly

t = \frac{\bar{x}_1 - \bar{x}_2}{s_p\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}}

The numerator is the effect. The denominator shrinks as $n$ grows. Hold the effect constant and increase $n$ , and $t$ grows without bound -- so any non-zero difference eventually becomes statistically significant.

Worked illustration. Two filling lines differ in mean fill weight by 0.05 g, with $s = 1.2$ g on a product with a tolerance of $\pm 8$ g.

$n$ per group	Approximate $t$	p-value	Statistically significant at 0.05?
30	0.16	0.87	No
500	0.66	0.51	No
5,000	2.08	0.037	Yes
50,000	6.59	< 0.001	Yes

At 50,000 units per group the difference is overwhelmingly significant and completely irrelevant: 0.05 g against an 8 g tolerance is 0.6% of the specification width. Nobody should spend a dollar closing it.

Measuring effect size

Report magnitude, not just the p-value.

Difference in means with a confidence interval. The most direct and the most useful. "Mean cycle time fell by 1.8 days, 95% CI 1.1 to 2.5 days" tells the reader the size, the direction, and the precision in the units of the problem.

Cohen's d, the standardized difference:

d = \frac{\bar{x}_1 - \bar{x}_2}{s_{pooled}}

$d$	Conventional label
0.2	Small
0.5	Medium
0.8	Large

Proportion of variance explained. $r^2$ in regression, or eta-squared in ANOVA: the share of output variation attributable to the factor.

Practical thresholds derived from the process. The most defensible option where they exist:

A shift smaller than the measurement system's uncertainty cannot be acted on reliably.
A shift smaller than a set fraction of the tolerance is not worth chasing.
A saving below the organization's project threshold does not justify the implementation cost.

Set the practical threshold before the analysis, in the charter. Setting it afterwards invites choosing the threshold that makes the result look good.

The four combinations

	Practically significant	Not practically significant
Statistically significant	Act. Real and worth doing.	Do not act. Real but trivial; usually a symptom of a very large sample.
Not statistically significant	Investigate further. Potentially valuable but not yet demonstrated -- check power and collect more data.	Stop. Neither real nor important.

The bottom-left cell is the one teams handle worst. A promising effect that fails to reach significance in an underpowered study is frequently reported as "no difference", and the opportunity is discarded. The correct response is to compute the power the study actually had, and if it was low, treat the result as inconclusive rather than negative.

What a p-value is not

Misinterpretation	Correction
"p = 0.03 means there is a 3% chance the null is true"	The p-value is the probability of data this extreme given the null is true, not the probability the null is true
"p = 0.001 means the effect is large"	It means the evidence against the null is strong; magnitude is a separate question
"p = 0.06 proves there is no effect"	Failure to reject is not acceptance; the effect may be real but undetected
"p < 0.05 means the result will replicate"	Replication probability depends on power, not on the p-value

Reporting practice for a Six Sigma project

A defensible Analyze-phase conclusion states all four of:

The effect size in process units -- "mean cycle time reduced by 1.8 days".
The confidence interval -- "95% CI 1.1 to 2.5 days".
The p-value and the test used -- "two-sample t-test, p = 0.002".
The practical threshold and the verdict -- "the charter set a 1.0 day threshold; the entire confidence interval exceeds it, so the effect is both real and material".

Point 4 is what distinguishes a Black Belt's analysis from a statistics exercise. The confidence interval lying entirely above the practical threshold is the strongest possible statement, because it resolves both questions at once.

CSSBB

7.5 Statistical Versus Practical Significance

Key Takeaways