8.2 DOE Design Principles: Randomization, Blocking, Replication, and Resolution

The three foundational principles

R. A. Fisher's three principles underpin every valid experiment.

The division of labour is worth memorizing: block what you know, randomize against what you do not.

Randomization

If temperature and pressure runs are performed in a systematic order, and ambient humidity happens to rise through the day, the humidity effect is inseparable from whatever factor was varied last. Randomize the run order and the humidity effect is distributed unpredictably across all factors, converting a systematic bias into ordinary experimental error.

Randomization also protects the validity of the statistical tests themselves. The $F$ tests in an ANOVA assume independent errors, and independence is what randomization delivers.

Where full randomization is expensive -- for example when a factor requires a lengthy oven change -- a split-plot design randomizes the hard-to-change factor at the whole-plot level and the easy-to-change factors within it. This is a legitimate design, but it has two error terms and must be analyzed accordingly. Simply running the hard-to-change factor in convenient order and analyzing as though it were fully randomized is not.

Replication versus repetition

This distinction is a reliable exam question.

Treating repetition as replication understates experimental error, which inflates the $F$ statistic and makes effects appear significant when they are not. If a run's setup is done once and five parts are measured, that is repetition; averaging them gives a better estimate of that run's response, but it says nothing about run-to-run variation.

Blocking

A block is a group of experimental units expected to be more homogeneous within than between: one raw material lot, one day, one operator, one machine. Assign a complete set of treatments within each block, and the block-to-block variation is estimated and removed from the error term.

The result is a more sensitive experiment. The cost is one or more degrees of freedom, and the assumption that the blocking factor does not interact with the treatment factors.

Balance, order, and efficiency

Balance. A design is balanced when each factor level appears the same number of times and each combination appears equally often. Balanced designs make the effect estimates orthogonal -- estimated independently of one another -- so removing a term from the model does not change the other estimates. Unbalanced designs can still be analyzed but the estimates are correlated and interpretation is harder.

Order. Standard order (Yates order) is a canonical listing of runs used for constructing and analyzing the design. Run order must be randomized. Never confuse the two: the design matrix is written in standard order, and the experiment is executed in random order.

Efficiency. The information obtained per experimental run. Factorial designs are efficient because every run contributes to the estimate of every effect; one-factor-at-a-time experimentation is inefficient because each run informs only one comparison, and it cannot detect interactions at all.

Power and sample size in DOE

Power is the probability of detecting a real effect of a given size. In a two-level factorial, the number of runs required to detect an effect $\delta$ with power $1-\beta$ depends on $\delta/\sigma$ and on the number of replicates.

$\text{Effect estimate} = \bar{y}_{+} - \bar{y}_{-}, \qquad SE_{\text{effect}} = \frac{2\sigma}{\sqrt{N}}$

where $N$ is the total number of runs. Because every run contributes to every effect estimate, adding one replicate of a $2^4$ design (16 more runs) reduces the standard error of every effect by a factor of $\sqrt{2}$.

Two practical rules: an unreplicated two-level factorial has no degrees of freedom for error unless higher-order interactions are pooled into error or a normal probability plot of effects is used; and adding centre points provides both an error estimate and a test for curvature without disturbing the factorial structure.

Interaction and confounding

An interaction exists when the effect of one factor depends on the level of another. Interaction plots with non-parallel lines indicate interaction; crossing lines indicate a strong one. Interactions are the main reason factorial designs beat one-factor-at-a-time work, because OFAT cannot detect them.

Confounding (aliasing) occurs when two effects cannot be separately estimated because they are calculated from the same contrast. It is the price of fractional designs: reduce the number of runs and some effects merge.

Resolution

Design resolution states which effects are confounded, and it is written as a Roman numeral.

A useful mnemonic: the resolution number is the length of the shortest word in the defining relation, and an effect of order $p$ is confounded with effects of order $R - p$. In Resolution III, a main effect ($p=1$) is confounded with $3-1 = 2$-factor interactions; in Resolution V, a main effect is clear of two-factor interactions and two-factor interactions ($p=2$) are confounded with $5-2 = 3$-factor interactions.

Because three-factor and higher interactions are usually negligible in physical processes -- the sparsity of effects principle -- Resolution V designs are treated as effectively unconfounded.

A design checklist

1. Have the response, factors, levels, and ranges been defined, with the response measured by a validated measurement system?
2. Are the factor ranges wide enough to produce a detectable effect, but inside safe operating limits?
3. Is the design balanced, and what is its resolution?
4. Which effects are confounded, and is that acceptable given what is known about the process?
5. Is the run order randomized, and if not, is a split-plot analysis planned?
6. Are there replicates (not merely repetitions) to estimate error, or centre points?
7. Are known nuisance factors blocked?
8. Has the power to detect the practically important effect been computed before running?