8.3 Planning and Evaluating Designed Experiments
Key Takeaways
- The planning sequence is objective, response, factors and levels, design selection, run plan, execution, analysis, and confirmation.
- Factor ranges must be wide enough to produce a detectable effect but inside safe and feasible operating limits.
- Screening designs identify the vital few factors; characterization designs quantify effects and interactions; optimization designs locate the best settings.
- A confirmation run at the predicted optimum is mandatory, because a model is a prediction until it is verified.
- The measurement system must be validated before the experiment, because measurement variation enters the error term and masks real effects.
The planning sequence
| Step | Question | Failure if skipped |
|---|---|---|
| 1. Objective | Screening, characterization, or optimization? | Wrong design chosen |
| 2. Response | What is measured, how, and is the gauge validated? | Measurement noise masks real effects |
| 3. Factors and levels | Which factors, at what settings, held how constant? | Missing factor becomes a lurking variable |
| 4. Design | Which design, at what resolution, with what replication? | Confounded results that cannot be resolved |
| 5. Run plan | Randomized order, blocking, who runs it, over what period | Systematic bias |
| 6. Execution | Actual settings recorded; deviations logged | Analysis of the design you intended, not the one you ran |
| 7. Analysis | Effects, ANOVA, residual diagnostics, reduced model | Overfitted or invalid model |
| 8. Confirmation | Runs at the predicted optimum | Prediction never verified |
Step 1: The objective determines the design
| Objective | Typical factor count | Design family | Runs |
|---|---|---|---|
| Screening -- find the vital few from many candidates | 5-15 | Resolution III or IV fractional factorial; Plackett-Burman | 8-32 |
| Characterization -- quantify main effects and interactions | 2-5 | Full factorial or Resolution V fraction, with centre points | 16-64 |
| Optimization -- find the settings that optimize the response | 2-4 | Response surface: central composite, Box-Behnken | 15-30 |
| Robustness -- find settings insensitive to noise | 2-6 controllable plus noise | Robust parameter design, inner/outer array | Varies |
The sequential logic matters: screen first, characterize the survivors, then optimize. Attempting to optimize eight factors in one experiment is expensive and usually finds that five of them do not matter.
Step 2: The response
The response must be:
- Continuous where possible. Attribute responses need far more runs for the same sensitivity.
- Measured by a validated system. Observed variance is process variance plus measurement variance; a gauge consuming 30% of the variation can hide a real effect entirely. Complete the gage R&R before the experiment, not after a disappointing result.
- Sensitive to the factors. A response measured with resolution coarser than the effect size cannot detect it.
- Related to the CTQ. Optimizing a convenient surrogate that the customer does not experience is a common waste.
Multiple responses are normal, and they frequently conflict -- higher throughput against lower defect rate. Plan for that with desirability functions or overlaid contour plots rather than optimizing one and hoping.
Step 3: Factors, levels, and ranges
Sources for the candidate factor list: the process map, the cause-and-effect diagram, the PFMEA, the Analyze-phase findings, and operator knowledge. Then classify each candidate:
| Class | Treatment in the experiment |
|---|---|
| Controllable and of interest | Include as an experimental factor |
| Controllable but not of interest | Hold constant and record the value |
| Uncontrollable but measurable | Record as a covariate; consider blocking |
| Uncontrollable and unmeasurable | Randomize against it |
Setting the levels
Range selection is the judgment that most affects whether the experiment succeeds.
- Too narrow and the effect is smaller than experimental error, so a real factor is declared inactive. This is by far the more common error, because engineers instinctively stay near the current operating point.
- Too wide and you risk scrap, safety issues, equipment damage, or a region where the response is no longer approximately linear between the two levels.
Practical guidance: set the levels near the edges of the safe operating window, not near the current setting. A useful check is whether the two levels would be expected to produce a visible difference to someone who knows the process; if not, widen them.
Two-level factors assume approximate linearity across the range. Centre points test that assumption: if the average of the centre points differs significantly from the average of the factorial points, curvature is present and a response surface design is needed.
Step 4-6: Design, run plan, execution
Once objective, response, and factors are set, the design follows almost mechanically from the run budget and the required resolution. The run plan then adds:
- Randomized run order, generated and printed before starting.
- Blocking for known nuisance sources -- day, lot, operator, machine.
- A data sheet with the actual settings achieved, not just the intended ones, plus a notes column.
- A defined procedure for what to do when a run cannot be made: never silently substitute another setting.
During execution, record what actually happened. Analyzing the design you intended when the operator could not reach 220 C and ran 210 C instead produces a wrong model. Deviations are recoverable if recorded and unrecoverable if not.
Step 7: Analysis
- Plot the raw data first -- main effects plots and interaction plots.
- Estimate the effects; use a normal probability plot of effects or a Pareto of effects to identify the active ones. Inactive effects fall on a straight line through zero.
- Build the ANOVA and check significance.
- Check the residuals: normal probability plot, residuals versus fits, residuals versus run order. The last one detects time-related drift that randomization converted into error.
- Reduce the model to the significant terms, respecting hierarchy -- if an interaction is retained, keep both parent main effects even if individually non-significant.
- Report $R^2$ and adjusted $R^2$. A large gap between them indicates overfitting.
Step 8: Confirmation
The confirmation run is not optional. Run several replicates at the predicted optimum and compare the observed response with the model's prediction interval.
| Outcome | Interpretation |
|---|---|
| Observed inside the prediction interval | Model validated; proceed to pilot and implementation |
| Observed outside the interval | Model is wrong: a missing factor, extrapolation beyond the design space, or an unstable process |
A model that predicts well inside the experimental region can still fail outside it. Never extrapolate: the model is only supported over the ranges actually run.
Common planning failures
| Failure | Consequence | Prevention |
|---|---|---|
| Measurement system not validated | Real effects hidden in noise | MSA before the experiment |
| Ranges too narrow | Important factors declared inactive | Set levels near the safe operating limits |
| Important factor omitted and not held constant | Becomes a lurking variable | Structured factor identification from the PFMEA and process map |
| Run order not randomized | Time-related drift confounded with factors | Generate and follow a randomized order |
| No replication or centre points | No valid error estimate | Add replicates or centre points |
| Process unstable during the experiment | Error term inflated; nothing is significant | Establish statistical control first |
| No confirmation run | Unverified prediction implemented | Always confirm |
A designed experiment on cure temperature finds no significant effect, but the two levels were 178 C and 182 C around a current setpoint of 180 C. What is the most likely explanation?
Why must a confirmation run be performed at the predicted optimum settings?
A factor is controllable but is not of interest in the experiment. How should it be handled?