8.3 Planning and Evaluating Designed Experiments
Key Takeaways
The planning sequence is objective, response, factors and levels, design selection, run plan, execution, analysis, and confirmation.
Factor ranges must be wide enough to produce a detectable effect but inside safe and feasible operating limits.
Screening designs identify the vital few factors; characterization designs quantify effects and interactions; optimization designs locate the best settings.
A confirmation run at the predicted optimum is mandatory, because a model is a prediction until it is verified.
The measurement system must be validated before the experiment, because measurement variation enters the error term and masks real effects.
The planning sequence
| Step | Question | Failure if skipped |
|---|---|---|
| 1. Objective | Screening, characterization, or optimization? | Wrong design chosen |
| 2. Response | What is measured, how, and is the gauge validated? | Measurement noise masks real effects |
| 3. Factors and levels | Which factors, at what settings, held how constant? | Missing factor becomes a lurking variable |
| 4. Design | Which design, at what resolution, with what replication? | Confounded results that cannot be resolved |
| 5. Run plan | Randomized order, blocking, who runs it, over what period | Systematic bias |
| 6. Execution | Actual settings recorded; deviations logged | Analysis of the design you intended, not the one you ran |
| 7. Analysis | Effects, ANOVA, residual diagnostics, reduced model | Overfitted or invalid model |
| 8. Confirmation | Runs at the predicted optimum | Prediction never verified |
Step 1: The objective determines the design
| Objective | Typical factor count | Design family | Runs |
|---|---|---|---|
| Screening -- find the vital few from many candidates | 5-15 | Resolution III or IV fractional factorial; Plackett-Burman | 8-32 |
| Characterization -- quantify main effects and interactions | 2-5 | Full factorial or Resolution V fraction, with centre points | 16-64 |
| Optimization -- find the settings that optimize the response | 2-4 | Response surface: central composite, Box-Behnken | 15-30 |
| Robustness -- find settings insensitive to noise | 2-6 controllable plus noise | Robust parameter design, inner/outer array | Varies |
The sequential logic matters: screen first, characterize the survivors, then optimize. Attempting to optimize eight factors in one experiment is expensive and usually finds that five of them do not matter.
Step 2: The response
The response must be:
- Continuous where possible. Attribute responses need far more runs for the same sensitivity.
- Measured by a validated system. Observed variance is process variance plus measurement variance; a gauge consuming 30% of the variation can hide a real effect entirely. Complete the gage R&R before the experiment, not after a disappointing result.
- Sensitive to the factors. A response measured with resolution coarser than the effect size cannot detect it.
- Related to the CTQ. Optimizing a convenient surrogate that the customer does not experience is a common waste.
Multiple responses are normal, and they frequently conflict -- higher throughput against lower defect rate. Plan for that with desirability functions or overlaid contour plots rather than optimizing one and hoping.
Step 3: Factors, levels, and ranges
Sources for the candidate factor list: the process map, the cause-and-effect diagram, the PFMEA, the Analyze-phase findings, and operator knowledge. Then classify each candidate:
| Class | Treatment in the experiment |
|---|---|
| Controllable and of interest | Include as an experimental factor |
| Controllable but not of interest | Hold constant and record the value |
| Uncontrollable but measurable | Record as a covariate; consider blocking |
| Uncontrollable and unmeasurable | Randomize against it |
Setting the levels
Range selection is the judgment that most affects whether the experiment succeeds.
- Too narrow and the effect is smaller than experimental error, so a real factor is declared inactive. This is by far the more common error, because engineers instinctively stay near the current operating point.
- Too wide and you risk scrap, safety issues, equipment damage, or a region where the response is no longer approximately linear between the two levels.
Practical guidance: set the levels near the edges of the safe operating window, not near the current setting. A useful check is whether the two levels would be expected to produce a visible difference to someone who knows the process; if not, widen them.
Two-level factors assume approximate linearity across the range. Centre points test that assumption: if the average of the centre points differs significantly from the average of the factorial points, curvature is present and a response surface design is needed.
Step 4-6: Design, run plan, execution
Once objective, response, and factors are set, the design follows almost mechanically from the run budget and the required resolution. The run plan then adds:
- Randomized run order, generated and printed before starting.
- Blocking for known nuisance sources -- day, lot, operator, machine.
- A data sheet with the actual settings achieved, not just the intended ones, plus a notes column.
- A defined procedure for what to do when a run cannot be made: never silently substitute another setting.
During execution, record what actually happened. Analyzing the design you intended when the operator could not reach 220 C and ran 210 C instead produces a wrong model. Deviations are recoverable if recorded and unrecoverable if not.
Step 7: Analysis
- Plot the raw data first -- main effects plots and interaction plots.
- Estimate the effects; use a normal probability plot of effects or a Pareto of effects to identify the active ones. Inactive effects fall on a straight line through zero.
- Build the ANOVA and check significance.
- Check the residuals: normal probability plot, residuals versus fits, residuals versus run order. The last one detects time-related drift that randomization converted into error.
- Reduce the model to the significant terms, respecting hierarchy -- if an interaction is retained, keep both parent main effects even if individually non-significant.
- Report and adjusted . A large gap between them indicates overfitting.
Step 8: Confirmation
The confirmation run is not optional. Run several replicates at the predicted optimum and compare the observed response with the model's prediction interval.
| Outcome | Interpretation |
|---|---|
| Observed inside the prediction interval | Model validated; proceed to pilot and implementation |
| Observed outside the interval | Model is wrong: a missing factor, extrapolation beyond the design space, or an unstable process |
A model that predicts well inside the experimental region can still fail outside it. Never extrapolate: the model is only supported over the ranges actually run.
Common planning failures
| Failure | Consequence | Prevention |
|---|---|---|
| Measurement system not validated | Real effects hidden in noise | MSA before the experiment |
| Ranges too narrow | Important factors declared inactive | Set levels near the safe operating limits |
| Important factor omitted and not held constant | Becomes a lurking variable | Structured factor identification from the PFMEA and process map |
| Run order not randomized | Time-related drift confounded with factors | Generate and follow a randomized order |
| No replication or centre points | No valid error estimate | Add replicates or centre points |
| Process unstable during the experiment | Error term inflated; nothing is significant | Establish statistical control first |
| No confirmation run | Unverified prediction implemented | Always confirm |
A designed experiment on cure temperature finds no significant effect, but the two levels were 178 C and 182 C around a current setpoint of 180 C. What is the most likely explanation?
Cure temperature genuinely has no effect on the response
The factor range was too narrow, so any real effect was smaller than experimental error
The design resolution was too high
Centre points should have been omitted
Why must a confirmation run be performed at the predicted optimum settings?
To satisfy the randomization requirement
To generate the degrees of freedom needed for the ANOVA
Because the model is a prediction until verified; if the observed response falls outside the prediction interval, the model is wrong
Because ASQ requires three confirmation runs for every experiment
A factor is controllable but is not of interest in the experiment. How should it be handled?
Include it as an experimental factor to be safe
Block on it
Randomize it across runs
Hold it constant at a defined value and record that value
Sections you finish are checked off in the contents.