9.10 Ongoing Evaluation of the Improved Process
Key Takeaways
- Ongoing evaluation combines control charts, control plan audits, leading and lagging indicators, and a scheduled review cadence.
- Leading indicators are process inputs that predict the outcome; lagging indicators confirm it after the fact.
- Control plan audits verify that the control is executed as written, which is different from verifying that the result is still good.
- A defined review cadence with a named owner is what distinguishes sustained improvement from a gain that quietly decays.
- Ongoing evaluation should feed the next improvement cycle, because a stable improved process reveals the next constraint.
What ongoing evaluation covers
Four questions, each with its own instrument and frequency.
| Question | Instrument | Frequency | Owner |
|---|---|---|---|
| Is the process still in statistical control? | Control chart with an out-of-control action plan | Per subgroup | Operator |
| Are the controls being executed as written? | Layered process audit, control plan audit | Daily to weekly | Supervisor |
| Is the outcome still at the improved level? | Lagging KPI against the post-project baseline | Weekly to monthly | Process owner |
| Is the financial benefit still being realized? | Finance validation | 6 and 12 months | Finance and sponsor |
The second row is the one most often omitted, and it is the leading indicator of sustainment failure: results usually decay after compliance decays, so an outcome-only monitoring plan detects the problem a month or two late.
Leading and lagging indicators in sustainment
| Leading | Lagging | |
|---|---|---|
| What it measures | Process inputs and behaviours ($X$s) | Process outcomes ($Y$s) |
| Timing | Available now; predictive | Available after the fact |
| Actionability | High -- corrective action still possible | Low -- the result is already produced |
| Examples | Control chart signals per week, audit compliance rate, calibration on-time rate, training coverage, setup verification completion | Defect rate, DPMO, first pass yield, customer complaints, warranty claims |
A sustainment monitoring plan needs both. Leading indicators tell you the improvement is about to fail; lagging indicators confirm whether it has. Monitoring only lagging indicators means learning about the failure from the customer.
Two or three leading indicators is the right number. A monitoring plan with fifteen measures is not monitored.
Control plan audits
A control plan audit answers "is the control being executed as written?" -- which is a different question from "is the output still good?" A process can produce acceptable output for weeks after its controls have lapsed, right up until the condition that the control was protecting against occurs.
Layered process audits (LPA) are the standard mechanism: short, frequent checks of the same small set of high-risk process controls, performed at several levels of the organization.
| Layer | Who | Frequency | Typical scope |
|---|---|---|---|
| 1 | Team leader or supervisor | Daily, each shift | 5-10 questions on the critical controls |
| 2 | Department manager | Weekly | Same questions, a sample of stations |
| 3 | Plant manager or senior staff | Monthly | Same questions, spot check |
The multi-layer structure is what makes it work: senior participation signals that the controls matter, and it removes the possibility of a supervisor quietly stopping the audits. Findings must be corrected immediately or logged with an owner and a date.
The review cadence
| Interval | Review | Content | Attendees |
|---|---|---|---|
| Daily | Shift review | Control chart signals, andon events, audit findings | Operators, supervisor |
| Weekly | Process review | KPI trend, open corrective actions, audit compliance | Process owner, quality |
| Monthly | Sustainment review | Performance versus the post-project baseline; leading indicator trends | Process owner, sponsor |
| 6 and 12 months | Benefit validation | Finance confirmation that the benefit persists | Finance, sponsor, Black Belt |
The 6 and 12-month checks are the ones that distinguish a real sustainment system from a paper one, and they should be scheduled at project closure with a calendar invitation and a named owner, not left to be remembered.
Detecting decay
| Symptom | Likely cause | Response |
|---|---|---|
| Control chart drifts back toward the old mean | The control is not being executed | Audit compliance; re-train; check whether the control is impractical |
| Audit compliance falls while results hold | Early sustainment failure | Intervene now, before results follow |
| A new special-cause pattern appears | New source of variation: material, personnel, equipment | Investigate as a new problem, not as project failure |
| Benefit erodes without a process change | Volume, mix, or cost-basis change | Recompute the benefit basis with finance |
| The control plan has not been updated after a process change | Documentation drift | Change control; update FMEA and control plan together |
The most common sustainment failure is not resistance but impracticality: a control that takes too long, obstructs the work, or conflicts with an output target will be abandoned quietly. When compliance falls, ask whether the control is workable before concluding that the people are the problem.
Feeding the next cycle
The Body of Knowledge asks specifically that ongoing evaluation identify additional opportunities for improvement, and this is not an afterthought. A stabilized process makes the next opportunity visible in a way that a chaotic one never could:
- The new constraint. Removing one bottleneck exposes another; the theory of constraints says to go back to step 1.
- The residual variation. A control chart on a stable improved process shows the common-cause variation that remains, which is the target for the next capability project.
- Newly visible special causes. Once the dominant noise is gone, smaller assignable causes become detectable.
- Replication targets. The solved problem probably exists elsewhere.
- The entitlement gap. The best week of the improved process is the new entitlement, and the gap between it and the new average is the next project.
Closing a project should therefore produce two artifacts: a control plan with a named owner, and a short list of the next opportunities the project revealed, handed to the project selection process.
A sustainment monitoring plan tracks only the monthly defect rate. What is the principal weakness?
What question does a layered process audit answer that a control chart does not?
Six months after a successful project, control chart data drifts back toward the pre-project mean and audit compliance has fallen to 40%. What should be investigated first?