11.3 Training Program Evaluation & Effectiveness Tracking
Key Takeaways
- Kirkpatrick's 4-Level Evaluation Model measures training impact at Level 1 (Reaction), Level 2 (Learning), Level 3 (Behavior/Transfer), and Level 4 (Results).
- Phillips Level 5 ROI extends Kirkpatrick by calculating net financial benefits divided by total training costs: ROI (%) = (Net Benefits / Total Costs) x 100.
- Level 3 (Behavior/Transfer) is the critical link in quality training, evaluating whether employees apply new skills on the shop floor 30, 60, or 90 days post-training.
- Total Training Costs must capture both direct expenses (instructors, materials, LMS) and indirect expenses (participant wages, lost production time, facility overhead).
- ISO 9001:2015 Clause 7.2 requires evaluating the effectiveness of training interventions, shifting focus from tracking training hours to validating performance capability.
11.3 Training Program Evaluation & Effectiveness Tracking
Quick Summary: Evaluating training effectiveness is essential to verify that workforce development investments produce measurable behavioral change and operational improvements. Using Kirkpatrick's Four-Level Evaluation Framework and Phillips' Level 5 ROI Formula, quality managers assess learner reaction, knowledge acquisition, workplace behavior transfer, business results, and financial return on investment.
Organizations spend substantial financial and human capital on workforce training. However, historically, quality managers have relied on superficial metrics—such as total training hours completed or immediate participant satisfaction scores ("smile sheets")—to justify training budgets. In modern quality management, training is treated as a strategic investment that must demonstrate verified competence, reduced operational risk, and positive financial return.
The Kirkpatrick Four-Level Evaluation Framework
Originally developed by Dr. Donald Kirkpatrick, this framework is the global standard for evaluating educational and training programs across four sequential levels:
Level 4: RESULTS ───────────> Business Impact (Scrap reduction, ROI, Yield)
Level 3: BEHAVIOR ──────────> Job Application (Transfer of learning on Gemba)
Level 2: LEARNING ──────────> Knowledge/Skill Gain (Pre/post-tests, simulations)
Level 1: REACTION ──────────> Learner Satisfaction (Surveys, utility ratings)
Level 1: Reaction
Level 1 measures how participants respond to the training experience. It evaluates learner satisfaction, engagement, instructor effectiveness, and perceived utility of the content.
- Instruments: Post-course surveys, numeric rating scales, open-ended feedback.
- Focus: Affective reaction (Did they like it?) and Utility reaction (Do they believe it will help their work?).
- Limitation: High satisfaction does not guarantee that learning occurred or that behavior will change.
Level 2: Learning
Level 2 measures the degree to which participants acquired the intended knowledge, skills, attitudes, confidence, and commitment during the training.
- Instruments: Written pre/post-tests, hands-on skill demonstrations, rubric-scored performance assessments.
- Focus: Quantifying the delta between entry knowledge and exit proficiency.
- Limitation: Demonstrating skill in a classroom does not guarantee the employee will apply it consistently on the job.
Level 3: Behavior (Learning Transfer)
Level 3 measures the degree to which participants apply what they learned during training when they return to their actual work environment (the Gemba).
- Instruments: 30/60/90-day post-training audits, direct supervisory observation checklists, 360-degree feedback, SPC chart audits.
- Key Challenge: Learning transfer requires a supportive work environment. Barriers to transfer include lack of supervisor reinforcement, time pressure, and peer resistance.
Level 4: Results
Level 4 measures the targeted business outcomes resulting from the training program.
- Instruments: Operational KPIs, Cost of Poor Quality (COPQ) tracking, scrap and rework logs, customer warranty return rates, safety incident counts.
- Focus: Demonstrating direct connection between training execution and macro business performance.
Kirkpatrick Level Comparison
| Level | Focus Question | Timing | Common Tools | Difficulty & Value |
|---|---|---|---|---|
| 1. Reaction | Did participants find the training engaging and relevant? | Immediate (End of class) | Smile sheets, Likert-scale surveys | Low effort / Low value |
| 2. Learning | Did participants acquire the targeted KSAs? | End of course | Pre/post-tests, skill demonstrations | Moderate effort / Moderate value |
| 3. Behavior | Are participants applying new skills on the job? | 30 to 90 days post-training | Gemba observations, process audits | High effort / High value |
| 4. Results | Did the training produce desired business metrics? | 3 to 12 months post-training | Scrap logs, COPQ reports, yield metrics | Highest effort / Highest value |
Phillips Level 5: Return on Investment (ROI)
Developed by Dr. Jack Phillips, Level 5 ROI extends Kirkpatrick's framework by converting Level 4 operational results into monetary value and comparing them to total program costs.
1. Accounting for Total Training Costs
To calculate an accurate ROI, organizations must capture all direct and indirect costs:
- Direct Costs: Instructor fees, courseware materials, facility rentals, e-learning software licenses, travel expenses.
- Indirect Costs: Participant wages during training hours, lost production opportunity costs, administrative setup time.
2. ROI Formulas
3. Isolating the Effects of Training
To ensure ROI validity, quality managers must isolate training impact from confounding operational variables (e.g., new raw material suppliers, new equipment, seasonal shifts). Common isolation methods include:
- Control Group Analysis: Comparing trained operator cohorts against an untrained control group.
- Trend Line Analysis: Projecting pre-training performance trends and measuring post-training divergence.
- Expert & Customer Estimation: Gathering conservative impact estimates from process engineers and supervisors.
ISO 9001:2015 Clause 7.2 Effectiveness Tracking
ISO 9001:2015 Clause 7.2 specifically requires organizations to "evaluate the effectiveness of the actions taken" to address competence gaps. Internal quality auditors will look for evidence that:
- Training needs were identified based on role requirements.
- Training was delivered as planned.
- Effectiveness was evaluated using objective criteria (Level 3 or 4), rather than merely checking attendance logs.
- Retraining or corrective actions were initiated if effectiveness criteria were not met.
CMQ/OE Exam Tip: The exam often asks how to evaluate whether a training program was successful. Be ready to identify that Level 3 (Behavior) and Level 4 (Results) provide true effectiveness evidence under ISO 9001. A Level 1 survey showing 95% satisfaction is insufficient evidence of competence under ISO audits!
Practical Example: Six Sigma Green Belt ROI Calculation
A manufacturing plant enrolled 10 manufacturing engineers in a Six Sigma Green Belt certification course to address a recurring machining defect.
Financial Data:
- Direct Training Costs: Tuition ($15,000) + Course Materials ($3,000) + Software Licenses ($2,000) = $20,000
- Indirect Training Costs: Engineer Wages during 80 hours of class time ($25,000) + Admin Overhead ($5,000) = $30,000
- Total Training Costs: $20,000 + $30,000 = $50,000
Operational Results (12 Months Post-Training):
- Control group analysis confirmed scrap reduction directly attributable to Green Belt project = $200,000 annual savings
Calculations:
- Net Financial Benefits: $200,000 - $50,000 = $150,000
- Benefit-Cost Ratio (BCR): $200,000 / $50,000 = 4.0 to 1
- Return on Investment (ROI): ($150,000 / $50,000) x 100 = 300%
Conclusion: For every $1.00 invested in Green Belt training, the organization recovered $4.00 in gross benefits, yielding a net ROI of 300%.
A Quality Auditor conducts a 60-day post-training Gemba audit to observe whether machine operators are correctly using standard work calibration procedures taught in class. Which level of Kirkpatrick's evaluation model is being measured?
An organization spends $40,000 in total training costs (direct plus indirect) for a poka-yoke error-proofing course. Over the following year, error-proofing improvements implemented by class graduates generate $160,000 in gross scrap reductions. What is the calculated Return on Investment (ROI)?
Why is relying solely on Level 1 'smile sheets' considered inadequate for proving compliance with ISO 9001:2015 Clause 7.2?
You've completed this section
Continue exploring other exams