9.3 Verifying CAPA Effectiveness
Key Takeaways
- CQA BoK II.D.3 (Evaluate) requires verifying that CAPA was implemented as planned and is effective—using document/procedure checks, process observation, records/data, and follow-up audits as appropriate.
- Implementation evidence (revised procedure, training roster) is necessary but not sufficient; effectiveness evidence shows the failure mode is controlled over a defined period or sample.
- Verification methods should match risk and action type: system queries, gemba observation, sampling of transactions, complaint/NC trend review, and targeted follow-up audits.
- Look for unintended consequences: fixes that create new nonconformities, workarounds, or shadow systems.
- Exam trap: closing CAPA because the due date passed and a procedure PDF exists, without performance or observation evidence of effectiveness.
9.3 Verifying CAPA Effectiveness (CQA BoK II.D.3 — Evaluate)
/practice/cqaPractice questions with detailed explanations
BoK II.D.3 is where audit follow-up becomes real. A beautiful plan that never works on the floor is a failed CAPA. Your job is to evaluate evidence that actions stuck and outcomes improved—not to accept hope as a method.
Implementation vs Effectiveness
| Concept | Question answered | Typical evidence |
|---|---|---|
| Implementation | Were planned actions completed? | Revised controlled docs, training records, software change tickets, new gages in service, updated contracts |
| Effectiveness | Did actions achieve the intended result (no recurrence / controlled risk)? | Zero/reduced recurrence over period, process capability, audit resampling, complaint codes, observation of correct behavior under real conditions |
Both matter. Implementation without effectiveness is hollow. Claimed effectiveness without implementation proof is luck or misattribution.
Scenario
CAPA for missing calibration control:
- Implementation only: New procedure issued; email to technicians.
- Effectiveness missing: Tools still overdue in the field; product acceptance continues with expired status.
- True effectiveness path: System blocks use of overdue tools; sample of tools in use all current; no acceptance records with expired IDs for 90 days; observation shows technicians check status before use.
Verification Methods (Match Method to Action)
1. Review procedures and related documentation
Confirm controlled documents reflect the change: correct revision, approval, training linkage, obsolete version removal, and consistency across related instructions/forms.
Evaluate traps:
- New procedure exists but old revision still at workstations.
- Document changed in a way that does not address the root cause.
- Conflicting SOPs still allow the old path.
2. Observe processes (gemba)
Watch the real process: operators, system screens, material flow, handoffs. Observation tests whether the “new way” is the actual way under production pressure.
| Observation focus | Why |
|---|---|
| Critical step that previously failed | Direct test of the fix |
| Workarounds / shadow spreadsheets | Bypass of official control |
| Interfaces (shift change, IT, suppliers) | Failures hide at boundaries |
| Peak workload moments | Controls often collapse when busy |
3. Examine records and data
Sample transactions after the effective date: batch records, CAPA logs, complaints, NC rates, first-pass yield, on-time calibration %, audit trails, environmental logs.
Good practice: Define the verification window and sample basis in advance (from the plan). Example: “All finished-good releases for 60 days” or “random 25 pick transactions per warehouse zone.”
4. Follow-up audits
A follow-up / CAPA verification audit is a purpose-driven audit (see also BoK I.A.3 purposes). Scope is typically centered on the CAPA and related process—not an automatic full system re-audit—unless risk or scheme rules expand it.
Use follow-up audits when:
- Risk is high or finding was major/critical.
- Actions are complex or multi-site.
- Desktop evidence is insufficient to prove behavior change.
- Prior CAPAs on the same theme failed.
5. Interviews and competence checks
Ask personnel to explain the new control and demonstrate it. Competence is more than a signed roster: can they perform the task and know what to do when the control fails?
Building a Verification Plan
| Step | Decision |
|---|---|
| 1. Re-read finding + accepted CAPA | Confirm what “done” and “effective” mean |
| 2. Identify critical control points | Where failure would reappear |
| 3. Select methods | Doc + observe + data + follow-up audit mix |
| 4. Set sample/period | Risk-based; longer windows for intermittent failures |
| 5. Check side effects | New NCs, cycle-time collapse, safety tradeoffs |
| 6. Record objective evidence | Who/what/when/result against criteria |
| 7. Conclude | Effective / partially effective / not effective |
Risk-Based Depth
| Finding risk | Typical verification depth |
|---|---|
| Low (isolated admin NC) | Document revision + limited record sample |
| Moderate | Records + observation + short trend review |
| High (safety, regulatory, customer escape) | Multi-method verification, longer trend, possible dedicated follow-up audit, management visibility |
| Repeat theme | Broader systemic audit; challenge CAPA system effectiveness |
Timing of Verification
Verify after actions are implemented and enough operating history exists to show sustained control. Verifying the day a procedure is published often proves only implementation intent, not effectiveness.
Exceptions: you may do early implementation checks for interim containment while waiting for long-term effectiveness windows.
| Too early | About right | Too late |
|---|---|---|
| Same day as doc release, no transactions | After defined production period with sample | Years later after multiple escapes |
Evidence Quality Standards
Apply the same objective-evidence mindset used in fieldwork (II.B.4–5):
- Traceable — finding ID, CAPA ID, dates, records sampled
- Sufficient — enough extent for confidence relative to risk
- Relevant — measures the failure mode that mattered
- Independent enough — verifier not solely the person who “fixed” their own mess without program rules allowing it
Metrics that often support effectiveness
| Failure mode | Effectiveness indicators |
|---|---|
| Wrong labels | Label scan fail rate; complaint code trend; audit sample of packed units |
| Overdue calibration | % tools current; blocked-use events; field sample of tools in use |
| Incomplete release records | % complete at release; system hard-stop logs; sample of released lots |
| Supplier defects | Incoming reject rate; escape rate; supplier process audit results |
| Training gaps | Competency assessment results; error rates post-training; observed practice |
Beware vanity metrics (“number of CAPAs closed”) that do not measure the failure mode.
Partial Effectiveness and Conditional Outcomes
Verification is not only binary.
| Outcome | Meaning | Typical next step |
|---|---|---|
| Effective | Implemented and performance evidence supports control | Close CAPA per procedure; retain evidence |
| Implemented but not yet effective | Actions done; window insufficient or early signals mixed | Extend monitoring; do not claim full effectiveness |
| Partially effective | Some causes controlled; residual gap remains | Additional actions; partial re-plan |
| Not effective | Recurrence or control fails | II.D.4 escalation path: reissue, escalate, re-audit |
| Cannot verify | Access denied, records missing, process idle | Document limitation; escalate; schedule when operable |
Unintended Consequences
Always ask: Did the fix break something else?
Examples:
- Hard system gate stops bad shipments but creates uncontrolled manual override culture.
- Extra inspection catches defects but hides process capability problems and burns capacity.
- New form fields ignored because cycle time unrealistic—shadow process returns.
Effective CAPA verification includes a quick scan for new nonconformities and override rates.
Roles: Who Verifies?
Programs differ: original auditor, independent auditor, quality assurance, or customer SQE. Evaluate competence and independence relative to risk. For high-risk CAPAs, independence from the action owner is best practice. Lead auditors should know the program rule and apply it consistently.
Mini Case — Multi-Method Verification
Finding: Incorrect firmware shipped.
Actions: ERP ship-hold without revision match; WI update; training; customer notification process for past shipments.
Verification package:
- Documents: WI Rev F controlled; obsolete Rev E removed from cells.
- System: Test attempt to ship mismatched revision—hard stop confirmed; override requires QA electronic approval with reason code.
- Records: 100% of Model X shipments for 45 days show match; override log empty.
- Observation: Pack-out associates demonstrate scan sequence on live order.
- Trends: Complaint code “wrong firmware” = 0 for two months post-fix.
- Side effects: Override governance reviewed—no informal bypass.
Conclusion: Implemented and effective for the defined window; continue monitoring in management review.
Common Exam Traps
| Trap | Better evaluation |
|---|---|
| Closed because calendar date passed | Date ≠ effectiveness |
| Training roster = done | Need competence + process results |
| One good day of data | Intermittent failures need longer windows |
| Auditee verbal assurance only | Seek objective evidence |
| Full system audit every time | Scope to purpose unless risk requires expansion |
| Ignoring recurrence of same NC elsewhere | Extent and systemic view |
Link to Audit Purpose and Reporting
Verification results should be recorded against the CAPA/finding IDs and available for management review and future audit planning (risk-based scheduling). Ineffective CAPA becomes input to II.D.4 escalation and may generate new findings against the CAPA process itself.
Exam Anchors
- Split implementation from effectiveness.
- Use procedures, observation, data, follow-up audits as a toolkit.
- Depth follows risk and action type.
- Watch side effects and overrides.
- “Procedure revised” alone is the classic wrong answer for effectiveness.
Three months after CAPA for recurring labeling errors, the procedure is revised and training is complete, but a sample of 40 recent packed units shows 3 wrong labels and complaint codes are unchanged. What is the best verification conclusion?
Which verification approach best matches a high-risk major CAPA that changed both software interlocks and shop-floor work sequence?
Why might an auditor observe the process at peak production rather than only during a quiet tour?
A CAPA introduced a hard ERP ship-hold. Verification finds zero mismatches shipped, but 120 QA overrides per week with no documented rationale. What should the auditor emphasize?