7.3 Organizing Evidence by Risk and Significance

7.3 Organizing Evidence by Risk and Significance (CQA BoK II.B.6 — Evaluate)

Quick Answer: After collecting objective evidence, auditors evaluate and organize it by significance, severity, frequency, and risk, including impact on product, process, system, and cost of quality (COQ). That evaluation drives whether to investigate further, how to classify issues, and what must be highlighted at the closing meeting and in the report.

A common exam failure mode is treating every nonconformity as equal: a missing training signature on a low-risk form is stacked beside an undetected release of nonconforming product. II.B.6 tests professional judgment—risk-based organization of evidence.

From Facts to Significance

Working papers may contain dozens of notes. Organization turns the pile into decisions:

1. Group evidence by process, criterion, or risk theme.
2. Separate conforming evidence (supports positive conclusions) from adverse evidence.
3. Rate adverse items for severity and extent (frequency/systemic nature).
4. Estimate impact pathways (product → customer/patient; process → capability; system → QMS failure modes; COQ → waste and failure cost).
5. Decide: enough to write / need more sample / escalate / limitation.

Note: Programs use different labels (critical/major/minor; NC vs. observation). CQA cares that you evaluate factors, not that you memorize one company’s color codes.

Impact Dimensions Required by the BoK

Product impact

Does the issue affect product conformity, safety, identity, strength, purity, labeling, or delivery of the intended function?

• Out-of-tolerance measurement used for release decisions
• Mixed status of nonconforming and conforming product
• Unapproved material in production

Process impact

Does the issue degrade process capability, stability, or control of critical parameters?

• Skipped in-process checks at a CTQ step
• Uncontrolled process changes
• Ineffective setup verification

System impact

Does the issue signal QMS design or implementation failure that could recur across processes?

• CAPA not identifying true root cause (recurrence engine)
• Document control systematically issuing obsolete specs
• Management review ignoring quality objectives trends

Cost of quality (COQ) impact

Does the issue drive prevention gaps, appraisal waste, internal failure (scrap/rework), or external failure (returns, warranty, complaints, recalls)?

Even when product escaped harm this time, high internal failure cost can be significant to the business case for action—and exam stems sometimes use COQ language deliberately.

Severity vs. Frequency: Two Different Levers

Trap: Counting findings instead of weighing them. Ten tiny administrative NCs do not automatically outweigh one product-safety NC—nor should ten tiny issues be ignored if they prove a broken document-control system.

Organizing for Further Investigation

Evaluate when the current sample is not enough:

Stopping rules should be risk-based. Do not over-sample a trivial issue while under-sampling a life-cycle critical process just because the trivial records were easy to pull.

Practical Organization Techniques

1. Risk wall / issue log (manual or electronic)

Columns that mirror Evaluate thinking:

• Criterion reference
• Evidence summary (objective facts)
• Product / process / system / COQ impact flags
• Severity & extent ratings
• Status: open / needs more data / draft finding / drop
• Owner on the audit team

2. Clustering

Cluster related symptoms into one systemic finding when root mechanism is shared (e.g., multiple obsolete specs all point to document-control failure). Avoid “death by a thousand identical NCs” that hide the system issue—or the opposite error of merging unrelated problems into a vague mega-finding.

3. Positive evidence organization

Conformity evidence also needs organization. A process with strong objective evidence of control supports a positive conclusion and balances the report. Case studies sometimes test whether you notice sufficient positive evidence to avoid inventing problems.

4. Pre-closing triage

Before exit/closing:

• Rank issues by risk/significance
• Confirm evidence packages are complete for top items
• Align team on classification language per program
• Identify factual points needing auditee clarification (not negotiation of criteria)

Scenario Set (Evaluate Practice)

Scenario A — Measurement escape
Final inspection gauge used for accept/reject found out of calibration; last calibration failed high. Last 3 weeks of product accepted with that gauge.
Evaluation: High product and process impact; system impact on metrology control; COQ external failure risk. Expand to affected lots, containment status, and prior similar events. Top-tier finding candidate.

Scenario B — Training matrix cosmetic gap
One employee’s matrix missing a refresher due date for a non-product administrative course; competence on job tasks verified by observation and assessments.
Evaluation: Low product/process impact; limited system signal if truly isolated. May be minor NC or observation per program—do not equate to Scenario A.

Scenario C — Recurring complaint theme
Four customer complaints in six months on labeling mix; CAPAs closed as “retrain operator” each time; no label-control process change.
Evaluation: Frequency + failed system learning → high significance even if each event looked “human error.” Organize as effectiveness/system issue, not four unrelated operator problems.

Link to Cost of Quality Language on the Exam

When stems mention scrap rates, rework hours, warranty cost, or appraisal overtime, map them:

• Prevention failure → weak design of controls, ineffective training system, poor change control
• Appraisal waste → redundant inspection because process is unstable
• Internal failure → scrap/rework/sort
• External failure → complaints, returns, recalls, lost customers

Significant evidence often shows up first as COQ pain; good auditors follow that pain into process and system causes.

Common CQA Traps for II.B.6

• Ranking findings by how angry the auditor feels, not risk.
• Ignoring systemic patterns across “small” items.
• Declaring effectiveness of a process based on thin samples at high risk.
• Failing to pursue further investigation when impact could be product safety/regulatory.
• Over-writing major classifications for trivia, which destroys credibility at closing.
• Forgetting positive evidence organization when the opinion is favorable.

Key Exam Takeaway

II.B.6 is the bridge between raw objective evidence and professional conclusions: weigh severity, frequency, and risk; score impact on product, process, system, and COQ; investigate further when high-risk questions remain unanswered. That triage is what the exit and closing meetings must reflect.