6.7 Attribute Agreement Analysis and Audit MSA

While continuous MSA evaluates physical gauges using numeric measurements, many quality decisions in production and administrative processes rely on discrete attribute inspection (e.g., visual surface defect classification, go/no-go plug gauges, invoice audit pass/fail decisions, call center compliance scoring). Subjective human judgment introduces significant inspection error. Attribute Measurement System Analysis (also termed Attribute Agreement Analysis) provides a structured statistical methodology to evaluate inspector consistency, repeatability, reproducibility, and accuracy against a verified master standard.

---

Experimental Design of an Attribute Agreement Study

To yield valid statistical conclusions, an Attribute Agreement study requires rigorous experimental design:

1. Sample Selection: Select 30 to 50 physical parts or transaction records. Crucially, the samples must cover the entire operational quality spectrum: approximately $50%$ conforming (clear pass), $25%$ non-conforming (clear fail), and $25%$ marginal / boundary samples (near specification limits).
2. Appraisers: Include 2 or 3 inspectors who routinely perform the inspection in daily operations.
3. Trials: Each inspector evaluates all samples across 2 or 3 independent trials in fully randomized order, blinding inspectors to sample identity.
4. Master Standard: A verified expert or precise destructive laboratory test establishes the true reference standard classification (Pass or Fail) for every sample.

---

Core Agreement Metrics & Benchmarks

Attribute MSA evaluates agreement across four distinct statistical dimensions:

---

Statistical Indicators of Agreement: Cohen's & Fleiss' Kappa

Simple percentage agreement can be deceptive because appraisers can agree purely by random chance (e.g., guessing on a binary pass/fail test yields $50%$ agreement by chance). Six Sigma requires statistical metrics that correct for chance agreement:

Cohen's Kappa ($k$) & Fleiss' Kappa Formulation

Kappa ($\kappa$) measures the degree of agreement between categorical ratings beyond what would be expected by random chance alone:

$\kappa = \frac{P_o - P_e}{1 - P_e}$

Where:

• $P_o$ = Observed proportion of concordant agreements across samples.
• $P_e$ = Expected proportion of agreements occurring purely by random chance.

Calculation of $P_e$ for a 2x2 Attribute Table

Given two appraisers inspecting $N$ items into Category 1 (Pass) and Category 2 (Fail):

$P_e = P_{\text{both pass}} + P_{\text{both fail}} = (p_{A1} \times p_{B1}) + (p_{A2} \times p_{B2})$

Interpretation of Kappa ($\kappa$)

• $\kappa = 1.00$: Perfect agreement.
• $\kappa > 0.75$: Good to Excellent Agreement (Measurement system is capable).
• $0.40 \le \kappa \le 0.75$: Fair to Moderate Agreement (Marginal; inspection criteria need standardization).
• $\kappa < 0.40$: Poor Agreement (Unacceptable; inspectors are guessing).

Kendall's Coefficient of Concordance ($W$)

When attribute inspection involves ordinal rankings (such as ranking defect severity on a scale from 1 = Minor Cosmetic to 5 = Critical Safety Failure), Kendall's $W$ is used. Kendall's $W$ ranges from $0.00$ (no agreement) to $1.00$ (complete concordance among appraisers).

---

Inspector Misclassification Risk Analysis

When inspectors disagree with the Master Standard, classification errors are divided into two operational risks:

$\text{Miss Rate (Type I Error / Consumer's Risk)} = \frac{\text{Number of Bad Parts Classified as Good}}{\text{Total Actual Bad Parts}} \times 100\%$

$\text{False Alarm Rate (Type II Error / Producer's Risk)} = \frac{\text{Number of Good Parts Classified as Bad}}{\text{Total Actual Good Parts}} \times 100\%$

• Miss Rate Risk: Non-conforming parts are shipped to customers, threatening product safety and customer satisfaction (Target $\le 2%$).
• False Alarm Rate Risk: Conforming parts are scrapped or reworked, creating unnecessary scrap costs and capacity losses (Target $\le 5%$).

Corrective Action Strategies for Attribute MSA Failures

• Physical Limit Samples (Boundary Boards): Mount physical reference specimens representing the maximum allowable defect size at every inspection station.
• Lighting & Ergonomics: Standardize ambient lighting intensity (lux level) and viewing angle in inspection booths.
• Inspection SOPs: Eliminate vague visual criteria (e.g., "paint must look smooth") with quantitative visual definitions.

---

Worked Attribute Agreement Analysis Numerical Example

An audit manager evaluates $a = 2$ auditors (Appraiser A and Appraiser B) conducting compliance audits on $N = 30$ loan files across $t = 2$ independent trials. The files are benchmarked against a Master Standard ($20$ compliant files, $10$ non-compliant files).

1. Cross-Tabulation Agreement Matrix (Appraiser A vs. Appraiser B)

2. Calculation of Observed Agreement ($P_o$) and Chance Agreement ($P_e$)

• Observed Proportion ($P_o$): $P_o = \frac{18 + 9}{30} = \frac{27}{30} = 0.900 \quad (90.0\%)$
• Expected Proportion by Chance ($P_e$): $P_{\text{both pass}} = \left( \frac{20}{30} \right) \times \left( \frac{19}{30} \right) = \frac{380}{900} = 0.4222$ $P_{\text{both fail}} = \left( \frac{10}{30} \right) \times \left( \frac{11}{30} \right) = \frac{110}{900} = 0.1222$ $P_e = 0.4222 + 0.1222 = 0.5444$

3. Compute Cohen's Kappa ($\kappa$)

$\kappa = \frac{P_o - P_e}{1 - P_e} = \frac{0.900 - 0.5444}{1 - 0.5444} = \frac{0.3556}{0.4556} = 0.7805$

• Conclusion: Since $\kappa = 0.7805 > 0.75$, the measurement system exhibits good to excellent inter-appraiser agreement.

---

Attribute MSA Troubleshooting & Best Practices

1. Boundary / Limit Samples: Establish visual physical limit boards representing maximum allowable defect thresholds at all inspection stations.
2. Blinding Operators: Ensure operators do not know which samples are reference standards or boundary samples during testing.
3. Periodic Re-certification: Conduct quarterly Attribute MSA studies to prevent inspector bias drift.

---

Audit MSA for Attribute Measurement Systems

The Body of Knowledge asks specifically that Black Belts use audit MSA for attribute measurement systems. An attribute agreement study answers whether appraisers agree; an audit MSA answers whether the whole inspection system -- appraiser, standard, procedure, environment, and decision rule -- produces the correct classification in normal operating conditions.

How it differs from a standard attribute agreement study

The difference matters because an inspection system can pass a controlled study and still fail in production. Appraisers who agree perfectly when handed 30 parts in a quiet room may miss defects when inspecting at takt on a moving line, or may drift after two hours.

Running an audit MSA

1. Define the audit sample. Draw units from the live flow immediately after the inspection station, including both accepted and rejected units. Rejected units matter: false rejects are as costly as escapes and are invisible if only accepted units are audited.
2. Establish the truth. Re-inspect with a reference method: a more capable gauge, an expert panel, or a downstream verification station.
3. Build the confusion matrix.

1. Compute the two error rates.

$\text{Miss rate} = \frac{\text{Escapes}}{\text{Total truly defective}}, \qquad \text{False alarm rate} = \frac{\text{False rejects}}{\text{Total truly good}}$

1. Audit the conditions, not just the outcome. Are limit samples current, unambiguous, and physically present? Is the standard written with decision rules for the borderline cases? Is the lighting to specification? Is the inspection time available at takt? Is the inspector rotating tasks to avoid vigilance decay?
2. Set a recurring interval and put it in the control plan, so the audit happens by schedule.

Typical findings

Human visual inspection is conventionally around 80% effective at detecting the defects present, and audit MSA is how an organization discovers where on that scale it sits. The most common systemic findings are ambiguous or missing limit samples, a written standard that does not resolve the borderline cases, and insufficient inspection time at the current takt -- none of which appear in a controlled attribute agreement study.

The corrective actions are correspondingly systemic: rewrite the decision rules, refresh the limit samples, fix the lighting, re-balance the line to release inspection time, or move detection upstream to error-proofing so that judgment is not required at all.