1.5 Probability of Detection, Reliability, and Technique Validation

1.5 Probability of Detection, Reliability, and Technique Validation

The current ASNT NDT Level III Magnetic Particle Testing exam topics place "assist in MT POD studies (e.g., reliability, sensitivity)" and "prepare MT technical demonstrations" inside MT Principles and Theory. That placement is deliberate: a Level III is expected to reason about MT as a statistical detection process, not as a pass/fail ritual. A technician asks "did the indication form?" A Level III asks "what fraction of cracks of this size, in this material, under this technique, would form an indication that this inspector would call?"

---

Why MT Capability Is a Probability, Not a Guarantee

ASME Section V, Article 7 states the physical boundary plainly: "the sensitivity is greatest for surface discontinuities and diminishes rapidly with increasing depth of subsurface discontinuities below the surface." That single sentence is a capability statement — MT is a surface and near-surface method whose detection rate falls off steeply with depth.

Detection is probabilistic because every link in the chain varies:

Because all of these vary, the honest expression of method capability is a POD curve: detection probability plotted against flaw size, rising from near zero at small sizes toward an asymptote below 100%.

---

Reading a POD Curve: a50, a90, and a90/95

Three points on the curve carry contractual weight:

• $a_{50}$ — the flaw size detected 50% of the time. It is the midpoint of the curve and a poor engineering basis, because half the flaws of that size are missed.
• $a_{90}$ — the flaw size detected 90% of the time. A point estimate from one data set.
• $a_{90/95}$ — the flaw size for which detection is 90% probable at 95% statistical confidence. This is the value damage-tolerance analysts use to set the assumed initial flaw size for inspection intervals, because it accounts for the finite size of the demonstration sample.

$a_{90/95} \ge a_{90} \quad \text{always; the gap shrinks only as the number of characterized targets grows.}$

Level III reasoning trap: a demonstration that found 9 of 10 cracks does not establish $a_{90}$. With 10 targets the 95% lower confidence bound on a 90% observed rate sits near 60%, which is why capability demonstrations require dozens of characterized flaws rather than a handful.

Two Analysis Models

MIL-HDBK-1823, Nondestructive Evaluation System Reliability Assessment, is the reference most aerospace customers invoke for POD demonstrations, and it distinguishes two data types:

1. Hit/miss analysis — each target is recorded only as found or not found. This is the natural model for MT, because a magnetic particle indication has no calibrated amplitude. Hit/miss analysis is statistically inefficient, so the handbook guidance calls for a substantially larger target set (on the order of 60 flawed sites) to support a defensible $a_{90/95}$.
2. Signal response ("$\hat{a}$ versus $a$") analysis — each target yields a measured signal amplitude that is regressed against flaw size. Fewer targets are needed, but the method requires a quantitative signal, which eddy current and ultrasonic testing provide and conventional MT does not.

A Level III should also insist that the demonstration record the false call rate alongside POD. A technique tuned to maximum sensitivity by over-magnetizing will show high POD and an unusable false-call rate from background furring; reporting one without the other is meaningless.

---

System Performance Verification Is Not a POD Study

This is the distinction most often confused on the examination:

A QQI shim indication proves the field is adequate at the shim. It says nothing about whether a 0.020 in. deep tight fatigue crack in the part's fillet radius would be found.

---

Designing an MT Technical Demonstration

"Prepare MT technical demonstrations" is an explicit Level III task, and demonstrations are what auditors, customers, and design authorities actually witness. A defensible demonstration package contains:

1. Objective statement — the specific claim being demonstrated (for example, "detection of surface-breaking cracks of 0.060 in. length or greater through 0.004 in. of epoxy primer using the AC yoke technique").
2. Representative specimens — same alloy, heat treatment, surface finish, and geometry as production hardware, containing documented flaws whose size has been established by an independent means (fracture, sectioning, replication, or a calibrated secondary method). Notches are not cracks; a notch demonstration over-states capability.
3. Controlled process — the exact written procedure under demonstration, with field verification, lighting readings, bath concentration, and current settings recorded as data, not as assumptions.
4. Blind conditions where possible — the inspector should not know flaw locations or counts. A demonstration in which the operator is pointing at a known crack proves only that the crack is visible.
5. Recorded outcome — hits, misses, false calls, and the identity and qualification of the personnel performing and witnessing the work. ASME Section V Mandatory Appendix I requires exactly this kind of record for the coated-surface AC yoke demonstration: identification of the qualification, the personnel performing and witnessing, equipment and materials, illumination levels, and results.
6. Disposition — an explicit statement of what the demonstration does and does not qualify, signed by the Level III.

---

Worked Level III Scenario: An Over-Claimed Capability

Scenario: A turbine overhaul shop's MT procedure states that wet fluorescent MT "detects all surface cracks greater than 0.030 in. long." The customer's damage-tolerance engineer asks for the $a_{90/95}$ value supporting a 2 000-cycle inspection interval. The shop offers its daily Ketos ring logs (three holes at 1 400 A for the last two years) and a photograph of a 0.035 in. crack found on a QQI-verified setup.

Level III analysis:

1. The claim is unsupported. "Detects all" is an absolute statement no NDE method can support. The procedure language must be corrected to a probabilistic statement or removed.
2. The Ketos ring evidence is the wrong evidence. Ring performance is a system baseline check under ASME V T-766; it demonstrates day-to-day reproducibility of the bath and power supply, not flaw-size capability on turbine hardware.
3. The single photograph is anecdote, not data. One hit provides no confidence bound.
4. Correct path forward. Commission a hit/miss POD demonstration on retired blades containing characterized service fatigue cracks, with a target set large enough to bound $a_{90/95}$, multiple certified inspectors, blind presentation, and recorded false calls. Until that data exists, the Level III must tell the customer that the shop can support a procedure qualification claim but not an $a_{90/95}$ value — and must not let an interval be set on a number the data cannot carry.
5. Document the limitation. The honest interim statement is that the technique has been qualified by demonstration to detect the smallest rejectable indication defined in the acceptance criteria, with capability below that size uncharacterized.