10.5 Software for Test Equipment & Software Validation

Why Software Is a Metrology Problem

Almost every instrument in a modern inspection department is software-driven. A coordinate measuring machine executes a part program that defines datums, probing strategy, and evaluation. A vision system runs a recipe of lighting, thresholds, and measurement tools. A hardness tester applies dwell profiles and converts scales. A surface profilometer applies filters and computes parameters. Data collection packages calculate control limits and capability indices.

That means a large share of the reported measurement is produced not by the sensor but by code and configuration. A calibration certificate proves the hardware measured a standard correctly. It says nothing about whether the program picked the right datums, whether the filter cutoff matched the drawing, or whether last week's software update changed the circle-fitting algorithm.

This is why ISO 9001:2015 clause 7.1.5.2 requires that monitoring and measuring resources be suitable and that their fitness be maintained, and why ISO/IEC 17025 clause 7.11 imposes explicit requirements on the management of data and information systems in accredited laboratories. In regulated sectors the obligation is sharper still: FDA-regulated manufacturers must validate software used as part of production or the quality system, and electronic records and signatures fall under 21 CFR Part 11.

Body of Knowledge topic III.E is tagged (Remember), so the exam tests recognition of the four control categories and the vocabulary of validation, not the ability to write a validation protocol.

---

1. Safeguarding

Safeguarding is protecting the software, the programs, and the data from unauthorized, accidental, or undetected change.

Access Control

Define user levels and enforce them with individual credentials, never a shared login taped to the monitor:

Individual credentials matter because they make the audit trail attributable. A shared account produces records that no one can be held to.

Program and Configuration Protection

• Released programs are locked against editing at the operator level.
• Master copies are stored write-protected in a controlled location separate from the machine.
• A checksum or hash of the released program lets anyone verify at a glance that the file running today is byte-for-byte the one that was approved.
• Change control governs modification: a request, a technical review, a re-verification, an approval, a new revision identifier, and a record of what changed and why.
• Configuration settings — units, temperature compensation coefficients, probe qualification files, filter defaults — are as controllable as the program itself and must be captured in the baseline.

Data and System Protection

• Backup and restore on a defined schedule, with restore actually tested rather than assumed. An untested backup is a hope, not a control.
• Malware protection and network segregation. Metrology workstations frequently run legacy operating systems and are commonly isolated from general business networks for this reason.
• Physical security of the workstation and of removable media.
• Audit trail: a time-stamped, non-editable log of who changed what, when, and why. In FDA-regulated environments this is a Part 11 requirement; in general industry it is the practical evidence an auditor asks for first.
• Retention and archiving of programs and results for the contractually or legally required period, including the ability to still read archived data years later.

---

2. Identification of Attributes and Parameters

The BoK names this control explicitly, and it is the one candidates most often overlook. A measurement result is not reproducible unless the settings that produced it are recorded.

What must be identified with the result:

The practical consequence: a CMM report that states only "diameter 1.2503 in, PASS" is a weaker record than one that also states the program revision, the alignment, the number of points, and the fit method. When a customer and supplier disagree, this is exactly the information that resolves it — and section 4.5 shows why differing evaluation settings are one of the leading causes of correlation failure.

---

3. Functional Checks

A functional check verifies that the software, as configured today, still produces the right answer. It is the software analogue of running a check standard.

Common forms:

• Known artifact verification. Run a calibrated master — a ring gage, a step gage, a ball bar, a hardness test block, a roughness comparison specimen — through the actual production program and confirm the reported value matches the certified value within an agreed limit. Running the artifact through a separate "verification routine" instead of the real program tests the wrong thing.
• Startup and periodic checks. Daily or shift-start verification, recorded, with defined action limits and an escalation path when a check fails.
• Built-in self-test and diagnostics. Probe qualification, scale checks, communication tests, and system health routines supplied by the instrument maker.
• Verdict logic testing. Feed the program a known-bad part or a simulated out-of-tolerance value and confirm that it actually reports a failure. Software that passes everything looks perfect until you discover the limits were never loaded.
• Boundary and limit checks. Test values exactly at the specification limit, just inside, and just outside, to confirm the comparison operators behave as intended.
• Range and reasonableness checks. Confirm the program rejects impossible inputs rather than silently computing on them.

The failed-check rule mirrors the out-of-calibration rule from section 5.2: when a functional check fails, the results produced since the last successful check are suspect and require an impact assessment.

---

4. Comparison of Test Results

Comparison provides independent evidence that the software is right, because it checks the answer against something that did not come from the same code.

• Parallel running. Before releasing a new software version or a revised program, measure the same parts on the old and new configurations and compare. Unexplained differences block the release.
• Independent manual measurement. Verify a software-reported dimension against a hand measurement made with a different instrument and method.
• Independent calculation. Take the raw coordinates from a CMM report and hand-calculate the true position with the formula from section 7.4, then confirm the software's reported value. This single technique catches datum errors, sign errors, and radial-versus-diametral reporting mistakes.
• Inter-system and inter-laboratory comparison. The round-robin approach described in section 4.5, applied specifically to confirm software agreement.
• Reference data sets. Feed a known set of input data with a known correct answer through the calculation and confirm the output. National metrology institutes publish reference data sets for exactly this purpose.

---

Software Validation Methods

Validation is documented evidence that the software consistently does what it is intended to do in its intended use. The scope of validation is the intended use, not the product in the abstract — the same commercial software may need extensive validation in one application and very little in another.

The IQ / OQ / PQ Model

Testing Approaches

• Black-box testing exercises the software through its inputs and outputs without reference to internal code. This is what a user of purchased software can do, and it is normally sufficient.
• White-box (structural) testing examines internal logic and code paths. It applies to software developed in-house.
• Regression testing re-runs a defined set of previously passing tests after any change, to confirm the change broke nothing else.
• Risk-based scaling. Validation effort should be proportional to the consequence of the software being wrong. A program that dispositions a flight-safety characteristic warrants far deeper testing than one that logs a cosmetic count.
• Supplier assessment. A vendor's own development and testing evidence can reduce, but never eliminate, the user's obligation. The user must still confirm correct operation on site, in the intended use.
• Retrospective validation. Legacy systems already in service are validated using accumulated historical performance data plus targeted current testing.

Revalidation Triggers

Validation is not a one-time event. Revalidate, at least in part, after a software or firmware upgrade, an operating-system or hardware change, a change to the measurement method or algorithm, relocation of the system, a change in the intended use, or a discovered defect. Regression testing is the usual minimum response to any change.

Documentation Set

A validation package normally contains a validation plan (scope, approach, acceptance criteria, roles), protocols (the specific tests to be run, written and approved before execution), executed results with raw evidence, a traceability matrix mapping each requirement to the test that demonstrates it, deviation records with dispositions, and a summary report with an explicit release statement. Writing the protocol after seeing the results is a documentation failure auditors look for specifically.

---

Real Shop Scenario

A CMM passes its annual calibration and its daily ball-bar check. Three weeks later a customer rejects a lot for hole position, measuring 0.014 in against a 0.010 in position tolerance. The shop's CMM reported 0.007 in on the same parts.

The investigation follows the comparison-of-results technique. The inspector takes the raw X and Y coordinates from the shop's own CMM report and hand-calculates the position deviation, obtaining a radial offset of 0.0069 in — which doubles to 0.0138 in diametrally, closely matching the customer.

The root cause was a program revision made after a software upgrade in which the position output had been configured to report the radial deviation rather than the diametral value required by ASME Y14.5. The hardware was flawless and the calibration was current; the software configuration was wrong, and no functional check had ever fed a known-position artifact through the revised program to catch it.

Corrective action covered all four control categories: the program was corrected and locked with a checksum (safeguarding), the output parameter and its units were added to the report header (identification of attributes and parameters), a certified position artifact was added to the startup verification (functional check), and a manual recalculation was made mandatory during first-piece approval on any position characteristic (comparison of results).

---

Common Exam Traps

• Calibration does not cover software. A current calibration sticker on a CMM says nothing about the part program.
• Commercial off-the-shelf software still requires validation of its intended use; the vendor's testing supplements, but does not replace, the user's confirmation.
• The settings are part of the result. Version, program revision, datums, fit algorithm, and filter cutoff belong in the record.
• Test the pass/fail logic with a known-bad input. Software that never fails anything may simply have no limits loaded.
• Untested backups are not controls. Restore must be demonstrated.
• Protocols are written and approved before execution, not reconstructed afterward.
• Any change triggers regression testing, and significant changes trigger revalidation.