Resolution, Confidentiality, and Data Integrity

Resolution, Quantization Error, and Digit Flicker

Resolution (VIM 4.14)

Resolution is the smallest change in a quantity being measured that causes a perceptible change in the corresponding indication. In a digital device, display resolution commonly corresponds to one Least Significant Digit (LSD).

Quantization Error and the GUM Formulation

Digital systems convert continuous analog signals into discrete digital counts via an analog-to-digital converter (ADC). Because the true input voltage can sit anywhere within an interval of width $d = 1\text{ LSD}$ before toggling the display, digital conversion introduces an inescapable quantization error of $\pm 0.5\text{ LSD}$.

A rectangular model over $[-d/2,+d/2]$ is a common justified model for rounding to step $d$ when no position within a bin is favored. It is not a universal probability law for every ADC, noisy display, or threshold. Assess the indication process and avoid double-counting quantization already represented in observed repeatability.

The standard uncertainty associated with resolution is calculated as:

$$u_{\text{resolution}} = \frac{a}{\sqrt{3}} = \frac{d / 2}{\sqrt{3}} = \frac{d}{2\sqrt{3}} \approx 0.2887 \cdot d$$

Practical Calculation

A digital pressure calibrator reads to $0.001\text{ bar}$ ($d = 0.001\text{ bar}$). The half-width is $a = 0.0005\text{ bar}$. The standard uncertainty contributed to the Type B budget is:

$$u_{\text{res}} = \frac{0.001\text{ bar}}{2\sqrt{3}} = \frac{0.0005\text{ bar}}{1.732} = 0.000289\text{ bar}$$

Digit Flicker (Jitter)

When a measured input sits exactly on the transition boundary between two adjacent quantization codes, electronic thermal noise causes the ADC to toggle rapidly between the two values (e.g., bouncing between $10.004\text{ V}$ and $10.005\text{ V}$).

Technicians mitigate digit flicker by:

1. Increasing the ADC integration time to an integer number of power line cycles (e.g., $10\text{ NPLC}$ or $100\text{ NPLC}$).
2. Activating internal digital smoothing (moving-window averaging).
3. Selecting an instrument range where the reading utilizes the upper portion of the scale, moving the signal further above the ADC noise threshold.

---

Suitability for Use and Fitness for Purpose

An instrument can possess excellent calibration credentials yet remain completely unsuitable for use if its physical dynamics, operating envelope, or electrical loading characteristics do not match the target application.

Operating range

A 20–80% rule can be a useful heuristic for some analog instruments, but it is not universal. Modern meters can specify valid performance near full scale and at low readings. Choose the range by its complete reading, range, count, floor, bandwidth, and overload terms. A 0.01 V absolute bound on a 100 V range is 1% of a 1 V reading, even though it is only 0.01% of range. Compare that bound with the task requirement.

Circuit Loading and Impedance Matching

Loading must be evaluated and limited or corrected to meet the measurement requirement:

• Voltage Measurements: The voltmeter's input resistance ($R_{\text{in}}$) forms a voltage divider with the source resistance ($R_s$). To keep the loading error below $0.1\%$ ($1000\text{ ppm}$), the instrument's input impedance must satisfy:

$$R_{\text{in}} \ge 1000 \cdot R_s$$

• RF and Microwave Connections: Transmission lines require matched characteristic impedances ($Z_0 = 50\ \Omega$ or $75\ \Omega$). Mismatches cause standing-wave reflections governed by the voltage reflection coefficient:

$$\Gamma = \frac{Z_L - Z_0}{Z_L + Z_0}$$

Reflected power introduces severe ripple in frequency response calibrations.

---

Confidentiality, Data Integrity, and Metrological Compliance

ISO/IEC 17025:2017 Requirements

Data management is strictly regulated under international calibration laboratory standards:

• Confidentiality: Protect customer information under enforceable commitments and law. Information already made public by the customer or agreed for release has specific treatment. Required or authorized disclosure can require notification unless prohibited by law; confidentiality is not subject only to a law exception.
• Clause 7.11 (Control of data and information management): Laboratory Information Management Systems (LIMS), custom calibration software, and automated calculation spreadsheets (e.g., Excel workbooks) must be fully validated for functionality prior to use. Protect calculations and records from unauthorized changes and loss through validated, effective controls; cell locking is one implementation.

The ALCOA+ Framework in the Calibration Laboratory

Originally developed in regulated life sciences (FDA/cGMP), the ALCOA+ framework represents global best practice for data integrity in ISO/IEC 17025 accredited metrology:

The acronym means attributable, legible, contemporaneous, original, and accurate. Added principles are complete, consistent, enduring, and available. Legibility includes interpretable units and identifiers; completeness preserves required failing observations as well as passing ones. Consistency preserves chronological meaning, endurance protects records for the required period, and availability permits authorized retrieval.

1. Attributable: Link observations and approvals to the responsible person or automated system.
2. Contemporaneous: Record observations when made. Controlled paper notes can be valid original records if preserved and incorporated into the technical record. Transcription from memory or discarding the original is unacceptable.
3. Original: Preserve original observations and traceable amendments.
4. Accurate: Record available as-found and as-left information honestly and identify unavailable tests or unsafe conditions.

ALCOA+ is a useful data-integrity framework, especially in regulated environments; it is not a named mandatory ISO/IEC 17025 system. Cryptographic signatures and FDA Part 11 controls apply as required by the actual record scope and system, not automatically to every calibration.

Keep original observations and preserve an amendment history identifying the changed values, person, date, and reason where needed. Electronic controls must protect integrity, access, and recovery. An immutable database or UTC timestamp convention is one implementation choice, not a universal ISO prescription. FDA electronic-record requirements apply only when the relevant regulated records are in scope.

Hardcopy Error Correction Protocols

When recording calibration data on physical paper bench sheets, technicians must adhere to strict good documentation practices (GDP):

• The technician draws a single horizontal strikethrough line through the erroneous reading, ensuring the original value remains completely legible.
• The correct value is written immediately adjacent to the strikethrough.
• The correction is immediately initialed, dated, and accompanied by a brief explanation code (e.g., "transcription error" or "scale typo").
• The use of correction fluid (white-out), erasers, or obliteration scribbles is strictly prohibited in accredited calibration facilities.