Sensitivity, Variability, and Precision

Quantifying instrument operating characteristics is the core technical activity of calibration metrology. Every measuring device exhibits static and dynamic behaviors that dictate its fitness for purpose, including sensitivity, repeatability, reproducibility, bias, linearity, drift, hysteresis, and deadband. Calibration technicians must distinguish between random variability and systematic error, model instrument transfer functions, and recognize the physical mechanisms—such as mechanical backlash, viscoelastic creep, and thermal aging—that cause an instrument's indications to deviate from true reference values.

---

The Measurand and Influence Quantities

The Measurand (VIM 2.3)

In the International Vocabulary of Metrology (VIM), the measurand is defined as the "quantity intended to be measured". While deceptively simple, an incomplete definition of the measurand introduces definitional uncertainty ($u_{\text{def}}$), which acts as an irreducible uncertainty floor in the calibration budget.

A rigorous definition of a measurand requires stating:

1. The exact physical entity and state (e.g., "diameter of cylinder master SN-1048").
2. The spatial location and orientation of measurement (e.g., "at mid-height, along the orthogonal axis A-B").
3. The standard environmental reference state (e.g., "at $20.00^\circ\text{C}$ and $101.325\text{ kPa}$ barometric pressure").
4. The physical contact conditions (e.g., "under an applied measuring force of $0.50\text{ N} \pm 0.05\text{ N}$ using spherical carbide anvils of $5\text{ mm}$ radius").

Influence Quantities

An influence quantity is a quantity that, in a direct measurement, does not affect the quantity that is actually measured, but affects the relationship between the indication and the measurement result (VIM 2.52).

• Environmental Influence Quantities: Ambient temperature ($T$), relative humidity (RH), barometric pressure ($P_{\text{atm}}$), gravitational acceleration ($g$), and ambient magnetic or electrostatic fields.
• Operational Influence Quantities: Mains line voltage stability, RF radiated electromagnetic interference (EMI), mechanical foundation vibrations, and operator body heat dissipation.

Technicians must either actively control influence quantities within tightly specified tolerances (e.g., $20^\circ\text{C} \pm 0.2^\circ\text{C}$ in a dimensional laboratory) or continuously log them to apply deterministic mathematical corrections.

---

Variability and Random Dispersion

Nature of Measurement Variability

No measurement process yields identical results upon indefinite replication. Variability represents the random dispersion of measurement results obtained under repeated trials. It arises from an accumulation of microscopic, uncorrelated, and unpredictable physical variations (e.g., electronic thermal Johnson noise, air turbulence in optical paths, mechanical stick-slip friction in indicator pivots).

Statistical Metrics and Type A Evaluation

Under the GUM framework, variability is quantified through Type A evaluation of measurement uncertainty, calculated via statistical analysis of series of observations:

1. Sample Mean ($\bar{x}$): The best estimate of the expectation of the measurand:

$$\bar{x} = \frac{1}{n} \sum_{i=1}^n x_i$$

1. Sample Standard Deviation ($s$): Quantifies the dispersion of the individual observations around the mean:

$$s = \sqrt{\frac{1}{n - 1} \sum_{i=1}^n (x_i - \bar{x})^2}$$

1. Standard Error of the Mean ($s_{\bar{x}}$): Represents the standard uncertainty of the calculated mean value:

$$u_A = s_{\bar{x}} = \frac{s}{\sqrt{n}}$$

For independent observations from a stable process with finite variance, the standard uncertainty of the mean is $s/\sqrt{n}$. Statistical control or identical setup alone does not establish independence: check drift and autocorrelation. Pool compatible within-run estimates only when they represent the intended repeatability population; between-day shifts may need a separate component.

$$s_p = \sqrt{\frac{\sum_{j=1}^k (n_j - 1) s_j^2}{\sum_{j=1}^k (n_j - 1)}}$$

---

Sensitivity and Sensitivity Drift

Defining Sensitivity

Sensitivity ($S$) of a measuring system is the quotient of the change in an indication ($\Delta y$) and the corresponding change in the value of the quantity being measured ($\Delta x$):

$$S = \lim_{\Delta x \to 0} \frac{\Delta y}{\Delta x} = \frac{dy}{dx}$$

Sensitivity represents the derivative slope of the instrument's calibration curve (transfer function):

• In a linear instrument ($y = m x + b$), sensitivity is constant across the entire range ($S = m$).
• In non-linear sensors (such as thermocouples or thermistors), sensitivity varies dynamically with operating point. For example, a Type K thermocouple exhibits a sensitivity of $\approx 39\ \mu\text{V}/^\circ\text{C}$ at $0^\circ\text{C}$, rising to $\approx 42\ \mu\text{V}/^\circ\text{C}$ at $500^\circ\text{C}$.

Distinguishing Sensitivity from Resolution

A critical conceptual trap in metrology is confusing sensitivity with resolution:

• Sensitivity is a transfer ratio: output change divided by input change (e.g., $10.0\text{ mV/kPa}$ or $20\text{ mm/g}$).
• Resolution is the smallest input increment that produces a discernible output change (e.g., $0.01\text{ kPa}$ or $0.001\text{ g}$).

Example: An instrument can have exceptionally high sensitivity (e.g., a gain of $1,000,000\text{ V/V}$), but if internal amplifier noise is $50\text{ mV}$, its effective measurement resolution will be terribly degraded. Conversely, a low-sensitivity transducer paired with an ultra-low-noise 24-bit ADC can resolve minute signal changes.

Sensitivity Drift (Gain Error)

Sensitivity drift (also termed scale factor drift or span error) occurs when the slope of the calibration curve changes over time or across temperature, while the zero intercept remains fixed:

$$y(x, T) = S_0[1 + TC_S \cdot (T - 20^\circ\text{C})] \cdot x + y_0$$

Here $TC_S$ is a fractional change per degree in this equation. Convert ppm/°C to reciprocal degrees by multiplying by $10^{-6}$, or percent/°C by dividing by 100, before substitution. Coefficients expressed relative to full scale require the model specified for that instrument. Temperature can affect the elastic element and strain-gage response.

---

Repeatability vs. Reproducibility

The distinction between repeatability and reproducibility is central to the ASQ CCT Body of Knowledge, Measurement Systems Analysis (MSA), and the GUM.

Repeatability (Measurement Repeatability, VIM 2.21)

Repeatability is measurement precision under a set of repeatability conditions of measurement:

• The same measurement procedure.
• The same operators.
• The same measuring system.
• The same operating conditions and physical location.
• Replicate measurements on the same or similar objects over a short period of time.

Repeatability isolates the intrinsic short-term noise and mechanical stability of the equipment (often labeled Equipment Variation, $EV$). It is evaluated by having one technician measure a single artifact $10$ to $20$ consecutive times without disturbing the physical setup.

Reproducibility and intermediate precision

VIM reproducibility conditions include different locations, operators, and measuring systems, with the changed conditions specified. Changes of operator or day within one laboratory can instead be intermediate precision conditions. Traditional gage R&R uses “reproducibility” for between-appraiser variation; identify that study convention rather than silently treating it as the full VIM definition.

In Gage R&R studies, the total measurement system variance ($\sigma_{\text{gage}}^2$) is the orthogonal sum of repeatability and reproducibility:

$$\sigma_{\text{gage}}^2 = \sigma_{\text{repeatability}}^2 + \sigma_{\text{reproducibility}}^2$$ $$\text{Gage R\&R} = \sqrt{EV^2 + AV^2}$$

Where $EV$ is Equipment Variation (repeatability) and $AV$ is Appraiser Variation (reproducibility across operators).