Notation, Unit Conversions, Ratios, and Decibels

Daily operations in calibration laboratories require rapid, error-free execution of applied technical mathematics. From converting non-SI process transmitter readings into base SI units to resolving dynamic decibel attenuation on RF spectrum analyzers, the calibration technician must possess a rigorous command of mathematical principles.

---

Scientific vs. Engineering Notation and SI Metric Prefixes

In metrology, physical quantities span dozens of orders of magnitude—from sub-atomic displacements in laser interferometers ($10^{-12}\text{ m}$) to electrical insulation resistances ($10^{12}\ \Omega$).

Scientific Notation

A number expressed in scientific notation has the form:

$$m \times 10^n, \quad \text{where } 1 \le |m| < 10 \text{ and } n \in \mathbb{Z}$$

• Example: $1,475,000\ \Omega = 1.475 \times 10^6\ \Omega$
• Example: $0.000000035\text{ s} = 3.5 \times 10^{-8}\text{ s}$

Engineering Notation

Engineering notation imposes an additional operational constraint: the exponent $n$ must be an integer multiple of 3 ($n \in \{\dots, -9, -6, -3, 0, 3, 6, 9, 12, \dots\}$), and the mantissa $m$ satisfies $1 \le |m| < 1000$.

Engineering notation is preferred in calibration because every exponent of 3 corresponds directly to an established SI prefix, facilitating direct instrument range selection:

---

English to Metric Unit Conversions

Under international treaty agreements dating back to the 1959 International Yard and Pound Agreement, English (US Customary) units are legally defined in terms of exact SI metric equivalents.

Length, Area, and Volume Conversions

• Length:

$$1\text{ inch (in)} = 25.4\text{ mm} = 0.0254\text{ m} \quad \text{(EXACT by definition)}$$ $$1\text{ foot (ft)} = 12\text{ in} = 304.8\text{ mm} = 0.3048\text{ m}$$ $$1\text{ microinch (}\mu\text{in)} = 10^{-6}\text{ in} = 0.0254\ \mu\text{m} = 25.4\text{ nm}$$

• Area:

$$1\text{ in}^2 = (25.4\text{ mm})^2 = 645.16\text{ mm}^2 = 6.4516 \times 10^{-4}\text{ m}^2$$

• Volume:

$$1\text{ in}^3 = (25.4\text{ mm})^3 = 16,387.064\text{ mm}^3 = 16.387064\text{ cm}^3 = 16.387064\text{ mL}$$ $$1\text{ US Gallon} = 231\text{ in}^3 = 3.785411784\text{ Liters (L)}$$

Mass and Force Conversions

• Mass:

$$1\text{ pound-mass (lb)} = 0.45359237\text{ kg} \quad \text{(EXACT by definition)}$$ $$1\text{ ounce (oz)} = \frac{1}{16}\text{ lb} = 28.349523125\text{ g}$$

• Force:

$$1\text{ pound-force (lbf)} = 1\text{ lb} \times 9.80665\text{ m/s}^2 = 4.4482216152605\text{ N} \approx 4.44822\text{ N}$$

Pressure Conversions

Pressure is force per unit area ($P = F/A$). Converting between English and metric pressure units is a core CCT skill:

$$1\text{ psi} = \frac{1\text{ lbf}}{1\text{ in}^2} = \frac{4.4482216\text{ N}}{6.4516 \times 10^{-4}\text{ m}^2} = 6,894.757\text{ Pa} = 6.894757\text{ kPa}$$

Ratios, Percentages, Parts Per Million (ppm), and Decibels (dB)

Percentages vs. Parts Per Million (ppm)

In high-accuracy electrical and dimensional metrology, fractional tolerances are too minute to express comfortably as percentages. The metrology community uses parts per million (ppm):

$$\text{Fraction} = \frac{\text{Error}}{\text{Nominal}}$$ $$\text{Percentage (\%)} = \text{Fraction} \times 10^2 = \frac{\text{Error}}{\text{Nominal}} \times 100$$ $$\text{Parts Per Million (ppm)} = \text{Fraction} \times 10^6 = \frac{\text{Error}}{\text{Nominal}} \times 1,000,000$$

Direct Conversion Rules

• $1\% = 10,000\text{ ppm}$
• $0.1\% = 1,000\text{ ppm}$
• $0.01\% = 100\text{ ppm}$
• $0.001\% = 10\text{ ppm}$
• $0.0001\% = 1\text{ ppm}$

Worked Example: Converting ppm Tolerance to Absolute Voltage

A multifunction calibrator is specified on its $10.00000\text{ V}$ range with a 1-year specification of:

$$\text{Tolerance} = \pm (15\text{ ppm of Reading} + 3\text{ ppm of Range})$$

If the technician sets the calibrator output to $5.00000\text{ V}$:

$$\text{Reading error} = 5.00000\text{ V} \times (15 \times 10^{-6}) = 0.000075\text{ V} = 75\ \mu\text{V}$$ $$\text{Range error} = 10.00000\text{ V} \times (3 \times 10^{-6}) = 0.000030\text{ V} = 30\ \mu\text{V}$$ $$\text{Total Absolute Tolerance} = 75\ \mu\text{V} + 30\ \mu\text{V} = 105\ \mu\text{V} = 0.000105\text{ V}$$

Decibels (dB) in Power and Voltage Metrology

The decibel (dB) is a logarithmic ratio used in RF, acoustics, and signal processing.

Power Ratios ($10 \log_{10}$)

Because power is directly proportional to energy, the decibel power ratio is defined as:

$$\text{dB} = 10 \log_{10}\left(\frac{P_2}{P_1}\right)$$

• $+3\text{ dB} \approx 2\times\text{ power}$ (e.g., $10\text{ mW} \to 20\text{ mW}$)
• $+10\text{ dB} = 10\times\text{ power}$ (e.g., $1\text{ mW} \to 10\text{ mW}$)
• $+20\text{ dB} = 100\times\text{ power}$ (e.g., $1\text{ mW} \to 100\text{ mW}$)
• $-3\text{ dB} \approx 0.5\times\text{ power}$ (half-power cut-off frequency)
• $-10\text{ dB} = 0.1\times\text{ power}$ ($90\%$ attenuation)

Voltage and Field Ratios ($20 \log_{10}$)

By Joule's law, power is proportional to voltage squared ($P = V^2 / R$). When impedances are equal ($R_1 = R_2$):

$$\text{dB} = 10 \log_{10}\left(\frac{V_2^2 / R}{V_1^2 / R}\right) = 10 \log_{10}\left[\left(\frac{V_2}{V_1}\right)^2\right] = 20 \log_{10}\left(\frac{V_2}{V_1}\right)$$

• $+6\text{ dB} \approx 2\times\text{ voltage}$ (doubling voltage)
• $+20\text{ dB} = 10\times\text{ voltage}$ (decade voltage increase)
• $+40\text{ dB} = 100\times\text{ voltage}$
• $-6\text{ dB} \approx 0.5\times\text{ voltage}$ (half-voltage)
• $-20\text{ dB} = 0.1\times\text{ voltage}$ ($90\%$ voltage drop)

Worked conversion checks with units

Consider a flow indication of 2 US gallons per minute. The volume factor is approximately 3.785412 litres per US gallon, giving about 7.570824 litres per minute. Dividing by sixty gives about 0.1261804 litre per second. The gallon definition matters: the imperial gallon has a different volume. A number labeled only “gallons” is incomplete information for a defensible conversion. Record the specified convention before calculating.

Area and volume provide another check. A rectangular opening measuring 2 inches by 3 inches has area 6 square inches. Multiplying by 645.16 square millimetres per square inch gives 3870.96 square millimetres. Multiplying only by the linear factor 25.4 would leave incorrect dimensions. A cubed-length conversion likewise needs the cubed linear factor, not the area factor. Cancel units on paper before entering numbers into a calculator.

Now suppose an amplifier’s output power is ten times its input power. The gain is 10 dB. If input and output resistances are equal, the associated voltage ratio is the square root of ten, approximately 3.162. A tenfold voltage ratio would instead correspond to a hundredfold power ratio and 20 dB under those equal-resistance conditions. With different resistances, calculate the actual power ratio from voltage squared divided by resistance. Do not use twenty times the voltage logarithm as a substitute for that impedance information.

To reverse the calculation, a power ratio equals ten raised to the gain in decibels divided by ten. Thus a negative 10 dB attenuation leaves one tenth of the initial power. Check direction: output divided by input gives negative gain for attenuation, while input divided by output gives a positive loss. Both conventions can appear in instrument documentation, so identify the stated ratio and sign before comparing results.

A conversion factor can be exact while a measurement remains uncertain. The defined inch-to-metre factor does not improve the uncertainty of the measured length. Convert the result and its uncertainty consistently, retaining intermediate precision until reporting.

Pressure-column conventions matter. The 248.84 Pa water-column factor above is for 60°F, not 20°C. Conventional inch-of-water uses a different factor, approximately 249.0889 Pa. Check the instrument convention before conversion. See NIST SP 811 conversion factors.