Rounding, Algebra, Graphs, and Angular Conversions

Significant Figures and ASTM E29 Rounding Rules

Rules for Significant Figures

• Non-zero digits are always significant ($14.8$ has 3 sig figs).
• Any zeros between non-zero digits are significant ($100.05$ has 5 sig figs).
• Leading zeros are never significant; they serve only as decimal place-holders ($0.0042$ has 2 sig figs).
• Trailing zeros to the right of the decimal point are significant ($12.000$ has 5 sig figs).

Arithmetic Operation Rules

• Multiplication/division classroom convention: Match the least precise factor’s significant figures when no uncertainty information is provided. In calibration reporting, use the actual evaluated uncertainty and prescribed reporting rules rather than replacing them with this shortcut.

$$A = (12.45\text{ cm}) \times (2.3\text{ cm}) = 28.635\text{ cm}^2 \implies \mathbf{29\text{ cm}^2} \quad \text{(2 sig figs)}$$

• Addition / Subtraction: The final result is limited by the least precise decimal position (fewest decimal places), regardless of the total number of significant figures.

$$m = 102.450\text{ g} + 0.12\text{ g} + 1.2345\text{ g} = 103.8045\text{ g} \implies \mathbf{103.80\text{ g}} \quad \text{(2 decimal places)}$$

Banker's Rounding: The ASTM E29 "Round-to-Even" Standard

Rounding ties upward can create an upward tendency for some data distributions. Round-to-even is one way to avoid always favoring the upper value at exact ties; it does not prove zero bias for every dataset. Use the convention required by the method and validate its implementation.

ASTM E29 describes a round-to-even convention for exact ties. IEEE 754 supports several rounding modes, with ties-to-even commonly the default; it does not require every application to use that mode. Follow the convention specified for the result being reported:

1. If the digit following the last retained digit is strictly less than 5, round down ($14.24 \to 14.2$).
2. If the digit following the last retained digit is greater than 5, or is 5 followed by any nonzero digits, round up ($14.26 \to 14.3$).
3. If the digit following the last retained digit is exactly 5 (or 5 followed only by zeros):
   • If the last retained digit is EVEN, leave it unchanged (round down):

$$12.450 \implies \mathbf{12.4}$$

• If the last retained digit is ODD, round it up to make it even:

$$12.550 \implies \mathbf{12.6}$$

Notice that both $12.450$ and $12.550$ round to an even final digit ($12.4$ and $12.6$). Over millions of measurements, approximately half round up and half round down, reducing systematic tie-rounding bias when ties are reasonably balanced.

Truncation vs. Rounding

• Truncation: Drop digits beyond the specified position without rounding. For a positive 14.999 truncated to one decimal, the result is 14.9. Truncation toward zero moves negative values in the opposite numerical direction; it differs from floor and can introduce bias. Use it only when the task calls for it, such as conservative degrees-of-freedom lookup.
• Intermediate precision and order: Follow parentheses, powers, multiplication/division, then addition/subtraction. Retain adequate intermediate precision for the error budget and decision; two extra digits are a rule of thumb, not a universal guarantee. Round the final reported result by the required convention.

Algebraic Formula Manipulation

Technicians must be proficient in isolating unknown calibration variables:

• Solving for voltage from power and resistance: $P = \frac{V^2}{R} \implies V^2 = P \cdot R \implies V = \sqrt{P \cdot R}$
• Solving for temperature change from RTD resistance: $R_T = R_0 (1 + \alpha \Delta T) \implies \frac{R_T}{R_0} = 1 + \alpha \Delta T \implies \Delta T = \frac{R_T - R_0}{\alpha R_0}$

Graphical Analysis: Slope, Intercept, and Linear Interpolation

Linearity calibration curves relate instrument indication ($Y$) to assigned reference input value ($X$):

$$y = m x + b$$

• Slope ($m$): Represents sensitivity or scale factor:

$$m = \frac{y_2 - y_1}{x_2 - x_1} = \frac{\Delta y}{\Delta x}$$

• Y-intercept ($b$): Represents the zero offset (indication when input is zero):

$$b = y_1 - m x_1$$

Linear Interpolation

When a sensor calibration certificate provides discrete calibration points, intermediate values are determined by linear interpolation:

$$y = y_1 + \left( \frac{x - x_1}{x_2 - x_1} \right) (y_2 - y_1)$$

Worked Example

A thermocouple calibration table records output voltages for Type K thermocouples:

• At $T_1 = 100.0^\circ\text{C}$, $V_1 = 4.096\text{ mV}$
• At $T_2 = 110.0^\circ\text{C}$, $V_2 = 4.509\text{ mV}$

If the technician measures an output voltage of $V = 4.300\text{ mV}$, find the interpolated temperature $T$:

$$T = T_1 + \left( \frac{V - V_1}{V_2 - V_1} \right) (T_2 - T_1) = 100.0^\circ\text{C} + \left( \frac{4.300 - 4.096}{4.509 - 4.096} \right) (110.0 - 100.0)$$ $$T = 100.0 + \left( \frac{0.204}{0.413} \right) (10.0) = 100.0 + (0.49395 \times 10.0) = 100.0 + 4.94 = \mathbf{104.94^\circ\text{C}}$$

Angular Conversions and the Radian Small-Angle Approximation

Angular measurement is central to optical alignment, autocollimators, theodolites, and sine bar height calculations.

Angular Systems and Conversion Factors

• Degrees, Minutes, Seconds (DMS):

$$1^\circ = 60' = 3,600''$$ $$1' = \left(\frac{1}{60}\right)^\circ \approx 0.016667^\circ$$ $$1'' = \left(\frac{1}{3,600}\right)^\circ \approx 0.0002778^\circ$$

• Gradians (Gons): A full circle equals $400\text{ grads}$ ($100\text{ grads} = 90^\circ$).

$$1\text{ grad} = 0.9^\circ = 54' = 3,240''$$

• Radians (rad): The SI coherent derived unit of plane angle.

$$2\pi\text{ rad} = 360^\circ \implies 1\text{ rad} = \frac{180^\circ}{\pi} \approx 57.2957795^\circ \approx 206,264.8''$$ $$1\text{ arcsecond (1'')} = \frac{1}{206,264.8}\text{ rad} \approx 4.848137 \times 10^{-6}\text{ rad}$$

The Radian Small-Angle Approximation in Metrology

In dimensional metrology, the Taylor series expansion for trigonometric functions around $\theta = 0$ is:

$$\sin\theta = \theta - \frac{\theta^3}{3!} + \frac{\theta^5}{5!} - \dots$$ $$\tan\theta = \theta + \frac{\theta^3}{3} + \frac{2\theta^5}{15} + \dots$$

For small angles ($\theta < 1^\circ \approx 0.0175\text{ rad}$), the higher-order terms may be small, but acceptability depends on the error budget (the relative tangent approximation error at 1 degree is about 0.0102%). Therefore, when $\theta$ is expressed in radians:

$$\sin\theta \approx \tan\theta \approx \theta \quad (\text{in radians})$$

Calibration Application: Sine Bar Elevation Calculation

A sine bar of length $L$ resting on a surface plate is elevated at one end by a stack of gage blocks of height $H$ to set an angle $\theta$:

$$\sin\theta = \frac{H}{L} \implies H = L \cdot \sin\theta$$

Using the small-angle approximation for small angular adjustments:

$$H \approx L \cdot \theta \quad (\text{with } \theta \text{ in radians})$$

Worked Example

A technician sets up a $5.0000\text{ inch}$ sine bar to generate an angle of nominally $\theta = 10.00\text{ arcseconds}$ ($10''$). What end-height difference $H$ is required? In a practical setup use suitable equal-height supports and a fine height difference; this tiny difference is not a conventional standalone gage-block stack.

1. Convert angle from arcseconds to radians:

$$\theta = 10.00'' \times \left(4.848137 \times 10^{-6}\text{ rad/arcsec}\right) = 4.848137 \times 10^{-5}\text{ rad}$$

1. Calculate the required support-height difference $H$:

$$H = L \cdot \theta = (5.0000\text{ in}) \times (4.848137 \times 10^{-5}\text{ rad}) = 2.42407 \times 10^{-4}\text{ in} = 0.0002424\text{ in} = 242.4\ \mu\text{in}$$

If the technician attempted to use a standard calculator without radians, rounding errors would easily destroy the sub-microinch precision required for precision angle standards!

Reading slope, intercept, and extrapolated values

Suppose a hypothetical linear sensor gives 1 V at an input of zero units and 5 V at an input of 100 units. Its slope is 0.04 V per input unit and its intercept is 1 V. An observed 3 V corresponds to 50 units by interpolation. A 6 V observation would correspond to 125 units only if the same linear model remains valid beyond the measured interval. That last calculation is extrapolation, not evidence that the sensor is calibrated there.

Check the axes, units, scale, and whether the plot shows indication or error. A slope in an error plot represents change in error per input unit; it is not necessarily the sensor’s overall sensitivity. A nonzero error-plot intercept indicates an offset at the plotted origin. Residuals around a fitted line help reveal nonlinearity, but a high correlation or R-squared alone does not establish negligible residuals or fitness for the required tolerance.