11.5 Types of Measurements, Unit Conversion, and Rounding Rules

11.5 Types of Measurements, Unit Conversion, and Rounding Rules

Why a Technician Is Tested on This

An inspection result is only as defensible as the method and the arithmetic behind it. Two technicians measuring the same part by two legitimate methods can produce different numbers, and a third technician converting those numbers into different units and rounding them can turn a conforming part into a rejection. The CQT Body of Knowledge therefore expects a technician to distinguish between types of measurement, convert between metric and English units, and apply truncation, rounding rules, and significant digits correctly on both positive and negative numbers.

---

The Four Types of Measurement

1. Direct Measurement

The instrument's own scale spans the dimension. An outside micrometer reading 1.2504 in on a shaft is a direct measurement: the reading is the size.

Selection note: direct measurement is preferred whenever the instrument's accuracy satisfies the 10:1 rule (Section 9.1), because it involves the fewest error contributors.

2. Differential (Comparative) Measurement

The instrument is set to zero, or to a known reference value, against a master, and thereafter reports only the deviation. A dial bore gage set with a master ring at 1.2500 in that reads +0.0004 on a part means the bore is 1.2504 in.

Why it is used despite needing a master:

• Much higher amplification is practical over a narrow range.
• It removes most of the instrument's absolute-scale error, because both the master and the part are read the same way.
• It is fast and repeatable in production, and it is the only practical approach for deep bores, thin walls, and high-volume checks.

Cardinal rule: the master's calibration is now part of the measurement. A comparative gage mastered against an out-of-calibration ring produces a perfectly repeatable wrong answer.

3. Derived Measurement

The value is calculated rather than read. Familiar examples:

The trap in derived measurement is error propagation. A wall thickness derived from an OD and an ID inherits the uncertainty of both. If OD and ID are each measured to plus or minus 0.0005 in, the derived wall carries roughly plus or minus 0.0005 in on the radius, which is worse than either input, and it is a poor substitute for measuring the wall directly with a tube micrometer or an ultrasonic thickness gage when the tolerance is tight.

4. Transfer Measurement

The tool has no readout. A telescoping gage is expanded inside a bore, locked, withdrawn, and then measured with a micrometer. A small-hole gage does the same for holes too small for a telescoping gage. An inside caliper transfers a bore or slot width to a rule or micrometer.

Why transfer measurement is the weakest method:

1. The operator must feel the correct contact in the bore (first feel).
2. The operator must then feel the correct contact when the micrometer closes on the gage (second feel).
3. The micrometer's own error is added on top.
4. Rocking the tool through the true diameter is a learned skill; an inexperienced technician consistently reads undersize.

Transfer tools remain valuable for access, such as blind bores, grooves, and internal recesses, and for one-off work where no comparative gage exists. They are a poor choice for a tight tolerance in production, where a dial bore gage or air gage should be used instead.

[!IMPORTANT] Match the method to the tolerance, then to the access. Ask in order: (1) Can a direct instrument meet the 10:1 rule? (2) If not, can a comparative gage with a proper master meet it? (3) Is the value better measured directly than derived from other measurements? (4) Only if access forces it, use a transfer tool, and widen the expected measurement uncertainty accordingly.

---

Conversion Between Metric and English Units

Since 1959 the inch has been defined as exactly 25.4 millimetres. Length conversion is therefore exact, and any error comes from how the result is rounded, not from the factor.

The Converted Tolerance Trap

Converting a dimension is arithmetic. Converting a tolerance is a decision.

A drawing calls a bore 1.2500 in plus or minus 0.0005 in. Converted exactly:

• Nominal: 1.2500 x 25.4 = 31.750 mm
• Tolerance: 0.0005 x 25.4 = 0.0127 mm

If the technician tidies the tolerance to plus or minus 0.013 mm, the metric limits are now wider than the inch limits, and parts that the original drawing rejects will pass in metric. Converted tolerances are rounded inward, to plus or minus 0.012 mm in this case, so the converted specification is never more permissive than the original. Better still, keep enough decimal places that no rounding decision is needed, and always record which set of limits is the governing requirement.

Significant-Digit Discipline in Conversion

Carry full precision through the whole calculation and round once, at the end, to the resolution the record requires. Rounding at each intermediate step accumulates error. A converted value should also keep the same relative precision as the original: converting a caliper reading of 1.250 in (four significant figures) into 31.750000 mm falsely advertises eight significant figures of measurement that never existed. 31.75 mm is the honest result.

---

Significant Digits, Truncation, and Rounding

Significant Figures

The significant figures of a recorded value are the digits that carry real measurement information.

On an inspection record, the trailing zeros matter. Writing 1.25 in when the micrometer resolves to 0.0001 in throws away information and makes the record look as though it came from a caliper. Record to the instrument's resolution: 1.2500 in.

Truncation Versus Rounding

• Truncation discards the unwanted digits with no adjustment. It always moves the value toward zero, so it introduces a systematic bias: on a set of positive measurements, truncation systematically under-reports.
• Rounding adjusts the retained digit according to what was discarded, so the residual error is random rather than systematic.

Read the negative rows carefully: truncating -2.78 gives -2.7, which is larger (less negative) than the original, while rounding gives -2.8. Truncation moves toward zero from both directions. This is exactly why the number of distinct categories in a Gage R&R study is truncated rather than rounded (Section 9.4): the standard deliberately takes the conservative integer.

The Tie Rule: Round Half to Even

When the discarded portion is exactly 5 with nothing after it, always rounding up biases a long series of results upward. ASTM E29 therefore specifies rounding so that the last retained digit is even.

If the discarded portion is greater than 5, or is a 5 followed by any non-zero digit, always round up in magnitude: 2.2501 rounds to 2.3, not 2.2.

Deciding Conformance: The Two ASTM E29 Methods

The question "is 1.2505 in inside a limit of 1.250 in maximum?" has two defensible answers, and the governing document must say which applies.

The absolute method is the stricter interpretation and is the usual default in dimensional inspection unless the specification invokes the rounding method. Do not choose per part. Fix the method in the inspection plan and apply it consistently; a technician who switches methods to make a borderline part pass has committed a data-integrity violation, not a rounding judgment.

[!CAUTION] Round once, at the end. Converting, averaging, and computing derived values all carry full precision until the final reported number. A technician who rounds a converted value, then averages the rounded values, then rounds the average, has quietly changed the result.

---

Worked Examples

Worked Example 1: Deciding a Borderline Dimension

Scenario: A print calls a shoulder at 0.750 in maximum. A digital micrometer reads 0.7504 in. The inspection plan invokes the absolute method.

1. The limit is 0.750 in, absolute.
2. The observed value 0.7504 in exceeds the limit.
3. Disposition: nonconforming. Tag it, write the NCR, and route it to MRB.

If the plan had instead invoked the rounding method, the observed value would be rounded to three decimals, 0.7504 becomes 0.750, and the part would conform. The measured number did not change; the rule agreed in advance decided the outcome.

Worked Example 2: Converting a Toleranced Dimension

Scenario: Convert 2.3750 in plus or minus 0.0015 in to millimetres for an overseas supplier, stating three decimal places.

1. Nominal: 2.3750 x 25.4 = 60.325 mm (exact).
2. Tolerance: 0.0015 x 25.4 = 0.0381 mm.
3. Round the tolerance inward to the nearest 0.001 mm: plus or minus 0.038 mm, because 0.038 is smaller than 0.0381 and therefore sits inside the inch limits.
4. State the result: 60.325 mm plus or minus 0.038 mm, with a note that the inch dimension governs.

Rounding the tolerance outward to plus or minus 0.039 mm would have widened the acceptance band by 0.0009 mm on each side and accepted parts the original drawing rejects.

Worked Example 3: A Derived Value and Its Hidden Assumption

Scenario: A tube is specified at a wall thickness of 0.065 in plus or minus 0.005 in. The technician measures OD = 1.0620 in and ID = 0.9330 in, each with an expanded uncertainty of about plus or minus 0.0004 in.

1. Derived wall: (1.0620 - 0.9330) / 2 = 0.1290 / 2 = 0.0645 in.
2. Propagated uncertainty: both inputs contribute, and the division by two halves them, giving roughly plus or minus 0.0003 in on the wall, but only if the OD and ID are measured on the same axis.
3. The hidden assumption: if the tube is eccentric, the derived wall is the average wall and says nothing about the thin side. A direct tube-micrometer or ultrasonic reading at the thin side is the correct method.

Conclusion: the derived value of 0.0645 in is inside the tolerance, but it does not demonstrate conformance. Derived measurements must always be checked against the assumptions built into the formula.

---

Common Exam Traps for CQT Candidates

[!CAUTION] Trap 1: Calling a dial bore gage reading a direct measurement. It is comparative: the reading is a deviation from a master, and the master's calibration is part of the result.

Trap 2: Treating a transfer tool as equivalent to a direct instrument. A telescoping gage plus a micrometer stacks two operator feels and two instruments; expect substantially worse repeatability in a Gage R&R study.

Trap 3: Rounding a converted tolerance outward. Converted tolerances are rounded inward so the converted specification is never looser than the original.

Trap 4: Confusing lbf-ft with lbf-in. They differ by a factor of 12, and a torque specification converted with the wrong one is off by an order of magnitude.

Trap 5: Applying the degF-to-degC offset to a temperature interval. Thermal expansion calculations use intervals, where 1 degC equals 1.8 degF with no 32-degree offset.

Trap 6: Assuming 5 always rounds up. ASTM E29 rounds an exact half to the nearest even last digit, while a 5 followed by any non-zero digit always rounds up in magnitude.

Trap 7: Truncating and rounding negative numbers the same way. Truncation moves toward zero (-2.78 becomes -2.7); rounding moves to the nearest value (-2.78 becomes -2.8).