7.3 Validity, Reliability (ICC, CV%, TE), and Smallest Worthwhile Change

Validity, Reliability (ICC, CV%, TE), and Smallest Worthwhile Change

Quick Answer: Validity confirms that a tool measures what it claims to measure (evaluated against a gold standard via Standard Error of the Estimate [$\text{SEE}$] and Bland-Altman $95\%$ Limits of Agreement [$\bar{d} \pm 1.96 \times \text{SD}_{d}$]). Reliability quantifies trial-to-trial reproducibility: Relative reliability ($\text{ICC}$) measures rank-order consistency ($\ge 0.80\text{–}0.90$), while Absolute reliability measures raw trial-to-trial noise via Typical Error ($\text{TE} = \text{SD}_{\text{diff}} / \sqrt{2}$) and Coefficient of Variation ($\text{CV}\% = [\text{TE} / \text{Mean}] \times 100\% \le 5\%$). Comparing Noise ($\text{TE}$) against the Signal ($\text{SWC} = 0.2 \times \text{SD}_{\text{between}}$) determines whether an athlete's observed change exceeds random error ($\text{MDC}_{95} = 1.96 \times \sqrt{2} \times \text{TE} \approx 2.77 \times \text{TE}$, or $\ge 1.5\text{–}2.0 \times \text{TE}$ in applied monitoring).

Before selecting a new wearable sensor, force plate metric, or field fitness test, the sport scientist must establish confidence in the measurement by evaluating three hierarchical psychometric properties: Validity (accuracy relative to truth), Reliability (precision and repeatability across trials), and Sensitivity / Responsiveness (the signal-to-noise capacity to detect meaningful biological change in an individual athlete).

---

Taxonomy of Measurement Validity in Sport Science

Validity defines the extent to which an instrument or test protocol accurately measures the specific physiological, biomechanical, or perceptual construct it is intended to assess.

1. Logical (Face) and Content Validity

• Face (Logical) Validity: The superficial perception by athletes and coaches that a test appears relevant to their sport. While scientifically subjective, high face validity is essential for securing maximal athlete effort and coaching buy-in.
• Content Validity: The systematic evaluation by domain experts that an assessment battery comprehensively samples all critical physiological and mechanical sub-components of competitive performance (for example, verifying that an ice hockey physiological profile includes repeated-sprint skating ability, anaerobic glycolytic power, and lower-body force production rather than solely an upper-body bench press).

2. Criterion Validity (Concurrent and Predictive)

Criterion validity evaluates a practical or novel testing device against an established gold-standard criterion reference.

• Concurrent Validity: Determined when the practical field device and the gold-standard criterion are measured simultaneously (e.g., validating a linear position transducer or optical jump mat against a $1000\text{ Hz}$ dual force-plate system, or validating a portable metabolic analyzer against a stationary Douglas bag / gold-standard laboratory cart). Rigorous concurrent validation requires three distinct statistical evaluations:
  1. Linear Correlation ($r$): Pearson product-moment correlation ($r$) quantifies whether the two devices rank subjects in a similar linear order. Crucial Limitation: A near-perfect correlation ($r = 0.98$) cannot detect systematic offset bias! If a contact mat overestimates every athlete's jump height by exactly $+6.0\text{ cm}$ compared to a force plate, $r$ will equal $1.00$ despite severe inaccuracy.
  2. Standard Error of the Estimate ($\text{SEE}$ / $\text{TEE}$): Also termed the Typical Error of the Estimate ($\text{TEE}$), this metric quantifies the typical residual scatter (prediction error) around the linear regression calibration line in the raw units of the criterion test:

$$\text{SEE} = \text{SD}_{y} \times \sqrt{1 - r^2}$$

1. Bland-Altman Agreement Analysis: The gold-standard graphical and statistical method for evaluating agreement between two measurement tools. For every subject $i$, the difference between the practical device and the criterion ($d_{i} = X_{\text{practical},i} - X_{\text{criterion},i}$) is plotted on the vertical axis against the mean of the two devices on the horizontal axis. Bland-Altman analysis yields two essential parameters:
   • Systematic (Fixed) Bias ($\bar{d}$): The mean difference across all subjects, showing whether the practical tool systematically overestimates ($+\bar{d}$) or underestimates ($-\bar{d}$) the gold standard.
   • $95\%$ Limits of Agreement ($\text{LoA}_{95}$): The interval within which $95\%$ of individual discrepancies between the two devices are expected to lie:

$$\text{LoA}_{95} = \bar{d} \pm 1.96 \times \text{SD}_{d}$$

 Bland-Altman plots also expose **proportional bias (heteroscedasticity)**—where the magnitude of error expands at higher movement velocities or force outputs.

• Predictive Validity: Evaluates how well a baseline assessment score predicts a future criterion outcome (such as in-season match high-speed running, draft selection, or prospective non-contact soft-tissue injury). Predictive validity is quantified via regression $r^2$, odds/hazard ratios, Receiver Operating Characteristic ($\text{ROC}$) Area Under the Curve ($\text{AUC}$) (where $\text{AUC} = 0.50$ represents random chance and $\ge 0.80$ indicates strong discrimination), sensitivity (true positive rate), and specificity (true negative rate).

3. Construct Validity (Convergent and Discriminant / Known-Groups)

When evaluating an abstract theoretical quality that lacks a single physical gold standard (such as "sport-specific agility" or "central neuromuscular fatigue"), practitioners establish construct validity:

• Convergent Validity: Confirms that the new assessment correlates moderately to strongly with other established tests measuring the same underlying construct.
• Discriminant (Known-Groups) Validity: Confirms two properties: (1) the test statistically differentiates between distinct athlete tiers known to differ in that construct (known-groups validity, such as national-team starters outperforming sub-elite academy players on a reactive agility test); and (2) the test does not simply duplicate an unrelated physical quality ($r^2 < 0.50$ between linear $20\text{ m}$ sprint time and change-of-direction deficit, proving they measure distinct neuromuscular qualities).

4. Ecological Validity

Ecological validity reflects how faithfully the testing environment, equipment constraints, and perceptual-cognitive stimuli mirror actual competitive sport demands (e.g., skating on ice in full hockey gear while reacting to a visual stimulus versus pedaling a stationary cycle ergometer in a laboratory).

---

Quantifying Relative vs. Absolute Reliability

Reliability quantifies the consistency or reproducibility of a test across repeated trials when no true biological adaptation or fatigue has occurred. Total observed variance in a dataset equals true between-athlete variance ($\sigma_{\text{between}}^2$) plus within-athlete error variance ($\sigma_{\text{within}}^2$, which combines equipment technical error and within-subject biological day-to-day fluctuation).

1. Relative Reliability — Intraclass Correlation Coefficient ($\text{ICC}$)

Relative reliability assesses the degree to which individuals maintain their rank order (relative position) within a group across repeated trials, quantified by the Intraclass Correlation Coefficient ($\text{ICC}$):

$$\text{ICC} \approx \frac{\sigma_{\text{between}}^2}{\sigma_{\text{between}}^2 + \sigma_{\text{within}}^2} = 1 - \frac{\text{TE}^2}{\text{SD}_{\text{between}}^2}$$

• Koo & Li (2016) Classification Benchmarks: Based on the $95\%$ confidence interval of the $\text{ICC}$ estimate, values $< 0.50$ indicate Poor, $0.50\text{–}0.75$ Moderate, $0.75\text{–}0.90$ Good, and $> 0.90$ Excellent relative reliability.
• Crucial Examination Concept — Sensitivity to Sample Heterogeneity: Because $\text{ICC}$ is a mathematical ratio of between-athlete variance ($\text{SD}_{\text{between}}^2$) to total variance, $\text{ICC}$ is heavily influenced by the heterogeneity (spread) of the tested group. If a test is administered to a wide, heterogeneous sample (combining recreational athletes, collegiate players, and Olympians), $\text{SD}_{\text{between}}$ is very large, driving the $\text{ICC}$ close to $0.99$. Conversely, if the exact same instrument is used on a highly homogeneous elite squad (where all starters have nearly identical $10\text{ m}$ sprint times, yielding a tiny $\text{SD}_{\text{between}}$), the $\text{ICC}$ will appear artificially low (e.g., $0.60$) even when absolute trial-to-trial error ($\text{TE}$) is exceptionally small! Consequently, sport scientists must always pair $\text{ICC}$ with absolute reliability metrics.

2. Absolute Reliability — Typical Error ($\text{TE}$ / $\text{SEM}$) and Coefficient of Variation ($\text{CV}\%$)

Absolute reliability quantifies within-athlete trial-to-trial measurement noise directly in the raw physical units of the test (or as a percentage), completely independent of group heterogeneity.

• Typical Error of Measurement ($\text{TE}$): Popularized in sport science by Will Hopkins, $\text{TE}$ is calculated from the standard deviation of the within-subject change scores ($d_{i} = \text{Trial}_{2,i} - \text{Trial}_{1,i}$) across a test-retest reliability study:

$$\text{TE} = \frac{\text{SD}_{\text{diff}}}{\sqrt{2}}$$

Why divide by $\sqrt{2}$? Because every difference score ($d_{i} = \text{Trial}_{2} - \text{Trial}_{1}$) combines the independent random error variance of two separate trials ($\sigma_{\text{diff}}^2 = \sigma_{\text{trial } 1}^2 + \sigma_{\text{trial } 2}^2 = 2 \times \text{TE}^2$). Taking the square root gives $\text{SD}_{\text{diff}} = \sqrt{2} \times \text{TE}$, so the single-trial noise is $\text{TE} = \text{SD}_{\text{diff}} / \sqrt{2}$. In classical psychometric theory, $\text{TE}$ is equivalent to the Standard Error of Measurement ($\text{SEM}$):

$$\text{SEM} = \text{SD}_{\text{pooled}} \times \sqrt{1 - \text{ICC}}$$

• Coefficient of Variation ($\text{CV}\%$): Expresses the Typical Error as a dimensionless percentage of the sample grand mean ($\bar{X}$), allowing practitioners to compare relative noise across tests with different units (e.g., comparing force in Newtons against sprint time in seconds):

$$\text{CV}\% = \left(\frac{\text{TE}}{\bar{X}}\right) \times 100\%$$

Standard Sport Science Reliability Benchmark: A variable is generally considered highly reliable for applied athlete monitoring when $\text{CV}\% \le 5.0\%$ (and $\text{ICC} \ge 0.80\text{–}0.90$). If a metric exhibits higher noise (e.g., $\text{CV}\% = 7\%$ for early-phase $\text{RFD}$), averaging the mean of $k$ trials within a session reduces the effective Typical Error by a factor of $\sqrt{k}$ ($\text{TE}_{\text{mean of } k} = \text{TE} / \sqrt{k}$).

---

Establishing Confidence in Individual Change (Signal vs. Noise)

Once a test's Noise ($\text{TE}$) is known, the sport scientist must define the Signal—the minimum change in performance that actually matters in competition—and establish decision thresholds for individual athlete monitoring.

1. Smallest Worthwhile Change ($\text{SWC}$)

Based on Cohen's smallest worthwhile effect size ($d = 0.20$) for team-sport athletes, the Smallest Worthwhile Change ($\text{SWC}$) is calculated as one-fifth of the between-athlete standard deviation ($\text{SD}_{\text{between}}$) of a homogeneous peer reference group:

$$\text{SWC}_{0.2} = 0.2 \times \text{SD}_{\text{between}}$$

(When a more conservative moderate threshold is desired, practitioners use $\text{SWC}_{0.5} = 0.5 \times \text{SD}_{\text{between}}$; for solo individual events such as track sprinting or swimming, $\text{SWC}$ is calculated as $0.3 \times \text{CV}_{\text{within-athlete race-to-race}}$).

2. Evaluating Test Sensitivity ($\text{TE}$ vs. $\text{SWC}$)

Comparing the instrument's Noise ($\text{TE}$) against the practical Signal ($\text{SWC}$) classifies the sensitivity of the assessment:

• Good Sensitivity ($\text{TE} < \text{SWC}$): Measurement noise is smaller than a small worthwhile change; the test can confidently detect meaningful adaptations from a single measurement.
• Marginal / OK Sensitivity ($\text{TE} \approx \text{SWC}$): Noise equals the signal; the practitioner should average $2\text{ to } 4$ repeated trials per session so that $\text{TE} / \sqrt{k} < \text{SWC}$.
• Poor Sensitivity ($\text{TE} > \text{SWC}$): Measurement noise masks small worthwhile changes; only moderate-to-large changes can be detected.

3. Minimal Detectable Change ($\text{MDC}_{95}$) vs. Applied Monitoring Thresholds

• Minimal Detectable Change at $95\%$ Confidence ($\text{MDC}_{95}$ / Smallest Real Difference): Defines the statistical threshold above which an observed difference between two single measurements ($\Delta = X_{2} - X_{1}$) has less than a $5\%$ probability of occurring due to random measurement error alone:

$$\text{MDC}_{95} = 1.96 \times \sqrt{2} \times \text{TE} \approx 2.77 \times \text{TE}$$

At the $90\%$ confidence level, the threshold is:

$$\text{MDC}_{90} = 1.645 \times \sqrt{2} \times \text{TE} \approx 2.33 \times \text{TE}$$

• Applied Daily Monitoring Bands (Hopkins Framework): In high-performance sport, waiting for an athlete to change by $2.77 \times \text{TE}$ is often overly conservative and risks overlooking early neuromuscular fatigue (Type II error). In applied monitoring dashboards, sport scientists plot the athlete's score with an error bar of $\pm 1.0 \times \text{TE}$ (or $\pm 1.5\text{ to } 2.0 \times \text{TE}$, roughly $87\text{–}95\%$ confidence for a single score) against the shaded $\pm \text{SWC}$ band. Whenever an athlete's change $|\Delta|$ exceeds both the $\text{SWC}$ and the chosen $\text{TE}$ error threshold, the practitioner reports a clear, actionable real change.

---

Feasibility and Selecting KPIs from Validated Tools

Task 3D ends with two practical judgments: whether a tool is feasible in your setting, and which of its outputs should become key performance indicators (KPIs). A tool can be valid and reliable in a laboratory and still fail in your program.

When choosing KPIs from a feasible tool, keep a metric only if it passes four checks:

1. Relevance: It maps onto a determinant identified in the needs analysis, such as eccentric braking or maximal sprint speed.
2. Measurement quality: It has acceptable validity and absolute reliability, with $\text{TE} \le \text{SWC}$ or a manageable $\text{TE}/\sqrt{k}$ after averaging trials.
3. Sensitivity: It changes with training or fatigue in your population.
4. Actionability: A pre-agreed threshold triggers a specific decision.

For example, a force-plate CMJ export may offer more than 100 variables. A feasible KPI set might be jump height (outcome), $\text{RSI}_{\text{mod}}$ (strategy), and braking impulse or eccentric RFD (braking). Each should pass the reliability and SWC checks above, and none should be a near-duplicate of another.

---

Worked Calculation Example: Countermovement Jump Psychometrics

Scenario: A sport scientist conducts a test-retest reliability trial of countermovement jump ($\text{CMJ}$) height ($48\text{ hours}$ apart) on $N = 20$ varsity basketball players:

• Group Grand Mean ($\bar{X}$) $= 50.0\text{ cm}$
• Between-Athlete Standard Deviation ($\text{SD}_{\text{between}}$) $= 6.0\text{ cm}$
• Standard Deviation of Trial 2 minus Trial 1 Difference Scores ($\text{SD}_{\text{diff}}$) $= 1.414\text{ cm}$

Step 1 — Calculate Typical Error ($\text{TE}$):

$$\text{TE} = \frac{\text{SD}_{\text{diff}}}{\sqrt{2}} = \frac{1.414\text{ cm}}{1.4142} \approx 1.00\text{ cm}$$

Step 2 — Calculate Coefficient of Variation ($\text{CV}\%$):

$$\text{CV}\% = \left(\frac{1.00\text{ cm}}{50.0\text{ cm}}\right) \times 100\% = 2.0\% \quad (\text{Passes } \le 5.0\% \text{ benchmark!})$$

Step 3 — Calculate Intraclass Correlation Coefficient ($\text{ICC}$):

$$\text{ICC} \approx 1 - \frac{\text{TE}^2}{\text{SD}_{\text{between}}^2} = 1 - \frac{1.00^2}{6.0^2} = 1 - \frac{1.00}{36.0} \approx 0.97 \quad (\text{Excellent relative reliability!})$$

Step 4 — Calculate Smallest Worthwhile Change ($\text{SWC}_{0.2}$) and Rate Sensitivity:

$$\text{SWC}_{0.2} = 0.2 \times 6.0\text{ cm} = 1.20\text{ cm}$$

Because $\text{TE} (1.00\text{ cm}) < \text{SWC} (1.20\text{ cm})$, the test exhibits Good sensitivity to detect small meaningful changes!

Step 5 — Calculate Minimal Detectable Change ($\text{MDC}_{95}$ and $\text{MDC}_{90}$):

$$\text{MDC}_{95} = 1.96 \times \sqrt{2} \times 1.00\text{ cm} \approx 2.77\text{ cm} \quad \text{and} \quad \text{MDC}_{90} = 1.645 \times \sqrt{2} \times 1.00\text{ cm} \approx 2.33\text{ cm}$$

Step 6 — Interpret Individual Athlete Pre- to Post-Mesocycle Changes:

• Athlete A improves from $48.0\text{ cm}$ to $48.8\text{ cm}$ ($\Delta = +0.80\text{ cm}$): Because $+0.80\text{ cm}$ is less than both $\text{TE}$ ($1.00\text{ cm}$) and $\text{SWC}$ ($1.20\text{ cm}$), the change is unclear / within random measurement noise.
• Athlete B improves from $48.0\text{ cm}$ to $51.0\text{ cm}$ ($\Delta = +3.00\text{ cm}$): Because $+3.00\text{ cm}$ exceeds both $\text{SWC}$ ($1.20\text{ cm}$) and $\text{MDC}_{95}$ ($2.77\text{ cm}$), the practitioner has $> 95\%$ statistical confidence that a practically meaningful real adaptation occurred.

---

Summary of Psychometric Formulas and Benchmarks