9.3 Systematically Reevaluating Assessments, Protocols, and Interventions

Systematically Reevaluating Assessments, Protocols, and Interventions

A sport science monitoring system is not a static monument; it is a closed-loop cybernetic feedback system. At the conclusion of every training block or competitive season, the sport scientist must turn the analytical lens inward to audit three distinct targets: (1) the assessment battery itself, (2) the efficacy and fidelity of the prescribed training or recovery interventions, and (3) the longitudinal continuity of database records when protocols evolve.

---

Auditing the Assessment Battery: Combatting "Protocol Creep"

Over multiple seasons, performance departments frequently suffer from "protocol creep"—adding new tests requested by incoming coaches, medical staff, or researchers without ever retiring legacy assessments. A bloated battery wastes valuable training time, induces cumulative athlete fatigue, degrades athlete buy-in ("testing fatigue"), and overwhelms coaches with redundant dashboards.

Every assessment in the battery must undergo an annual post-season audit against four quantitative pruning criteria:

1. Ceiling and Floor Effects (Range Restriction)

If an athlete population adapts or matures to the point where $> 80\%$ of the squad achieves the maximum possible score (a ceiling effect, such as senior elite athletes maxing out a basic movement screen or a submaximal shuttle stage) or clusters near the minimum (floor effect), the between-subject standard deviation ($SD_{\text{between}}$) collapses toward zero. Mathematically, range restriction destroys both the test's Intraclass Correlation Coefficient ($\text{ICC}$) and its capacity to detect positive training adaptations. The protocol must be retired or upgraded to a more demanding biomechanical progression.

2. Redundancy and Multicollinearity ($r > 0.85$ / $R^2 > 0.72$)

By computing an inter-test correlation matrix or Principal Component Analysis ($\text{PCA}$) across all battery metrics, the sport scientist identifies collinear tests measuring the exact same underlying construct. When two assessments exhibit $r > 0.85$ (sharing $> 72\%$ common variance, or showing a Variance Inflation Factor $\text{VIF} > 5\text{–}10$—for instance, $10\text{ m}$ and $20\text{ m}$ sprint splits, or unloaded $\text{CMJ}$ peak power and Squat Jump peak power), retaining both provides negligible unique information. The practitioner should prune the test that has the inferior signal-to-noise ratio ($\text{TE} > \text{SWC}$), higher setup complexity, or greater injury/fatigue cost.

3. Predictive, Discriminative, and Ecological Validity

Retrospective post-season modeling should test whether baseline or in-season changes in the assessment actually correlated with on-field Key Performance Indicators ($\text{KPIs}$), discriminated elite starters from developmental squad members, or predicted soft-tissue injury incidence (e.g., Receiver Operating Characteristic Area Under the Curve $\text{AUC}$). An injury-screening metric that repeatedly yields an $\text{AUC} \approx 0.50\text{–}0.55$ (no better than flipping a coin) across multiple seasons should not be used for return-to-play clearance or prophylactic grouping.

4. Actionability Audit (The "Decision-Impact Ratio")

For every test in the battery, ask: "How many times during the past season did a score on this assessment directly trigger an individualized change in training prescription, load management, recovery modality, or tactical selection?" Any metric with a Decision-Impact Ratio of zero is merely "museum data" and must either be connected to an explicit, pre-agreed coaching decision threshold or removed from the battery.

---

Evaluating Intervention Efficacy: Implementation Fidelity vs. Biological Responsiveness

When a 6-to-8-week training or recovery intervention fails to produce a statistically or practically meaningful shift in the squad's target KPI at retesting, the sport scientist must systematically diagnose two distinct failure pathways before discarding the underlying physiological theory.

Pathway 1: Ruling Out Type III Error (Implementation and Dose-Fidelity Failure)

In applied sport science, a Type III Error occurs when a practitioner concludes that a physiological intervention was ineffective (or asks the wrong analytical question) when, in reality, the intervention was never executed as prescribed. Before blaming the physiological stimulus design, audit Training Dose Compliance and Implementation Fidelity:

• Attendance and Volume Completion: Did the athlete actually complete $\ge 85\text{–}90\%$ of the planned sessions, sets, and repetitions, or were sessions repeatedly truncated by congested match travel and medical modifications?
• Intensity and Velocity Fidelity: In a maximal strength or Velocity-Based Training ($\text{VBT}$) block targeting $0.50\text{ m}\cdot\text{s}^{-1}$ ($\sim 85\%\ 1\text{RM}$), did the athletes actually lift at the prescribed load and maximal intended concentric velocity, or did they under-load the bar? In a High-Intensity Interval Training ($\text{HIIT}$) block targeting $\ge 12\text{ min}$ above $90\%\ \text{HR}_{\max}$, did internal heart-rate telemetry confirm that the physiological target zone was actually achieved?

Pathway 2: Individual Responder vs. Non-Responder Classification

Even when implementation fidelity is $100\%$, evaluating an intervention solely by the group mean change is deeply flawed because a near-zero group change often masks polarized subgroups of strong positive responders and negative responders. By combining the test's Typical Error ($\text{TE}$) and Smallest Worthwhile Change ($\text{SWC} = 0.2 \times SD_{\text{between}}$) into an individual responsiveness threshold—either by plotting the athlete's observed change ($\Delta = X_{\text{post}} - X_{\text{pre}}$) with its $90\%$ Confidence Interval ($\Delta \pm 1.645 \times \sqrt{2} \times \text{TE}$) against the trivial band ($\pm \text{SWC}$), or by requiring $\Delta > \sqrt{\text{TE}^2 + \text{SWC}^2}$—each athlete is classified into one of four actionable categories:

---

Version Control and Longitudinal Continuity When Updating Protocols

When an audit reveals that a legacy device or protocol must be upgraded (for example, replacing a contact jump mat with dual piezoelectric force plates, or migrating from a single-beam timing system with a $1.0\text{ m}$ setback to a dual-beam system with a $0.30\text{ m}$ setback), abruptly swapping systems destroys multi-year longitudinal Z-scores and individual historical baselines.

To preserve longitudinal continuity without corrupting the database, the sport scientist must execute a three-step protocol migration workflow:

1. Parallel Overlap / Cross-Calibration Study ($n = 20\text{–}30$ Athletes): Before retiring the legacy protocol (v1.0), test a representative sample of $20\text{–}30$ athletes simultaneously (or in counterbalanced order) on both the legacy (v1.0) and updated (v2.0) systems across the full span of team scores.
2. Bland-Altman Agreement and Deming / Passing-Bablok Regression:
   • Perform a Bland-Altman analysis to quantify the fixed systematic bias ($\bar{d} = \bar{Y}_{\text{v2.0}} - \bar{X}_{\text{v1.0}}$), check for proportional bias (whether the difference grows at higher performance values), and compute the $95\%$ Limits of Agreement ($\text{LoA} = \text{Bias} \pm 1.96 \times SD_{\text{diff}}$).
   • Fit a Deming regression or non-parametric Passing-Bablok regression ($Y_{\text{v2.0}} = \beta_0 + \beta_1 X_{\text{v1.0}}$), which properly accounts for random measurement error in both the legacy and new instruments (unlike Ordinary Least Squares regression, which assumes $X$ is error-free and attenuates the slope $\beta_1$).
   • Decision Rule: For example, if a legacy optical jump mat (v1.0) overestimates force-plate impulse-momentum jump height (v2.0) by a fixed bias of $+4.2\text{ cm}$ with a Deming conversion equation of $Y_{\text{v2.0}} = -3.8 + 0.98 X_{\text{v1.0}}$ and a $95\%\ \text{LoA}$ of $\pm 0.9\text{ cm}$ (narrower than the squad's $\text{SWC}$ of $1.2\text{ cm}$), historical v1.0 records can be mathematically harmonized into v2.0-equivalent units. Conversely, if the random scatter ($\text{LoA} = \pm 3.4\text{ cm}$) exceeds the $\text{SWC}$, mathematical conversion is invalid for individual athletes, and a hard baseline reset must be declared.
3. Athlete Management System ($\text{AMS}$) Schema Versioning: Never overwrite or silently concatenate raw metrics from different protocols into the same database column. Tag every record with an explicit protocol_version (v1.0 vs. v2.0), hardware_model, sampling_rate_hz, and scoring_method so longitudinal visualizations automatically display a protocol-transition vertical marker or filter by version.