2.1 Measurement Theory, Reliability, & Validity

2.1 Measurement Theory, Reliability, & Validity

Core Principle: Psychoeducational assessment scores never represent immutable, error-free reflections of a student's inner cognitive or academic capability. Every assessment score is an observed estimate governed by Classical Test Theory, subject to random and systematic measurement errors. Competent school psychologists must evaluate the reliability and validity of assessment tools to interpret test performance defensibly and communicate confidence bands to multidisciplinary evaluation teams.

---

1. Classical Test Theory (CTT) & Measurement Error

Classical Test Theory (CTT), often termed true-score theory, serves as the mathematical foundation for the vast majority of standardized cognitive, achievement, and behavioral rating batteries used in school psychology.

The Mathematical Formulation

CTT asserts that any score an examiner obtains during an evaluation—the Observed Score (X)—is composed of two additive, independent components:

X = T + E

Where:

• X (Observed Score): The actual number of raw points or derived standardized score earned by the examinee during a specific testing administration.
• T (True Score): The theoretical value that would be obtained if the examinee were assessed an infinite number of times with identical or perfectly parallel instruments, assuming no memory, practice, fatigue, or developmental changes occurred. It represents the student's true, unadulterated ability level on that construct.
• E (Error Score): The cumulative discrepancy between the observed score and the true score resulting from unsystematic, random measurement fluctuations.

Core Assumptions of Classical Test Theory

For CTT formulas to function mathematically, test developers and psychometricians rely on several explicit postulates:

1. Expected Value of Error is Zero: Across an infinite number of repeated administrations, random errors cancel each other out: Expected Value of Error E(E) = 0. True score is mathematically defined as the expected value (mean) of observed scores: T = E(X).
2. Zero Correlation Between True Score and Error: The magnitude of measurement error is completely independent of the examinee's true ability level (r_TE = 0). A student with an exceptionally high true cognitive ability is just as likely to experience positive or negative random error as a student with a lower ability level.
3. Zero Correlation Between Errors Across Administrations: Measurement errors occurring during one test administration have zero correlation with errors occurring during a second administration or on an alternate test (r_E1,E2 = 0).

Systematic Error vs. Random Error

Understanding the distinction between random error and systematic error is essential for legal and psychometric defensibility:

---

2. Types of Reliability Coefficients

In psychometrics, reliability refers to the consistency, dependability, and reproducibility of test scores across time, test forms, item samples, and raters. Mathematically, the reliability coefficient (r_xx) represents the proportion of observed score variance (σ²_X) that is attributable to true score variance (σ²_T):

r_xx = σ²_T / σ²_X = 1 - (σ²_E / σ²_X)

Reliability coefficients range from 0.00 (all observed variance is random error) to 1.00 (perfect measurement with zero error). In psychoeducational evaluation, different types of reliability isolate distinct sources of measurement error.

Stability Across Time: Test-Retest Reliability

• Mechanics: The identical test is administered to the same group of students on two separate occasions, and the resulting score pairs are correlated to calculate the coefficient of stability.
• Error Source Captured: Time-sampling error (fluctuations in examinee state, alertness, mood, or environment over time).
• Interval Considerations:
  • Too Short (e.g., 1–3 days): Results in artificially inflated reliability due to memory, carryover, and practice effects.
  • Too Long (e.g., 6–12 months): Conflates measurement instability with authentic developmental maturation, instruction, or intervention effects.
  • Optimal Window: A 2- to 4-week retest window is standard in commercial cognitive and academic battery standardization manuals.

Equivalence Across Forms: Alternate / Parallel Form Reliability

• Mechanics: Two distinct forms of a test (Form A and Form B) designed to parallel specifications (identical content domains, item counts, difficulty parameters, means, and variances) are administered to the same sample.
• Error Source Captured: Content-sampling error (specificity of particular items) when administered simultaneously; both content-sampling and time-sampling error when administered with a time delay.
• Clinical Application: Vital for progress monitoring tools (e.g., Curriculum-Based Measurement probes in reading and math) where students must be retested frequently without memorizing test items.

Internal Consistency Reliability

Internal consistency evaluates how well the individual items within a single test administration correlate with each other, capturing content-sampling error and item heterogeneity.

1. Split-Half Reliability & The Spearman-Brown Formula

• An examiner splits a test into two equivalent halves (commonly by assigning odd-numbered items to Half 1 and even-numbered items to Half 2 to counterbalance fatigue and progressive item difficulty).
• Correlating Half 1 with Half 2 (r_hh) yields the reliability of a test half as long. Because test length directly affects reliability, psychometricians apply the Spearman-Brown Prophecy Formula to estimate full-test reliability:

r_sb = (2 * r_hh) / (1 + r_hh)

• General Spearman-Brown Formula: For altering test length by a factor of k (where k = new length / old length):

r_kk = (k * r_xx) / [1 + (k - 1) * r_xx]

Exam Key Takeaway: Adding homogeneous items of equal quality systematically increases test reliability. Shortening a test to save administration time systematically decreases its reliability.

2. Cronbach's Alpha (Coefficient α)

• Developed by Lee Cronbach, Coefficient Alpha represents the mathematical mean of all possible split-half combinations for an instrument.
• Item Suitability: Required for tests containing multi-point, continuous, or Likert-scale items (e.g., behavioral rating scales like the BASC-3, Conners 4, BRIEF-2, or cognitive subtests scored on 0, 1, or 2-point rubrics).

α = [k / (k - 1)] * [1 - (Σ σ²_i / σ²_X)]

Where k is the number of items, σ²_i is the variance of each individual item, and σ²_X is the total test variance.

3. Kuder-Richardson Formula 20 (KR-20)

• A specialized algebraic simplification of Cronbach's alpha restricted strictly to dichotomously scored items (scored exclusively as correct [1] or incorrect [0], such as multiple-choice tests or standard cognitive item passes/fails):

KR-20 = [k / (k - 1)] * [1 - (Σ p_i * q_i / σ²_X)]

Where p_i is the proportion of examinees passing item i, and q_i = 1 - p_i is the proportion failing.

Inter-Rater / Inter-Scorer Reliability

• Error Source Captured: Scorer subjectivity, clerical grading variations, and inconsistent application of scoring rubrics.
• Metrics:
  • Cohen's Kappa (κ): Used for categorical, nominal, or diagnostic classification data between two raters. Critically, Kappa statistically corrects for agreement that would occur by chance alone:

κ = (P_o - P_e) / (1 - P_e)

Where P_o is the observed proportion of agreement, and P_e is the expected chance agreement. Widely used when validating direct classroom behavioral observation protocols (e.g., momentary time sampling of off-task behavior).

• Intraclass Correlation Coefficient (ICC): Used when multiple raters assign continuous scores or ratings (e.g., evaluating inter-rater agreement on the Vineland-3 or BASC-3 teacher forms across three classroom teachers).

Professional Standards for Reliability Thresholds

In school psychology practice, acceptable reliability thresholds vary depending on how scores are utilized:

---

3. Standard Error of Measurement (SEM) & Confidence Intervals

The Standard Error of Measurement (SEM) is the practical operationalization of Classical Test Theory. It quantifies the standard deviation of observed scores an individual would obtain if tested an infinite number of times under identical conditions.

Mathematical Formula

SEM = SD * √(1 - r_xx)

Where:

• SD = Standard deviation of the standardized assessment tool (e.g., 15 for standard scores, 3 for scaled scores, 10 for T-scores).
• r_xx = Published reliability coefficient of the scale.

Core Psychometric Law: There is an inverse mathematical relationship between test reliability and the Standard Error of Measurement. As test reliability increases toward 1.00, the SEM approaches 0. Conversely, as reliability drops, the SEM expands, increasing measurement uncertainty.

Calculating and Interpreting Confidence Intervals

Because an observed score is merely an estimate of a student's true capability, school psychologists report confidence intervals (typically 68%, 90%, or 95%) around the score.

Confidence intervals are calculated using the standard normal distribution critical values (z):

Confidence Interval = Observed Score ± (z_crit * SEM)

Clinical Case Vignette: Evaluating Confidence Bands in Intellectual Disability Eligibility

Case Scenario: Marcus, a 9-year-old student, is evaluated for special education eligibility under the Intellectual Disability (ID) category. On a standardized cognitive battery (Mean = 100, SD = 15, r_xx = 0.96), Marcus earns an observed Full Scale IQ (FSIQ) of 69.

Psychometric Calculation:

1. SEM = 15 * √(1 - 0.96) = 15 * √(0.04) = 15 * 0.20 = 3.0
2. 95% Confidence Interval = 69 ± (1.96 * 3.0) = 69 ± 5.88 = [63.12, 74.88] → [63 to 75]

Clinical Interpretation at the IEP Table: Under state criteria, an intellectual disability requires cognitive performance falling at or below approximately 2 standard deviations below the mean (SS ≤ 70), accompanied by concurrent deficits in adaptive functioning across environments. The school psychologist must explain to the team that Marcus's observed score of 69 cannot be treated as an absolute pinpoint value. His 95% confidence interval spans from 63 to 75. Because this confidence interval crosses the clinical cutoff of 70, the team must not base eligibility solely on this single score. The evaluation team must carefully weigh adaptive behavior assessment data (e.g., Vineland-3 or ABAS-3), developmental history, and classroom observational data to determine whether pervasive intellectual and adaptive limitations exist.

True Score Centering vs. Observed Score Centering

Technically, Classical Test Theory demonstrates regression to the mean: an individual who scores far from the population mean (Mean = 100) will have an estimated true score (T') that is slightly closer to the population mean than their observed score (X):

T' = Mean + r_xx * (X - Mean)

If a student scores an observed X = 70 on a test with r_xx = 0.90 (Mean = 100):

T' = 100 + 0.90 * (70 - 100) = 100 - 27 = 73

While advanced computer scoring platforms construct asymmetric confidence intervals centered around estimated true scores (T'), traditional psychoeducational reports frequently center confidence intervals around the observed score for clarity when presenting to multidisciplinary teams.

---

4. Factors That Influence Reliability Coefficients

School psychologists must critically evaluate technical manuals to understand how sample characteristics and test design artificially inflate or deflate reported reliability:

1. Test Length: As established by the Spearman-Brown formula, adding homogeneous items increases reliability. Brief or "abbreviated" test forms inevitably sacrifice reliability.
2. Sample Heterogeneity vs. Range Restriction: Reliability coefficients are correlation-based and depend directly on sample variance. When a test is normed on a wide, heterogeneous general population sample (e.g., all 8-year-olds), reliability appears high. If the test is evaluated on a homogeneous, restricted clinical group (e.g., only students diagnosed with severe reading disabilities), score variance shrinks, drastically deflating the observed reliability coefficient.
3. Test-Taker Characteristics: Student fatigue, anxiety, lack of motivation, or bilingualism introduce extraneous random variance that lowers reliability.
4. Administration Consistency: Deviations from standardized instructions, non-standard prompts, or irregular timing introduce unquantified error into observed scores.
5. The Speeded Test Trap: On a speeded test (where performance reflects rate of completion rather than item difficulty), split-half reliability (odd-even) is spuriously and artificially inflated. Students answer nearly all attempted items correctly before time expires, yielding virtually identical half-scores. For speeded measures, only test-retest or alternate-form reliability provides valid estimates.

---

5. Modern Validity Theory: From Trinitarian to Unitary Construct Validity

Historically, psychometrics viewed validity through a "trinitarian" model: content validity, criterion validity, and construct validity were treated as separate, independent types of validity.

Under modern standards established by the Standards for Educational and Psychological Testing (AERA, APA, & NCME), validity is conceptualized as a unitary construct. Validity is not a property of the test itself; rather, it refers to the degree to which accumulated empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretations and actions based on test scores.

                          CONSTRUCT VALIDITY
                 (The Overarching Evidentiary Umbrella)
                                  │
         ┌────────────────────────┼────────────────────────┐
         ▼                        ▼                        ▼
  Content-Related        Criterion-Related         Internal Structure
     Evidence                 Evidence                  Evidence
  • Domain sampling        • Concurrent evidence    • Factor analysis (EFA/CFA)
  • Expert panel review    • Predictive evidence    • Multitrait-Multimethod
  • Alignment to standards • Standard error of est. • Invariance testing

1. Content-Oriented Validity Evidence

• Definition: The extent to which the items, tasks, and questions on a test adequately sample and represent the complete universe of the domain the test is intended to measure.
• Establishment: Built into the instrument during development via detailed test blueprints, curricular alignment analyses, and expert review panels (e.g., Lawshe's Content Validity Ratio).
• Key Threats:
  • Construct Underrepresentation: The test is too narrow; it fails to capture essential, critical aspects of the domain (e.g., assessing a child's reading comprehension using only single-word vocabulary definitions, completely omitting paragraph-level inference).
  • Construct-Irrelevant Variance: The test measures extraneous, irrelevant variables that contaminate test performance (e.g., a math problem-solving test featuring complex, paragraph-long linguistic narratives that penalize students with reading disabilities or English Learners, rather than purely measuring mathematical reasoning).

2. Criterion-Related Validity Evidence

Criterion validity evaluates the extent to which test scores correlate with a meaningful external behavioral or diagnostic outcome (the criterion).

• Concurrent Validity: The new assessment and the established external criterion are administered at the same point in time.
  • Example: Correlating scores on a newly developed 15-minute dyslexia screener with scores on the comprehensive Woodcock-Johnson IV Tests of Achievement administered in the same week.
• Predictive Validity: The assessment scores predict performance on a criterion measured at a future point in time.
  • Example: Evaluating whether a kindergarten universal screening assessment (e.g., DIBELS/Acadience Phonemic Segmentation Fluency) accurately predicts 3rd-grade performance on state-mandated high-stakes reading tests.
• Standard Error of Estimate (SE_est): While SEM quantifies error around an observed score on the same test, SE_est quantifies the margin of error when using test score X to predict performance on criterion Y:

SE_est = SD_Y * √(1 - r_XY²)

3. Construct-Oriented Validity Evidence: The Core Umbrella

Construct validity examines whether an assessment truly measures the theoretical psychological trait, construct, or attribute it claims to measure. Key empirical methods include:

Convergent Validity

• Evaluates whether the test correlates strongly with existing, validated instruments that measure the same or theoretically related constructs.
• Example: The Working Memory Index of a new cognitive battery should demonstrate high positive correlations (r ≥ 0.70) with the Working Memory Index of the WISC-V.

Discriminant (Divergent) Validity

• Evaluates whether the test exhibits low or near-zero correlations with instruments measuring theoretically unrelated, distinct constructs.
• Example: A newly developed measure of childhood depression should correlate near zero (r ≤ 0.15) with a measure of phonological awareness.

Campbell & Fiske's Multitrait-Multimethod Matrix (MTMM)

• A rigorous experimental validation framework that crosses multiple traits (e.g., Inattention, Hyperactivity, Anxiety) with multiple measurement methods (e.g., Teacher Rating Scale, Parent Rating Scale, Direct Classroom Observation).
• Core Expectation: Correlations between the same trait measured across different methods (monotrait-heteromethod) must be substantially higher than correlations between different traits measured via the same method (heterotrait-monomethod). If method variance exceeds trait variance, the test suffers from severe method bias.

Factor Analytic Evidence (Internal Structure)

• Exploratory Factor Analysis (EFA) & Confirmatory Factor Analysis (CFA): Statistical procedures that evaluate the internal covariance structure of subtests.
• Example: CFA evidence in the WISC-V technical manual confirms that a 5-factor model (Verbal Comprehension, Visual Spatial, Fluid Reasoning, Working Memory, Processing Speed) provides a statistically superior fit to student response data compared to a 1-factor general intelligence (g) or 4-factor model.

---

6. Common Exam Pitfalls & Psychometric Fallacies