16.1 Psychometric Theory: Classical Test Theory, Reliability Types, and Validity Constructs

Psychometric Theory: Classical Test Theory, Reliability Types, and Validity Constructs

Psychometrics is the scientific discipline concerned with the theory and techniques of psychological measurement. Because psychological attributes—such as intelligence, extraversion, neuroticism, or perceptual speed—cannot be directly observed or physicalized like mass or length, psychometricians construct formal mathematical models to evaluate whether assessment instruments measure target constructs reliably, validly, and with minimal measurement error.

1. Classical Test Theory (CTT) and the True Score Model

Classical Test Theory (CTT), often termed the true score model, was formalized by Charles Spearman and later refined by Harold Gulliksen and Frederic Lord. CTT posits that every observed test score ($X$) is an additive composite of two independent, unobservable quantities: an individual's latent true score ($T$) and unsystematic random measurement error ($E$).

$X = T + E$

Theoretical Assumptions of CTT

Classical Test Theory rests upon several foundational axiomatic assumptions regarding random error:

1. Zero Expected Error: Across an infinite number of repeated administrations of parallel tests, random errors cancel each other out. The expected value (mean) of error is zero:

$E(E) = 0$

1. Uncorrelated True Scores and Errors: An examinee's true score bears no relationship to the magnitude or sign of the error affecting their score. High-ability and low-ability individuals are equally subject to positive or negative random error:

$r_{TE} = 0$

1. Uncorrelated Errors Across Administrations: The measurement error affecting a person on one test form ($E_1$) is completely uncorrelated with the measurement error affecting them on a second test form ($E_2$):

$r_{E_1 E_2} = 0$

1. Definition of the True Score: A person's true score is formally defined as the expected value (theoretical mean) of observed scores obtained if the individual were tested an infinite number of times under identical, independent conditions:

$T = E(X)$

Variance Partitioning and the Definition of Reliability

Because true scores and random errors are assumed to be uncorrelated ($r_{TE} = 0$), the total observed score variance ($\sigma_X^2$) across a population of examinees partitions cleanly into the sum of true score variance ($\sigma_T^2$) and error variance ($\sigma_E^2$):

$\sigma_X^2 = \sigma_T^2 + \sigma_E^2$

From this variance decomposition, the reliability coefficient ($r_{xx'}$) is mathematically defined as the proportion of observed score variance that is attributable to genuine true score variance:

$r_{xx'} = \frac{\sigma_T^2}{\sigma_X^2} = \frac{\sigma_X^2 - \sigma_E^2}{\sigma_X^2} = 1 - \frac{\sigma_E^2}{\sigma_X^2}$

• When a test has zero measurement error ($\sigma_E^2 = 0$), reliability reaches unity ($r_{xx'} = 1.0$), meaning all observed variance reflects true differences among individuals.
• When all observed differences reflect pure random noise ($\sigma_T^2 = 0$), reliability drops to zero ($r_{xx'} = 0.0$).

2. The Standard Error of Measurement (SEM)

While the reliability coefficient ($r_{xx'}$) provides a single population-level metric of test consistency, clinicians and educational diagnosticians need to know how much uncertainty surrounds an individual examinee's observed score. This uncertainty is quantified by the Standard Error of Measurement ($SEM$).

Mathematical Formulation

The $SEM$ represents the standard deviation of observed scores an individual would obtain if tested repeatedly across an infinite number of parallel testing sessions. It is derived directly from the sample standard deviation ($s_x$) and the reliability coefficient ($r_{xx'}$):

$SEM = s_x \sqrt{1 - r_{xx'}}$

Relationship Between Reliability (r_xx') and SEM (Assuming sx = 15):

Reliability (r_xx')   √(1 - r_xx')   SEM = 15 · √(1 - r_xx')
────────────────────────────────────────────────────────────
1.00                  0.000          0.00  (Zero measurement error)
0.96                  0.200          3.00  (Typical of IQ composite scores)
0.91                  0.300          4.50
0.84                  0.400          6.00
0.75                  0.500          7.50
0.00                  1.000         15.00  (Pure error; SEM equals test SD)

Constructing Confidence Intervals Around Observed Scores

Because CTT assumes measurement errors are normally distributed around the true score with a mean of zero and a standard deviation equal to the $SEM$, diagnosticians construct confidence intervals (CIs) to express the margin of measurement precision around an observed score ($X$):

$\text{Confidence Interval} = X \pm (z_{\text{critical}} \times SEM)$

• $68\%$ Confidence Interval ($z = \pm 1.00$): $X \pm 1.00(SEM)$
• $95\%$ Confidence Interval ($z = \pm 1.96$): $X \pm 1.96(SEM)$
• $99\%$ Confidence Interval ($z = \pm 2.58$): $X \pm 2.58(SEM)$

3. Typology of Reliability Estimates

Reliability is not an inherent, immutable property of a test; rather, it is a property of test scores obtained from a specific sample under specific administration conditions. Different reliability methodologies isolate different sources of unwanted error variance.

The Spearman-Brown Prophecy Formula

Because reliability is directly proportional to test length—longer tests sample a broader domain and allow random errors to cancel out—splitting a test into halves to calculate internal consistency artificially depresses the reliability coefficient. In 1910, Charles Spearman and William Brown independently derived the Spearman-Brown Prophecy formula to estimate the reliability of a test if its length is altered by a factor of $k$:

$r_{SB} = \frac{k \cdot r_{xy}}{1 + (k - 1)r_{xy}}$

$\text{Where } k = \frac{\text{Number of items in the new test}}{\text{Number of items in the original test}}$

• Correcting Split-Half Reliability: When estimating the reliability of the full test from two half-tests, the test is doubled in length, so $k = 2$. The formula simplifies to:

$r_{\text{full}} = \frac{2 r_{1/2}}{1 + r_{1/2}}$

• If the correlation between the odd and even halves of a test is $r_{1/2} = .70$, the estimated reliability of the entire test is:

$r_{\text{full}} = \frac{2(.70)}{1 + .70} = \frac{1.40}{1.70} \approx .824$

Cronbach's Alpha ($\alpha$) and Kuder-Richardson Formula 20 (KR-20)

Lee Cronbach generalized split-half reliability in 1951 by demonstrating that Cronbach's Alpha ($\alpha$) equals the mathematical mean of all possible split-half reliability coefficients for a test. Its computational formula is:

$\alpha = \frac{k}{k - 1} \left(1 - \frac{\sum_{i=1}^k s_i^2}{s_X^2}\right)$

Where $k$ is the total number of items, $s_i^2$ is the variance of item $i$, and $s_X^2$ is the total observed test score variance.

When items are scored dichotomously (coded $1$ for correct and $0$ for incorrect), item variance simplifies to $p_i q_i$ (where $p_i$ is the proportion answering correctly and $q_i = 1 - p_i$). Substituting this into Cronbach's formula yields Kuder-Richardson Formula 20 (KR-20):

$KR\text{-}20 = \frac{k}{k - 1} \left(1 - \frac{\sum_{i=1}^k p_i q_i}{s_X^2}\right)$

Inter-Rater Reliability: Cohen's Kappa ($\kappa$)

When two independent clinical raters categorize psychiatric patients into nominal diagnostic categories (e.g., Schizophrenia, Bipolar I, Major Depressive Disorder), calculating simple percentage agreement is misleading because raters will agree on a substantial percentage of cases purely by chance. Jacob Cohen developed Cohen's Kappa ($\kappa$) to correct for chance agreement:

$\kappa = \frac{P_o - P_e}{1 - P_e}$

Where $P_o$ is the observed proportion of agreement, and $P_e$ is the expected proportion of agreement predicted by marginal probabilities under chance alone. A kappa of $\kappa = 1.0$ reflects perfect agreement; $\kappa = 0$ indicates agreement no better than chance; and $\kappa < 0$ reflects agreement worse than chance.

4. Validity Constructs: Content, Criterion-Related, and Construct Validity

While reliability establishes that a test measures something consistently, validity evaluates whether a test measures the specific construct it was designed to assess, and whether clinical or educational inferences drawn from test scores are scientifically justifiable. In classic psychometrics, validity is organized around the "trinitarian" model: content validity, criterion-related validity, and construct validity.

Content Validity

Content validity reflects the degree to which items on a test adequately and representatively sample the entire domain of knowledge, skills, or behaviors the test is intended to measure.

• Evaluation Method: Content validity cannot be summarized by a single correlation coefficient; it is established qualitatively during test construction by assembling panels of Subject Matter Experts (SMEs) who review test specifications, item blueprints, and alignment matrices.
• Lawshe's Content Validity Ratio (CVR): SMEs rate whether each item is "essential," "useful but not essential," or "not necessary." Items rated essential by a statistically significant majority are retained.
• Face Validity vs. Content Validity: Face validity is not a technical psychometric validity construct. It refers merely to whether a test looks valid to the untrained examinee on superficial inspection. A test can possess high scientific content validity while lacking face validity (e.g., subtle projective stimuli or empirically keyed personality items).

Criterion-Related Validity

Criterion-related validity evaluates how well scores on a predictor test ($X$) correlate with an independent, external outcome measure or standard ($Y$), termed the criterion.

1. Concurrent Validity: The predictor test and criterion are measured at approximately the same point in time. Example: Administering a newly designed 15-minute depression screening scale and correlating its scores with a concurrent 3-hour structured diagnostic clinical interview conducted that same afternoon.
2. Predictive Validity: The predictor test is administered first, and the criterion outcome is measured at a future point in time. Example: Administering the SAT or GRE to predict cumulative undergraduate or graduate GPA four years later.

Standard Error of Estimate ($SE_{\text{est}}$)

In criterion-related validity, the accuracy of predicting a criterion score ($Y$) from an observed test score ($X$) is quantified by the Standard Error of Estimate ($SE_{\text{est}}$):

$SE_{\text{est}} = s_y \sqrt{1 - r_{xy}^2}$

Where $s_y$ is the standard deviation of the criterion and $r_{xy}$ is the criterion validity coefficient. Note the structural distinction: the $SEM$ uses test reliability ($s_x \sqrt{1 - r_{xx'}}$) to estimate true score error, whereas the $SE_{\text{est}}$ uses criterion validity ($s_y \sqrt{1 - r_{xy}^2}$) to estimate prediction error in the criterion.

The Correction for Attenuation

In the real world, both the predictor test ($X$) and the criterion measure ($Y$) contain unsystematic measurement error. Because random error attenuates (dampens) linear relationships, the observed correlation between two imperfect tests ($r_{xy}$) is systematically lower than the true theoretical correlation between the underlying constructs ($r_{T_X T_Y}$).

Charles Spearman derived the correction for attenuation formula to estimate what the correlation between two variables would be if both were measured with perfect reliability ($r_{xx'} = 1.0$ and $r_{yy'} = 1.0$):

$r_{T_X T_Y} = \frac{r_{xy}}{\sqrt{r_{xx'} \cdot r_{yy'}}}$

• Upper Bound of Validity: A vital corollary tested frequently on the GRE is that a test's criterion validity cannot exceed the square root of its reliability when the criterion is perfectly reliable ($r_{yy'} = 1.0$):

$r_{xy(\max)} = \sqrt{r_{xx'}}$

If a test has a reliability of $r_{xx'} = .64$, its maximum possible theoretical correlation with any external criterion is $\sqrt{.64} = .80$.

5. Construct Validity and the Multitrait-Multimethod (MTMM) Matrix

Construct validity is the overarching psychometric framework assessing whether an assessment instrument truly measures a theoretical psychological construct (e.g., anxiety, intelligence, working memory). It subsumes all other forms of validity evidence and is demonstrated through an ongoing process of hypothesis testing within a theoretical nomological network.

Convergent vs. Discriminant Validity

• Convergent Validity: Demonstrates that the test correlates strongly with alternative measures designed to assess the same or theoretically similar constructs. (e.g., a new trait anxiety scale correlates $r = .82$ with the State-Trait Anxiety Inventory).
• Discriminant (Divergent) Validity: Demonstrates that the test has low, near-zero, or negligible correlations with measures designed to assess theoretically distinct constructs. (e.g., a new trait anxiety scale correlates $r = .12$ with a measure of mechanical reasoning ability).

Campbell & Fiske's Multitrait-Multimethod (MTMM) Matrix

In 1959, Donald T. Campbell and Donald W. Fiske revolutionized construct validation by introducing the Multitrait-Multimethod (MTMM) Matrix. They argued that every psychological measurement reflects two distinct sources of variance: the trait of interest and the method used to measure it (e.g., self-report inventory, peer rating, projective test, behavioral observation). To validate a construct, one must simultaneously measure at least two distinct traits using at least two distinct measurement methods.

          Hypothetical MTMM Matrix (2 Traits: Trait A & Trait B; 2 Methods: M1 & M2)

                       Method 1 (Self-Report)        Method 2 (Peer Rating)
                       Trait A1      Trait B1        Trait A2      Trait B2
 ───────────────────────────────────────────────────────────────────────────
 Method 1  Trait A1    ( .88 )       
           Trait B1      .32         ( .85 )

 Method 2  Trait A2    [ .68 ]         .25           ( .84 )
           Trait B2      .22         [ .64 ]           .30         ( .82 )
 ───────────────────────────────────────────────────────────────────────────
 Key to MTMM Coefficients:
 ( .88 ) = Monotrait-Monomethod (Reliability Diagonal)
 [ .68 ] = Monotrait-Heteromethod (Convergent Validity Diagonal)
   .32   = Heterotrait-Monomethod (Method Variance / Common-Method Bias)
   .22   = Heterotrait-Heteromethod (Discriminant Validity / Construct Uniqueness)

Decoding the Four MTMM Correlations

1. Monotrait-Monomethod: Same trait, same method. Located on the main diagonal. These are reliability coefficients (e.g., $(.88)$ and $(.85)$). They should represent the highest values in the entire matrix.
2. Monotrait-Heteromethod: Same trait, different methods. These are convergent validity coefficients (e.g., $[.68]$ and $[.64]$). They reflect whether Trait A measured by self-report correlates strongly with Trait A measured by peer rating. They must be statistically significant and substantially greater than zero.
3. Heterotrait-Monomethod: Different traits, same method (e.g., $.32$ and $.30$). These correlations reflect common-method bias (method variance). If these correlations are high, it indicates that shared measurement format (e.g., self-report response sets) artificially inflates relationships between unrelated constructs.
4. Heterotrait-Heteromethod: Different traits, different methods (e.g., $.22$ and $.25$). These represent pure construct uniqueness and provide evidence for discriminant validity. They should be the lowest coefficients in the matrix.

6. Classical Item Analysis vs. Item Response Theory (IRT)

Assessment quality ultimately depends on the psychometric performance of individual test items. Psychometricians analyze items using either Classical Test Theory or Item Response Theory.

Classical Item Analysis

Classical item analysis evaluates two primary parameters: item difficulty and item discrimination.

1. Item Difficulty ($p$-Value)

In classical testing, item difficulty is indexed by the proportion of examinees who answer the item correctly:

$p = \frac{N_{\text{correct}}}{N_{\text{total}}}$

• Counterintuitive Direction: A higher $p$-value indicates an easier item (e.g., $p = .90$ means $90\%$ got it right), whereas a lower $p$-value denotes a harder item (e.g., $p = .15$).
• Optimal Difficulty: To maximize item variance and test score dispersion, the optimal average item difficulty is approximately $p \approx .50$.
• Correction for Guessing: On multiple-choice tests, optimal difficulty is adjusted halfway between the guessing probability ($g$) and $1.00$:

$\text{Optimal } p = \frac{1.00 + g}{2}$

For a 4-option multiple-choice item ($g = .25$), the optimal difficulty is $\frac{1.00 + .25}{2} = .625$.

2. Item Discrimination Index ($D$)

Item discrimination measures how effectively an item differentiates between high-scoring and low-scoring examinees. The most common classical metric is the extreme groups index ($D$):

$D = P_U - P_L$

Where $P_U$ is the proportion of examinees in the upper scoring bracket (top $27\%$) answering correctly, and $P_L$ is the proportion in the lower scoring bracket (bottom $27\%$) answering correctly.

• $D$ ranges from $-1.00$ to $+1.00$.
• $D \ge .40$: Excellent discrimination; item cleanly distinguishes high performers.
• $D \approx 0.00$: Zero discrimination; item fails to differentiate high and low performers.
• $D < 0.00$: Negative discrimination; more low performers answered correctly than high performers. Such items are psychometrically defective, indicating confusing distractors, typographical errors, or incorrect answer keys, and must be eliminated.

Item Response Theory (IRT) and Item Characteristic Curves

A major limitation of Classical Test Theory is that item parameters ($p$ and $D$) are sample-dependent (an item appears hard in a low-ability sample and easy in a high-ability sample), and examinee ability scores are test-dependent (scores depend on the specific difficulty of the test administered).

Item Response Theory (IRT) overcomes these limitations by modeling the nonlinear mathematical relationship between an individual's unobservable latent trait level (denoted by Greek letter theta, $\theta$, where $\mu = 0$ and $\sigma = 1$) and the probability of correctly answering a specific item ($P(\theta)$). This relationship is depicted graphically as an S-shaped Item Characteristic Curve (ICC) or trace line.

                   Item Characteristic Curve (3PL Model)

    Probability P(θ)
         1.0 │                                             ● Upper Asymptote
             │                                        ..-''
         0.8 │                                    _.-'
             │                                 .-'
         0.6 │                              .-'  <-- Steepest Slope = Discrimination (a)
             │                           .-'
         0.4 │                        .-'  <-- Inflection Point θ = Difficulty (b)
             │                   _..-'
   c --> 0.2 │...........-------'
  (Guessing) │
         0.0 └─────────────────────────────────────────────────────────────
            -3          -2          -1           0          +1          +2          +3
                                    Latent Trait Level (θ)

The Three Logistic IRT Models

1. One-Parameter Logistic (1PL / Rasch Model):
   • Assumes all items have equal discrimination ($a$) and zero guessing ($c = 0$).
   • Models only item difficulty ($b$), defined as the latent trait level $\theta$ at which an examinee has exactly a $50\%$ probability of answering correctly ($P(\theta) = .50$).
2. Two-Parameter Logistic (2PL Model):
   • Models both item difficulty ($b$) and item discrimination ($a$).
   • The $a$-parameter represents the slope of the ICC at the inflection point ($b$). Steeper slopes reflect greater discrimination among examinees near the threshold $\theta$.
3. Three-Parameter Logistic (3PL Model):
   • Incorporates difficulty ($b$), discrimination ($a$), and a pseudo-guessing parameter ($c$).
   • The $c$-parameter represents the lower horizontal asymptote of the ICC—the probability that an examinee with infinitely low ability ($\theta \rightarrow -\infty$) will endorse or guess the item correctly purely by chance.

Sample Invariance Property

The defining triumph of IRT over CTT is parameter invariance: item parameters ($a, b, c$) are invariant across different examinee populations, and examinee trait estimates ($\theta$) are invariant across different item subsets. This mathematical foundation enables Computerized Adaptive Testing (CAT), where items are dynamically tailored in real time to the examinee's emerging $\theta$ estimate.