14.2 Validity, Reliability & Assessment Bias

Introduction: The Foundations of Psychometrics in Secondary Classrooms

Every secondary educator who constructs a unit exam, writes a classroom quiz, or evaluates an essay functions as an applied psychometrician. Psychometrics—the science of measuring psychological and educational traits, knowledge, and abilities—provides the essential theoretical framework that ensures classroom assessments are fair, consistent, and accurate. In grades 7 through 12, where assessment data determines course placements, graduation competencies, academic honors, and college readiness, educators cannot afford to treat test construction as an informal or haphazard activity.

An educational assessment is fundamentally an instrument designed to gather behavioral samples from which teachers make critical inferences about student knowledge, skills, and cognitive processing. If an assessment is poorly structured, flawed in its wording, or culturally biased, the scores it produces are distorted. As a result, the teacher draws erroneous inferences about student mastery, leading to misguided instructional decisions and pedagogical harm. Preparing for the Praxis Principles of Learning and Teaching (PLT): Grades 7-12 (5624) exam requires mastering the architectural principles of test item design, deeply understanding the classical psychometric constructs of validity and reliability, and rigorously identifying and eradicating assessment bias.

Validity: The Integrity of Assessment Inferences

The single most critical psychometric property of any educational assessment is validity. In contemporary educational measurement (following the Standards for Educational and Psychological Testing), validity is defined not as a property of the test itself, but as the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of the interpretations, inferences, and actions based on test scores.

Strictly speaking, an educator should never claim that "this test is valid." Rather, they must state that "the inferences drawn from this test regarding student mastery of 10th-grade biology standards are valid." If an exam designed to measure historical reasoning is administered to English Language Learners in overly complex English, the resulting low scores do not justify the inference that the students lack historical knowledge; the inference is invalid because the test measured English reading proficiency instead.

The Three Traditional Pillars of Validity Evidence

                         ┌─────────────────────────────┐
                         │   OVERARCHING VALIDITY:     │
                         │ Appropriateness of Inferences│
                         └──────────────┬──────────────┘
                                        │
         ┌──────────────────────────────┼──────────────────────────────┐
         ▼                              ▼                              ▼
┌───────────────────┐        ┌───────────────────┐        ┌───────────────────┐
│ CONTENT VALIDITY  │        │ CONSTRUCT VALIDITY│        │CRITERION VALIDITY │
│ - Curriculum      │        │ - Underlying      │        │ - Relationship    │
│   alignment       │        │   trait/theory    │        │   with external   │
│ - Table of        │        │ - Free from       │        │   benchmarks      │
│   Specifications  │        │   construct       │        │ - Concurrent &    │
│ - Representative  │        │   irrelevant bias │        │   Predictive      │
│   sampling        │        │                   │        │                   │
└───────────────────┘        └───────────────────┘        └───────────────────┘

1. Content Validity

• Definition: The extent to which the items on an assessment representatively sample the complete domain of knowledge, skills, and cognitive processes outlined in the curriculum and learning standards.
• Classroom Verification: Established systematically through a Table of Specifications (Test Blueprint). The teacher creates a two-dimensional grid crossing content topics with cognitive levels (Bloom's Taxonomy), ensuring that test items mirror the instructional time and cognitive depth spent during the unit. If a teacher spends 80% of a chemistry unit on chemical bonding and 20% on gas laws, but the exam devotes 70% of its items to gas laws, the exam suffers from severe content underrepresentation and lacks content validity.

2. Construct Validity

• Definition: The extent to which an assessment accurately measures the underlying theoretical psychological construct or cognitive trait it claims to measure, without interference from irrelevant factors.
• Threats to Construct Validity:
  • Construct Underrepresentation: The assessment is too narrow, failing to capture critical dimensions of the target construct (e.g., assessing oral communication competence purely through a written multiple-choice test).
  • Construct-Irrelevant Variance: The assessment introduces extraneous factors that artificially inflate or deflate student scores (e.g., an 8th-grade mathematics word problem that uses convoluted, archaic literary prose; a student who fails the item may fail because of reading comprehension deficits, not mathematical inability).

3. Criterion-Related Validity

• Definition: The extent to which scores on an assessment correspond to, or predict performance on, an established external criterion or independent benchmark.
• Two Forms of Criterion Validity:
  • Concurrent Validity: The degree of correlation between test scores and an external criterion measured at approximately the same time (e.g., comparing a teacher's newly developed 15-minute diagnostic algebra screener against a validated, full-length standardized algebra achievement test administered the same week).
  • Predictive Validity: The degree to which test scores accurately forecast a student's future performance on a subsequent criterion (e.g., evaluating how accurately 10th-grade PSAT scores predict 12th-grade AP exam scores or first-year college grade point average).

Reliability: The Consistency and Stability of Measurement

While validity addresses what a test measures and the truthfulness of its inferences, reliability refers to the consistency, stability, dependability, and repeatability of assessment results. If an assessment is reliable, a student would receive approximately the same score if tested on Monday or Tuesday, if evaluated by Teacher Smith or Teacher Jones, or if taking Form A or Form B (assuming no new learning or forgetting occurred).

Major Types of Reliability in Secondary Education

1. Test-Retest Reliability: Measures the stability of scores over time. The same assessment is administered to the same group of students on two separate occasions with a reasonable time interval. A high positive correlation between the two sets of scores indicates strong temporal stability.
2. Alternate-Form (Parallel-Form) Reliability: Measures consistency across two equivalent versions of an assessment designed to the same test specifications. If Form A and Form B of a high school chemistry semester exam yield virtually identical score distributions and rankings for the same cohort, the exam demonstrates high alternate-form reliability.
3. Internal Consistency Reliability: Measures the degree to which all individual items on an assessment measure the same unified construct. It evaluates item homogeneity.
   • Split-Half Reliability: The test is divided into two equivalent halves (typically odd-numbered items vs. even-numbered items), and a student's score on one half is correlated with their score on the other half.
   • Cronbach's Alpha (Coefficient Alpha) & Kuder-Richardson 20 (KR-20): Internal-consistency coefficients computed from item and total-score variances; coefficient alpha equals the average of all possible split-half estimates, and KR-20 is the special case for items scored right or wrong.
4. Inter-Rater Reliability (Scorer Reliability): Measures the degree of consistency or consensus between two or more independent evaluators scoring the exact same student work. Critical for grading subjective constructed-response items, essays, oral presentations, and portfolios. If two English teachers score the same batch of 10th-grade analytical essays and one teacher awards mostly A's while the other awards mostly C's, the assessment suffers from catastrophic inter-rater unreliability.

Factors Influencing Reliability

• Test Length: All else being equal, longer assessments with more items are more reliable than shorter assessments because larger samples of student behavior reduce the influence of random chance or isolated errors.
• Item Clarity: Vague, ambiguous questions introduce random measurement error, depressing reliability.
• Objective Scoring Criteria: Well-defined rubrics and explicit anchor papers increase scoring consistency.
• Testing Conditions: Extreme room temperatures, loud hallway noise, or inadequate time limits introduce random environmental variance.

The Interdependent Relationship Between Validity and Reliability

Understanding the precise relationship between validity and reliability is one of the most frequently tested concepts on the Praxis PLT exam. The two psychometric properties maintain an immutable, hierarchical relationship:

Reliability is a necessary, but not sufficient, condition for validity. A measurement instrument can produce remarkably consistent, reproducible results every single day, yet be measuring the completely wrong construct. Conversely, if an instrument is unstable, erratic, and noisy (unreliable), the scores it produces are meaningless and cannot form the basis for valid inferences.

The Target / Bullseye Metaphor

• Reliable, But Not Valid: A rifle marksman fires five shots that hit a tight, compact cluster in the far upper-right corner of the target, miles away from the center. The shooting is extraordinarily consistent (reliable), but completely misses the intended bullseye (invalid).
  • Classroom Equivalent: Using a digital bathroom scale to measure 9th-grade algebra competence. The scale yields exactly 135.4 lbs every time a student steps on it (high reliability), but weight has zero validity for inferring algebraic reasoning.
• Neither Reliable Nor Valid: The marksman's shots scatter wildly across the paper, the wall, and the floor. There is no consistency and no accuracy.
  • Classroom Equivalent: An ambiguous, poorly written 3-item pop quiz administered in a chaotic classroom that yields wildly unpredictable scores unrelated to student effort or knowledge.
• Both Reliable and Valid: The marksman fires five shots directly into the center bullseye, tightly clustered together. The shooting is both consistent and on-target.
  • Classroom Equivalent: A carefully constructed, 40-item physics exam built from a Table of Specifications, featuring clear items and objective scoring rubrics, that consistently and accurately measures physics mastery.

Assessment Bias, Fairness, and Universal Test Design (UTD)

An assessment cannot produce valid inferences if it is biased. Assessment bias occurs when test items contain construct-irrelevant features, cultural assumptions, or linguistic barriers that systematically disadvantage a particular subgroup of students (based on race, ethnicity, gender, socioeconomic status, religion, or language background).

Types of Assessment Bias in Secondary Items

1. Cultural and Experiential Bias: Items assume familiarity with specific cultural norms, traditions, or activities that are not universally experienced. For example, a math word problem calculating velocity that references polo mallets, lacrosse equipment, or sailing regattas introduces cultural assumptions that unfairly disadvantage students from working-class or urban backgrounds who lack exposure to elite leisure activities.
2. Linguistic Bias: Items incorporate unnecessary idiomatic expressions, regional slang, or overly complex syntactic structures that impede English Language Learners (ELLs) or multilingual students. Unless the assessment is explicitly evaluating English reading comprehension, the linguistic complexity of an item must not prevent a student from demonstrating content knowledge in science, mathematics, or history.
3. Socioeconomic Bias: Word problems or essay prompts that require students to draw on personal experiences accessible only to wealthy families (e.g., "Write an essay describing your favorite international family vacation").
4. Stereotyping and Offensive Content: Items that portray demographic groups in narrow, derogatory, or cliché roles (e.g., presenting women only in domestic roles or minority characters only in menial labor positions).

Universal Design for Learning (UDL) in Assessment

To ensure equity, secondary educators implement Universal Test Design:

• Clear, Uncluttered Visual Layout: Ample white space, readable typography, and organized item formatting that prevents visual sensory overload.
• Plain Language Conventions: Eliminating convoluted sentence structures, archaic vocabulary, and ambiguous idioms from subject-matter tests.
• Multiple Means of Action and Expression: Providing flexible avenues for students to demonstrate standard mastery (e.g., oral presentation, written report, or digital model) without lowering cognitive expectations.
• Fair Testing Accommodations: Properly administering legal accommodations for students with IEPs or 504 plans (e.g., extended time, read-aloud support for non-reading tests, quiet testing environments, speech-to-text tools) to level the playing field without modifying the target construct.

Psychometric Properties Comparison Matrix

Secondary Disciplinary Scenarios in Action

• 10th-Grade English Language Arts: When scoring the mid-term literary analysis essay, the English department chair notices that one teacher consistently gives an average score of 94%, while another teacher grading the same prompt gives an average score of 72%. Recognizing severe inter-rater unreliability, the department convenes a 90-minute calibration session. Together, they review five anonymous "anchor papers" representing advanced, proficient, and basic performance. They debate each score against their analytic rubric until their independent evaluations achieve a 90% consensus, establishing reliable scoring for the entire sophomore class.
• 9th-Grade Physical Science: A science teacher constructs a unit test on velocity, acceleration, and Newton's laws. To ensure content validity, the teacher builds a Table of Specifications reflecting their four-week unit: 30% kinematics equations, 40% Newton's three laws, 20% friction and free-body diagrams, and 10% momentum. By distributing test points exactly across these percentages and balancing lower-order recall with higher-order mathematical problem solving, the teacher ensures the test validly samples what was taught.
• 8th-Grade Pre-Algebra: An 8th-grade math teacher reviews a draft unit exam on linear functions. One question presents a multi-paragraph word problem about calculating interest rates on yacht maintenance and country club initiation fees. Recognizing that this scenario introduces construct-irrelevant socioeconomic bias that could alienate students unfamiliar with private clubs, the teacher revises the stem to focus on a realistic context familiar to all adolescents: calculating the hourly cost and data usage of smartphone service plans.

Praxis PLT Exam Traps & Psychometric Distinctions

• Believing a Test Can Be Valid Without Being Reliable: This is a classic psychometric trap on teacher certification exams. Test takers often intuitively think, "If it measures the right thing, who cares if it fluctuates a little?" Psychometrically, if an instrument produces unstable, fluctuating scores, it is fundamentally measuring random noise rather than true competence. An erratic test cannot be valid.
• Conflating Difficulty with Assessment Bias: A test item is not biased simply because it is difficult or because many students answer it incorrectly. A rigorous physics problem that requires deep conceptual reasoning is appropriately challenging. An item is biased only when construct-irrelevant factors (such as cultural idioms, socioeconomic assumptions, or linguistic jargon) systematically impede specific demographic subgroups from demonstrating their true ability.
• Confusing Face Validity with Content Validity: Face validity is merely the superficial, casual appearance of whether a test looks like it measures what it claims to measure (e.g., glancing at a test and saying, "Looks like a math test"). Face validity is not evidence that scores support valid inferences. Content validity requires rigorous, systematic alignment between test items and curriculum standards verified through a formal Table of Specifications.
• Assuming High Reliability Guarantees Valid Inferences: Just because a standardized test has a high reliability coefficient ($r = 0.95$) does not mean it can be used for any purpose. Using a highly reliable multiple-choice grammar test to infer a student's ability to write persuasive prose in a real-world setting is an invalid use of the test.