11.3 Assessment Design, Psychometrics & Measurement Validity

11.3 Assessment Design, Psychometrics & Measurement Validity

CPTD Exam Focus: Talent development professionals must design assessments that are not only instructionally effective but also psychometrically sound and legally defensible. Exam items regularly challenge candidates to distinguish formative from summative evaluation, contrast norm-referenced with criterion-referenced testing, calculate and interpret Item Difficulty ($p$-value) and Item Discrimination ($D$), diagnose flawed distractors, select appropriate reliability coefficients (Cronbach's alpha, Cohen's kappa), validate instruments using content, construct, and criterion-related validity, and execute the Modified Angoff standard-setting method to establish defensible cut scores.

---

1. Assessment Foundations: Formative vs. Summative Evaluation

In educational evaluation, the distinction between formative and summative assessment is immortalized by evaluation pioneer Robert Stake: "When the cook tastes the soup, that's formative; when the guests taste the soup, that's summative."

---

2. Norm-Referenced vs. Criterion-Referenced Measurement

A fundamental psychometric decision in assessment design is selecting the reference framework against which examinee performance will be judged:

 Measurement Paradigms in Talent Development
 ┌────────────────────────────────────────────────────────────────────────┐
 │ NORM-REFERENCED TESTING (NRT)                                          │
 │   • Purpose: Compare examinees against each other (Rank-ordering).     │
 │   • Standard: The bell curve / peer group mean and standard deviation. │
 │   • Corporate Use: Talent selection, graduate sorting, stack ranking.  │
 │   • Flaw in L&D: If all employees excel, some are still forced to fail.│
 ├────────────────────────────────────────────────────────────────────────┤
 │ CRITERION-REFERENCED TESTING (CRT)                                     │
 │   • Purpose: Compare examinee performance against fixed criteria.      │
 │   • Standard: Pre-established behavioral learning objectives / rubrics.│
 │   • Corporate Use: Technical certifications, compliance, onboarding.   │
 │   • Merit in L&D: All learners can pass if they demonstrate mastery.   │
 └────────────────────────────────────────────────────────────────────────┘

In talent development, Criterion-Referenced Testing is the standard operational paradigm. In workplace learning, the goal is not to force employees onto a Gaussian distribution where 20% fail regardless of ability. The objective is to bring 100% of employees to defined operational competence. Criterion-referenced assessments measure what an employee can do relative to an objective job performance standard, not how they perform compared to their peers.

---

3. Classical Test Theory & Item Analysis Metrics

To ensure an assessment is reliable, fair, and valid, talent development practitioners apply Classical Test Theory (CTT) to perform item analysis on test questions. Item analysis examines three critical parameters: Item Difficulty, Item Discrimination, and Distractor Effectiveness.

1. Item Difficulty Index ($p$-value)

The Item Difficulty index, denoted as $p$, measures the proportion of examinees who answered an item correctly: $p = \frac{R}{N}$ Where $R$ is the number of examinees who answered the item correctly, and $N$ is the total number of examinees who attempted the item.

• Counterintuitive Property: The higher the $p$-value, the easier the item. An item with $p = 0.95$ was answered correctly by 95% of test-takers (very easy), whereas an item with $p = 0.20$ was answered correctly by only 20% (very difficult).
• Optimal Ranges: For a four-option multiple-choice item, an optimal $p$-value generally ranges between $0.60$ and $0.80$ to maximize measurement variance. In criterion-referenced mastery tests, $p$-values often cluster higher ($0.70$ to $0.85$), reflecting successful instructional transfer.
• Extreme Values: Items with $p < 0.30$ (excessively difficult) or $p > 0.95$ (too easy) should be reviewed for ambiguity, trick wording, or triviality, unless intentionally designed as mastery gates.

2. Item Discrimination Index ($D$ and Point-Biserial Correlation)

The Item Discrimination index ($D$) measures how effectively a single test item differentiates between high-performing examinees (who have mastered the overall domain) and low-performing examinees (who lack mastery).

To compute $D$ using the extreme-groups method (typically the top 27% and bottom 27% of the scoring distribution): $D = p_{\text{upper}} - p_{\text{lower}} = \frac{R_{\text{upper}}}{N_{\text{upper}}} - \frac{R_{\text{lower}}}{N_{\text{lower}}}$ Where $p_{\text{upper}}$ is the difficulty index for the top-scoring group, and $p_{\text{lower}}$ is the difficulty index for the bottom-scoring group.

• Point-Biserial Correlation ($r_{pbis}$): In computerized testing, discrimination is often evaluated using the point-biserial correlation coefficient between examinees' scores on the specific item (0 or 1) and their total scores on the entire test. An $r_{pbis} \ge 0.25$ is generally considered acceptable, with values above $0.35$ indicating strong discrimination.

3. Distractor Analysis & Plausibility

In multiple-choice items, incorrect alternatives are called distractors. Effective distractors must appear plausible to examinees who have not mastered the material, while being clearly rejected by competent examinees.

• Non-Functioning Distractor: A distractor that is selected by fewer than 5% of examinees. These options fail to attract uninformed test-takers and effectively reduce a 4-option item to a 3-option or 2-option item, artificially inflating the probability of guessing.
• Misleading Distractor: A distractor that attracts a disproportionate percentage of high-scoring examinees. This indicates that the distractor contains subtle technical nuances or double meanings that confuse advanced thinkers.

---

4. Measurement Reliability: Consistency and Precision

Reliability refers to the degree to which an assessment tool produces consistent, stable, and reproducible scores across repeated administrations, different test forms, and varied raters. Reliability is an indispensable prerequisite for validity: an assessment cannot be valid unless it is first reliable.

The Four Core Forms of Reliability

 Psychometric Reliability Architecture
 ┌────────────────────────────────────────────────────────────────────────┐
 │ 1. Test-Retest Reliability      ─── Stability over time                │
 │ 2. Alternate-Form Reliability   ─── Equivalence across test versions   │
 │ 3. Internal Consistency         ─── Homogeneity of test items          │
 │    • Cronbach's Alpha (α)       ─── Scale / polytomous items           │
 │    • Kuder-Richardson 20 (KR-20)── Dichotomous (right/wrong) items     │
 │ 4. Inter-Rater Reliability      ─── Consistency across evaluators      │
 │    • Cohen's Kappa (κ)          ─── Corrects for chance agreement      │
 └────────────────────────────────────────────────────────────────────────┘

1. Test-Retest Reliability: Measures stability over time. The same assessment is administered to the same group of learners at two distinct time points. The Pearson correlation ($r$) between the two score distributions reflects stability. Threats: Memory carryover effects, practice effects, and developmental maturation during the interval.
2. Alternate-Form (Parallel-Form) Reliability: Measures equivalence across two distinct versions of an assessment built from the same test blueprint. Learners take Form A and Form B; a high correlation demonstrates that test versions are interchangeable.
3. Internal Consistency Reliability: Measures the degree to which individual items on an assessment measure the same unified psychological construct or competency:
   • Split-Half Reliability: The test is split into two halves (e.g., odd-numbered items vs. even-numbered items), and scores on each half are correlated. The Spearman-Brown prophecy formula is applied to correct for test length reduction.
   • Kuder-Richardson Formula 20 (KR-20): Used for assessments scored dichotomously (correct = 1, incorrect = 0).
   • Cronbach's Alpha ($\alpha$): The gold standard metric for internal consistency, accommodating both dichotomous and polytomous (Likert-scale) items. A value of $\alpha \ge 0.70$ is acceptable for exploratory research; $\alpha \ge 0.80$ is required for corporate workplace assessments; and $\alpha \ge 0.90$ is expected for high-stakes professional credentialing exams.
4. Inter-Rater Reliability: Measures consistency across multiple evaluators scoring subjective assessments (e.g., observational checklists, behavioral interviews, oral presentations):
   • Percentage Agreement: The percentage of times two raters assign the exact same score. Flaw: Inflated by chance agreement.
   • Cohen's Kappa ($\kappa$): A robust metric that calculates inter-rater agreement while mathematically removing agreement expected by chance alone. A kappa value of $\kappa \ge 0.70$ represents substantial agreement.

---

5. Measurement Validity: Truth in Inference

While reliability addresses consistency, validity addresses truth: does the assessment measure what it purports to measure? Validity is not a property of the test itself, but rather the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and decisions made based on test scores.

The Hierarchy of Validity Types

---

6. Standard-Setting & Defensible Cut-Score Determination

A critical legal and ethical vulnerability in corporate talent development is the arbitrary establishment of passing scores. When organizations declare, "Passing is 80% because 80% sounds like a solid B," they violate the Standards for Educational and Psychological Testing and create immense legal liability under the Equal Employment Opportunity Commission (EEOC) Uniform Guidelines on Employee Selection Procedures.

A cut score (passing threshold) must be empirically determined through formalized Standard-Setting Methodologies rooted in subject matter expert judgment.

The Concept of the Minimally Competent Candidate (MCC)

Every standard-setting method begins by establishing an operational profile of the Minimally Competent Candidate (MCC) (often termed the "borderline performer"). The MCC is an examinee who possesses just enough knowledge, skill, and judgment to perform the job safely and effectively, without possessing mastery or exemplary capability.

1. The Modified Angoff Method (The Industry Standard)

The Modified Angoff method is the most widely accepted and legally defensible standard-setting procedure for professional credentialing and certification exams:

A naming caution worth carrying into the exam: credentialing bodies label this family of procedures inconsistently. ATD CI describes the CPTD standard-setting process simply as the Angoff method, in which a panel of current CPTDs estimates, for each reviewed question, the percentage of qualified candidates expected to answer it correctly. Read the described procedure rather than the label.

 The Modified Angoff Standard-Setting Workflow
 ┌────────────────────────────────────────────────────────────────────────┐
 │ 1. Assemble Panel: Convene 8–15 qualified, diverse SMEs.               │
 │ 2. Define MCC: Facilitate consensus on the profile of an MCC.          │
 │ 3. Round 1 Ratings: SMEs independently review each item and estimate   │
 │    the probability (0.00–1.00) that an MCC would answer correctly.     │
 │ 4. Group Discussion: Review rating disparities and examine empirical   │
 │    item difficulty (p-values) from pilot test administrations.         │
 │ 5. Round 2 Ratings: SMEs re-rate items independently with new insights.│
 │ 6. Final Calculation: Sum the mean item probabilities across all items │
 │    to establish the mathematically defensible cut score.               │
 └────────────────────────────────────────────────────────────────────────┘

Angoff Calculation Example: On a 100-item exam, if the average SME probability across all 100 items sums to 72.4, the recommended cut score is 72 (or 72.4%).

2. The Bookmark Method

The Bookmark method is an Item Response Theory (IRT)-based procedure used for both multiple-choice and constructed-response tests. Test items are arranged in an ordered item booklet from easiest to most difficult based on empirical item parameter data. SMEs review the booklet sequentially and place a physical or digital "bookmark" at the boundary where a minimally competent candidate transitions from having a high probability of success to failing to master the item.

3. The Nedelsky Method

The Nedelsky method is designed specifically for multiple-choice items. For each item, SMEs determine which distractors a minimally competent candidate would be able to eliminate as obviously incorrect. If an item has four options and an MCC can eliminate two, they are left with two plausible options, yielding a guessing probability of 0.50. The reciprocal probabilities are summed across all items to derive the passing score.