5.1 Psychometrics: Reliability, Validity & Score Interpretation
Key Takeaways
Reliability (test-retest, internal consistency, inter-rater, parallel forms) measures consistency; validity concerns the evidence supporting score interpretations for an intended use — reliability is necessary but not sufficient for validity.
SEM = SD x sqrt(1 - r); use it to build a confidence band around an observed score instead of treating a single point as exact.
Standard scores translate raw performance onto a common scale: z-scores (mean 0, SD 1), T-scores (mean 50, SD 10), and IQ scores (mean 100, SD 15) can be interpreted using the 68-95-99.7 rule only if the distribution is normal.
Norm-referenced scores are only meaningful relative to the specific comparison group used to standardize the test; criterion-referenced scores compare against a fixed mastery standard instead.
Face validity is a judgment about apparent relevance, not technical validity evidence .
Why Psychometrics Is Tested So Heavily
Intake, Assessment, and Diagnosis is Domain 2 of the NBCC Content Outline — 12% of the exam, or 19 of the 160 scored items. Within that domain, item M, "select, use, and interpret appropriate assessment instruments," cannot be answered correctly without fluency in psychometrics: the science of how test scores are constructed, evaluated, and interpreted. Assessment and Testing is also one of the eight CACREP Common Core Areas the NCE is aligned to, so psychometric vocabulary resurfaces across the exam wherever a question references a test score, a screening result, or an evaluation report.
On the job, this knowledge is not abstract. A counselor who receives a client's WAIS-5 composite score, a school's WISC-V report, or a PHQ-9 total must be able to say what that number actually means — how much measurement error surrounds it, whether the instrument was appropriate for this client, and whether the result supports a clinical decision. The NCE tests this directly with items that present a coefficient, a percentile, or a scenario and ask you to identify the underlying psychometric concept.
Reliability: Is the Measurement Consistent?
Reliability refers to the consistency of a test's results — whether it produces the same or similar scores across repeated administrations, raters, or item sets, assuming the underlying trait has not changed. Classical reliability is conceptually between zero and one, though sample estimates may fall outside expected bounds. The precision needed depends on the intended use, especially whether results inform high-stakes individual decisions.
| Type of Reliability | What It Measures | Example |
|---|---|---|
| Test-retest | Stability of scores over time | Same anxiety inventory given twice, two weeks apart |
| Internal consistency (Cronbach's alpha, split-half) | How consistently items contribute to scores; consistency does not establish one underlying construct | All 9 PHQ-9 items correlate with each other |
| Inter-rater | Agreement between two or more raters scoring the same behavior | Two clinicians independently score the same intake video |
| Parallel/alternate forms | Equivalence between two versions of the same test | Form A and Form B of a career-interest inventory yield similar scores |
Standard Error of Measurement (SEM)
Because no test is perfectly reliable, every observed score contains some measurement error. The standard error of measurement (SEM) estimates how much an observed score is likely to differ from a client's hypothetical "true score," and it is calculated as:
SEM = SD × √(1 − r)
where SD is the test's standard deviation and r is its reliability coefficient. A smaller SEM means a more precise instrument.
Worked example: A test has SD = 15 and reliability r = .91. SEM = 15 × √(1 − .91) = 15 × √.09 = 15 × .30 = 4.5
For an observed score of 110, a simple band of one SEM on either side is 105.5–114.5. If normal error with constant SEM is reasonable, about 95% coverage uses ±1.96 SEM, or 101.2–118.8 (rounded). Use the instrument’s manual for individual interpretation and for judging changes between two scores.
Validity: Is the Score Interpretation Supported?
Validity concerns whether evidence and theory support a proposed interpretation of scores for a specific use. Reliability is necessary but not sufficient for validity — a bathroom scale that is consistently 10 pounds off every time is highly reliable but not valid.
| Type of Validity | What It Establishes | Example |
|---|---|---|
| Content validity | Items adequately sample the full domain being measured | A depression scale covers mood, sleep, appetite, energy, and concentration — not just sad mood |
| Criterion validity — concurrent | Test scores correlate with an established measure given at the same time | A brief screener is compared with diagnoses established independently in a clinical interview at the same visit |
| Criterion validity — predictive | Test scores predict a future outcome | A college-admissions test predicts first-year GPA |
| Construct validity | Evidence for a construct interpretation includes expected relationships with related measures and distinguishability from different constructs | An anxiety measure correlates with other anxiety measures but not with a measure of extraversion |
| Face validity | The test merely appears relevant to test-takers — does not establish validity | A stress questionnaire "looks like" it measures stress |
Score Interpretation: Norms, Standard Scores, and the Normal Curve
Two broad interpretive frameworks appear on the exam:
- Norm-referenced scores compare a client's performance to a reference (comparison) group — expressed as a percentile rank, standard score, or stanine.
- Criterion-referenced scores compare performance to a fixed standard or mastery cutoff, independent of how any comparison group performed (e.g., a licensing exam's pass/fail cut score).
When the reference distribution is approximately normal (bell-shaped), the 68-95-99.7 rule applies: about 68% of scores fall within 1 standard deviation (SD) of the mean, about 95% fall within 2 SDs, and about 99.7% fall within 3 SDs.
| Standard Score | Mean | SD | 1 SD Above Mean |
|---|---|---|---|
| z-score | 0 | 1 | z = +1.0 |
| T-score | 50 | 10 | T = 60 |
| IQ score (e.g., WAIS-5) | 100 | 15 | 115 |
Worked example: A client earns a T-score of 30 on a normally distributed scale. Converting: z = (T − 50) ÷ 10 = (30 − 50) ÷ 10 = −2.0. A z-score of −2.0 falls at roughly the 2nd–3rd percentile — using the 68-95-99.7 rule, about 95% of scores fall within ±2 SD, leaving about 2.5% below −2 SD.
Common Traps
- Treating a reliable result as automatically valid — the exam frequently tests this distinction with a scenario where a measure is consistent but clearly not measuring the intended construct.
- Confusing the standard deviation (spread of scores in the population) with the standard error of measurement (precision of an individual score).
- Accepting face validity as real evidence — it reflects only surface appearance, not psychometric data.
- Forgetting that a norm-referenced score is only meaningful relative to the specific norm group the test was standardized on (age, culture, language) — comparing a client's score to the wrong norm group invalidates the interpretation.
A counselor administers the same generalized-anxiety inventory to a group of clients twice, two weeks apart. The scores across clients correlate at .89. This is the strongest evidence of which type of reliability?
Internal consistency
Test-retest reliability
Inter-rater reliability
Parallel forms reliability
A new anxiety screener is compared with independent diagnosis by a structured clinical interview at the same visit, and the results agree closely. This is the best evidence of:
Face validity
Predictive validity
Concurrent validity
Internal consistency
A test has a standard deviation of 15 and a reliability coefficient of .91. Using SEM = SD x sqrt(1 - r), what is the approximate standard error of measurement?
1.5
4.5
9.0
13.5
A client has a T-score of 30 on a normally distributed clinical scale (mean = 50, SD = 10). Approximately what percentile does this represent?
50th percentile
84th percentile
16th percentile
2nd-3rd percentile
Sections you finish are checked off in the contents.