5.1 Psychometrics: Reliability, Validity & Score Interpretation
Key Takeaways
- Reliability (test-retest, internal consistency, inter-rater, parallel forms) measures consistency; validity (content, criterion, construct) measures whether a test measures what it claims — reliability is necessary but not sufficient for validity.
- SEM = SD x sqrt(1 - r); use it to build a confidence band around an observed score instead of treating a single point as exact.
- Standard scores translate raw performance onto a common scale: z-scores (mean 0, SD 1), T-scores (mean 50, SD 10), and IQ scores (mean 100, SD 15) all rely on the normal distribution and the 68-95-99.7 rule.
- Norm-referenced scores are only meaningful relative to the specific comparison group used to standardize the test; criterion-referenced scores compare against a fixed mastery standard instead.
- Face validity is the weakest form of validity evidence — it reflects surface appearance, not statistical support, and the exam treats it as a distractor, not a real answer.
Why Psychometrics Is Tested So Heavily
Intake, Assessment, and Diagnosis is Domain 2 of the NBCC Content Outline — 12% of the exam, or 19 of the 160 scored items. Within that domain, item M, "select, use, and interpret appropriate assessment instruments," cannot be answered correctly without fluency in psychometrics: the science of how test scores are constructed, evaluated, and interpreted. Assessment and Testing is also one of the eight CACREP Common Core Areas the NCE is aligned to, so psychometric vocabulary resurfaces across the exam wherever a question references a test score, a screening result, or an evaluation report.
On the job, this knowledge is not abstract. A counselor who receives a client's WAIS-IV composite score, a school's WISC-V report, or a PHQ-9 total must be able to say what that number actually means — how much measurement error surrounds it, whether the instrument was appropriate for this client, and whether the result supports a clinical decision. The NCE tests this directly with items that present a coefficient, a percentile, or a scenario and ask you to identify the underlying psychometric concept.
Reliability: Is the Measurement Consistent?
Reliability refers to the consistency of a test's results — whether it produces the same or similar scores across repeated administrations, raters, or item sets, assuming the underlying trait has not changed. Reliability is reported as a coefficient ranging from 0 to 1.00; coefficients of .80 or higher are generally considered acceptable for clinical use, and .90+ is preferred for high-stakes decisions.
| Type of Reliability | What It Measures | Example |
|---|---|---|
| Test-retest | Stability of scores over time | Same anxiety inventory given twice, two weeks apart |
| Internal consistency (Cronbach's alpha, split-half) | Whether items within one test measure the same construct | All 9 PHQ-9 items correlate with each other |
| Inter-rater | Agreement between two or more raters scoring the same behavior | Two clinicians independently score the same intake video |
| Parallel/alternate forms | Equivalence between two versions of the same test | Form A and Form B of a career-interest inventory yield similar scores |
Standard Error of Measurement (SEM)
Because no test is perfectly reliable, every observed score contains some measurement error. The standard error of measurement (SEM) estimates how much an observed score is likely to differ from a client's hypothetical "true score," and it is calculated as:
SEM = SD × √(1 − r)
where SD is the test's standard deviation and r is its reliability coefficient. A smaller SEM means a more precise instrument.
Worked example: A test has SD = 15 and reliability r = .91. SEM = 15 × √(1 − .91) = 15 × √.09 = 15 × .30 = 4.5
A client's observed score of 110 therefore falls within a 68% confidence band of roughly 105.5–114.5 (one SEM in each direction), and within a 95% confidence band of about 101–119 (about 2 SEMs in each direction). This is why a single point difference between two scores (e.g., 109 vs. 110) is rarely clinically meaningful — it likely reflects measurement error, not real change.
Validity: Does the Test Measure What It Claims To?
Validity is whether a test actually measures the construct it is designed to measure. Reliability is necessary but not sufficient for validity — a bathroom scale that is consistently 10 pounds off every time is highly reliable but not valid.
| Type of Validity | What It Establishes | Example |
|---|---|---|
| Content validity | Items adequately sample the full domain being measured | A depression scale covers mood, sleep, appetite, energy, and concentration — not just sad mood |
| Criterion validity — concurrent | Test scores correlate with an established measure given at the same time | A new burnout scale correlates highly with an already-validated burnout measure |
| Criterion validity — predictive | Test scores predict a future outcome | A college-admissions test predicts first-year GPA |
| Construct validity | The test truly measures the underlying theoretical construct (convergent: correlates with related measures; discriminant: does not correlate with unrelated ones) | An anxiety measure correlates with other anxiety measures but not with a measure of extraversion |
| Face validity | The test merely appears relevant to test-takers — weakest form, not true statistical evidence | A stress questionnaire "looks like" it measures stress |
Score Interpretation: Norms, Standard Scores, and the Normal Curve
Two broad interpretive frameworks appear on the exam:
- Norm-referenced scores compare a client's performance to a reference (comparison) group — expressed as a percentile rank, standard score, or stanine.
- Criterion-referenced scores compare performance to a fixed standard or mastery cutoff, independent of how any comparison group performed (e.g., a licensing exam's pass/fail cut score).
Most standardized psychological tests are norm-referenced and assume scores follow a normal distribution (bell curve), where the 68-95-99.7 rule applies: about 68% of scores fall within 1 standard deviation (SD) of the mean, about 95% fall within 2 SDs, and about 99.7% fall within 3 SDs.
| Standard Score | Mean | SD | 1 SD Above Mean |
|---|---|---|---|
| z-score | 0 | 1 | z = +1.0 |
| T-score | 50 | 10 | T = 60 |
| IQ score (e.g., WAIS-IV) | 100 | 15 | 115 |
Worked example: A client earns a T-score of 30 on a personality inventory subscale. Converting: z = (T − 50) ÷ 10 = (30 − 50) ÷ 10 = −2.0. A z-score of −2.0 falls at roughly the 2nd–3rd percentile — using the 68-95-99.7 rule, about 95% of scores fall within ±2 SD, leaving about 2.5% below −2 SD.
Common Traps
- Treating a reliable result as automatically valid — the exam frequently tests this distinction with a scenario where a measure is consistent but clearly not measuring the intended construct.
- Confusing the standard deviation (spread of scores in the population) with the standard error of measurement (precision of an individual score).
- Accepting face validity as real evidence — it reflects only surface appearance, not psychometric data.
- Forgetting that a norm-referenced score is only meaningful relative to the specific norm group the test was standardized on (age, culture, language) — comparing a client's score to the wrong norm group invalidates the interpretation.
A counselor administers the same generalized-anxiety inventory to a client twice, two weeks apart, and the two scores correlate at .89. This is the strongest evidence of which type of reliability?
A newly developed burnout questionnaire is given to a group of clinicians on the same day as an already well-validated burnout measure, and the two instruments show a strong positive correlation. This is the best evidence of:
A test has a standard deviation of 15 and a reliability coefficient of .91. Using SEM = SD x sqrt(1 - r), what is the approximate standard error of measurement?
A client scores a T-score of 30 on a clinical personality inventory subscale (mean = 50, SD = 10). Approximately what percentile does this represent?