5.3 Test Reliability Metrics, Standard Error of Measurement, Cut Scores & Banding

5.3 Test Reliability Metrics, Standard Error of Measurement, Cut Scores & Banding

In public sector selection, decisions to rank, certify, hire, or eliminate candidates carry profound constitutional, legal, and operational consequences. To be legally defensible and merit-compliant, selection instruments must demonstrate high reliability (measurement consistency) and transparent standard-setting (cut-off score establishment). Furthermore, modern public agencies utilize psychometric score banding to reconcile competitive ranking with practical administrative flexibility.

---

1. Classical Test Theory (CTT) and Types of Reliability

Under Classical Test Theory (CTT), every observed test score ($X$) consists of two additive components: an individual's true underlying competence (True Score, $T$) and random unobservable measurement error (Error, $E$):

$X = T + E$

Reliability ($r_{xx}$) is the proportion of total observed score variance ($\sigma^2_X$) that is attributable to true score variance ($\sigma^2_T$):

$r_{xx} = \frac{\sigma^2_T}{\sigma^2_X} = 1 - \frac{\sigma^2_E}{\sigma^2_X}$

+-----------------------------------------------------------------------------------------+
|                    CLASSICAL TEST THEORY: SCORE COMPOSITION                             |
|                                                                                         |
|   OBSERVED SCORE (X)  =======================================================> [100%] |
|   TRUE SCORE (T)      ===============================================> [85%]   (r_xx)   |
|   ERROR SCORE (E)     ========> [15%]                                          (1-r_xx) |
|                                                                                         |
|   *Reliability sets the theoretical ceiling for validity: r_xy <= sqrt(r_xx)            |
+-----------------------------------------------------------------------------------------+

The Golden Rule of Psychometrics: Reliability Precedes Validity

A test cannot be valid unless it is reliable. An unreliable test measures random noise. Mathematically, the maximum possible validity coefficient ($r_{xy}$) of a test is equal to the square root of its reliability ($r_{xx}$):

$r_{xy\text{ (max)}} = \sqrt{r_{xx}}$

Typology of Reliability Metrics in Civil Service Testing

Spearman-Brown Prophecy Formula

When calculating split-half reliability or projecting the reliability of a lengthened test, the Spearman-Brown formula is applied:

$r_{SB} = \frac{k \cdot r_{xx}}{1 + (k - 1)r_{xx}}$

Where $k$ is the factor by which the test length is altered (for split-half, $k = 2$, yielding $r_{SB} = \frac{2r_{half}}{1 + r_{half}}$).

Acceptable Reliability Thresholds in High-Stakes Public Selection:

• $r_{xx} \ge .90$: Required for high-stakes individual licensing, promotional ranking, or pass-fail cutoffs.
• $r_{xx} = .80 - .89$: Acceptable for general competitive selection batteries.
• $r_{xx} < .70$: Unacceptable for civil service decision-making.

---

2. The Standard Error of Measurement (SEM)

The Standard Error of Measurement (SEM) quantifies the standard deviation of observed scores an individual would obtain if they took the identical test an infinite number of times under identical conditions. It translates abstract reliability coefficients into the actual raw score units of the examination.

+-----------------------------------------------------------------------------------------+
|                        STANDARD ERROR OF MEASUREMENT (SEM)                              |
|                                                                                         |
|                               SEM  =  SD * sqrt(1 - r_xx)                               |
|                                                                                         |
|   Where:                                                                                |
|   SD   = Standard Deviation of the test score distribution                              |
|   r_xx = Reliability coefficient of the examination                                     |
+-----------------------------------------------------------------------------------------+

Practical Calculation and Confidence Intervals

Suppose a municipal Police Sergeant promotional exam has a mean of 75, a Standard Deviation ($SD$) of 10, and an internal consistency reliability ($r_{xx}$) of 0.84.

1. Compute SEM: $SEM = 10 \times \sqrt{1 - 0.84} = 10 \times \sqrt{0.16} = 10 \times 0.40 = 4.0\text{ points}$
2. Construct Confidence Intervals: If Candidate A scores 82 points:
   • 68% Confidence Interval ($\pm 1 SEM$): $82 \pm 4.0 = [78.0\text{ to }86.0]$
   • 95% Confidence Interval ($\pm 1.96 SEM$): $82 \pm (1.96 \times 4.0) = 82 \pm 7.84 = [74.16\text{ to }89.84]$

[!NOTE] Psychometric Reality: A candidate who scores an 82 on this examination does not possess a fixed ability of 82. We can only be 95% confident that the candidate's true competence falls somewhere between 74.2 and 89.8.

---

3. Standard-Setting: Establishing Defensible Cut Scores

Setting a cut-off score (passing score) is one of the most legally scrutinized phases of civil service administration. UGESP Section 5H mandates that cut-off scores must be reasonable and consistent with normal expectations of acceptable proficiency within the workforce.

+-----------------------------------------------------------------------------------------+
|                         METHODS FOR SETTING PASSING CUT SCORES                          |
|                                                                                         |
|   +------------------------------------+   +------------------------------------+       |
|   |         MODIFIED ANGOFF            |   |              NEDELSKY              |       |
|   | - Test-centered method             |   | - Distractor-elimination method    |       |
|   | - SMEs estimate probability that a |   | - SMEs assess probability of       |       |
|   |   Minimally Qualified Candidate    |   |   guessing among distractors       |       |
|   |   (MQC) will answer item correctly |   | - Used for multiple choice         |       |
|   +------------------------------------+   +------------------------------------+       |
|                     |                                         |                         |
|                     v                                         v                         |
|   +------------------------------------+   +------------------------------------+       |
|   |         BOOKMARK METHOD            |   |             EBEL METHOD            |       |
|   | - Item Response Theory (IRT) based |   | - Two-dimensional classification:  |       |
|   | - Ordered item booklet by diff.    |   |   Difficulty (Easy/Med/Hard) vs    |       |
|   | - SMEs place bookmark at MQC cut   |   |   Relevance (Essential/Important)  |       |
|   +------------------------------------+   +------------------------------------+       |
+-----------------------------------------------------------------------------------------+

The Modified Angoff Protocol (The Industry Standard)

The modified Angoff method is the most widely accepted standard-setting protocol in public sector testing:

1. Define the Minimally Qualified Candidate (MQC): A panel of 10–20 Subject Matter Experts (SMEs/senior supervisors) formulates a shared operational profile of the "minimally competent performer"—someone who possesses just enough knowledge and skill to perform the job safely and adequately.
2. Individual Item Ratings (Round 1): For each test question, each SME independently answers: "What percentage of 100 Minimally Qualified Candidates would answer this item correctly?" (e.g., 65%, 70%, 40%).
3. Panel Deliberation and Normative Feedback: SMEs review item difficulty statistics and discuss significant rating variances.
4. Second Round Ratings (Round 2): SMEs independently submit final percentage ratings for each item.
5. Aggregate Cut Score Computation: The average estimated probability across all items and all raters becomes the recommended raw cut score.

$\text{Angoff Cut Score} = \frac{1}{K} \sum_{i=1}^{K} \left( \frac{1}{N} \sum_{j=1}^{N} P_{ij} \right)$

Where $K$ is total test items, $N$ is total SME raters, and $P_{ij}$ is the probability assigned by rater $j$ to item $i$.

[!WARNING] The Arbitrary 70% Cut-Off Trap: Historically, many municipal civil service rules arbitrarily mandated a "70% passing score." Federal courts have repeatedly invalidated arbitrary 70% cutoffs under Title VII when they produced adverse impact and lacked empirical standard-setting grounding (Kirkland v. New York State Dept. of Correctional Services). Cut scores must be empirically tied to minimum job competence.

4. Score Banding in Civil Service Selection

Traditional civil service systems rely on strict top-down ranking, where a candidate with a score of 91.2 is legally certified ahead of a candidate with a 91.0, despite the difference being psychometrically meaningless measurement error. To mitigate this flaw, public employers utilize Score Banding.

Score banding is a psychometrically grounded procedure that groups a range of test scores into a single equivalence band based on the test's Standard Error of Measurement. All candidates within a band are treated as statistically equivalent regarding test performance, allowing hiring authorities to select among candidates within the band using secondary, job-related merit criteria (e.g., specialized certifications, bilingual capability, structured interview ratings, performance evaluations).

+-----------------------------------------------------------------------------------------+
|                    SCORE BANDWIDTH COMPUTATION: SEM VS. SED                             |
|                                                                                         |
|   [STANDARD ERROR OF MEASUREMENT (SEM) BANDWIDTH]                                       |
|                                                                                         |
|                       Bandwidth  =  C * SEM  =  C * [ SD * sqrt(1 - r_xx) ]             |
|                                                                                         |
|   Where C is a critical value constant based on statistical confidence:                 |
|   - C = 1.00  (68% Confidence Bandwidth)                                               |
|   - C = 1.64  (90% One-Tailed Confidence Bandwidth)                                     |
|   - C = 1.96  (95% Two-Tailed Confidence Bandwidth)                                     |
|                                                                                         |
|   -----------------------------------------------------------------------------------   |
|                                                                                         |
|   [STANDARD ERROR OF THE DIFFERENCE (SED) BANDWIDTH] (Recommended for Pairwise Comp)    |
|                                                                                         |
|                   SED  =  sqrt(SEM_1^2 + SEM_2^2)  =  sqrt(2) * SEM                     |
|                                                                                         |
|                   Bandwidth  =  C * SED  =  C * [ sqrt(2) * SD * sqrt(1 - r_xx) ]      |
+-----------------------------------------------------------------------------------------+

Banding Modalities: Fixed vs. Sliding Bands

+-----------------------------------------------------------------------------------------+
|                        FIXED BANDING VS. SLIDING BANDING                                |
|                                                                                         |
|   [FIXED BANDING MODEL]                                                                 |
|   Band 1: [Scores 91 to 100]  ===> All candidates in Band 1 must be hired or            |
|                                    exhausted before Band 2 can be accessed.             |
|   Band 2: [Scores 81 to 90]                                                             |
|   Band 3: [Scores 71 to 80]                                                             |
|                                                                                         |
|   -----------------------------------------------------------------------------------   |
|                                                                                         |
|   [SLIDING BANDING MODEL (DYNAMIC)]                                                     |
|   Initial Band: Top Score is 98. Bandwidth is 8 points. Band 1 = [90 to 98].            |
|                                                                                         |
|   Action 1: Candidate with score 98 is HIRED.                                           |
|   Action 2: Top remaining score is now 94.                                             |
|   Action 3: Band automatically SLIDES down from 94: New Band = [86 to 94] (94 - 8 = 86).|
|             Candidates with scores 86-89 now enter the active selection band!           |
+-----------------------------------------------------------------------------------------+

Legal Defensibility of Banding: Section 106 of CRA 1991

The legality of score banding was confirmed in Bridgeport Guardians v. City of Bridgeport (933 F.2d 1140) and Officers for Justice v. Civil Service Commission (979 F.2d 727). However, public employers must adhere strictly to Section 106 of the Civil Rights Act of 1991 (42 U.S.C. § 2000e-2(l)):

• The Rule: Employers may NOT alter test scores, use different cutoffs, or adjust scores on the basis of race, color, religion, sex, or national origin (prohibition of race norming).
• Compliant Application: Score banding is fully lawful because band widths are calculated based on overall psychometric measurement error for all candidates. However, once a band is established, selection within the band cannot be made strictly on the basis of race or gender. Secondary selection factors must be legitimate, job-related criteria (e.g., veteran status, education, tenure, work history, bilingual ability).