1.2 Computer-Adaptive Testing (CAT) Mechanics, Question Branching & Unscored Trial Items

Computer-Adaptive Testing (CAT) Mechanics, Question Branching & Unscored Trial Items

Core Principle: Unlike fixed-length paper or linear computer exams, the Civil Service Verbal Test is adaptive. GOV.UK states that "the tests adapt to your performance - if you get a question right the next question may become harder, and if you get a question wrong the next question may become easier", and that "the tests can vary in length, but are typically shorter than fixed-length tests". Adaptive assessments of this kind are built on Computer-Adaptive Testing (CAT) and Item Response Theory (IRT); the sections below explain that machinery so you understand what the adaptation means in practice. The Civil Service does not publish its supplier's scoring model, so treat the parameters below as the standard psychometric account of adaptive testing, not as published CSVT specifications. The practical takeaway is unchanged: receiving progressively more challenging passages is a good sign.

---

1. Classical Testing vs. Modern Item Response Theory (IRT)

To comprehend how the CSVT evaluates candidates, one must distinguish between Classical Test Theory (CTT) and Item Response Theory (IRT):

Classical Test Theory (Linear Exams)

In a traditional linear examination, every candidate receives an identical set of questions arranged in a fixed sequence. A candidate's final score is simply a raw count of correct answers (e.g., 24 out of 30). CTT possesses major psychometric flaws:

• Easy questions provide zero discriminatory insight when evaluating high-ability candidates.
• Overly difficult questions cause lower-ability candidates to guess randomly, introducing statistical noise.
• Scores are entirely dependent on test difficulty: scoring 80% on an easy form cannot be directly compared to scoring 70% on a difficult form without complex retrospective equating.

Item Response Theory (Adaptive Exams)

IRT models the relationship between a candidate's underlying latent ability—conventionally denoted by the Greek letter theta ($\theta$)—and the statistical properties of individual test questions. In IRT, candidate ability is not a percentage of correct answers; it is a position on a continuous mathematical scale (typically centered around a mean of 0.0, with a standard deviation of 1.0, spanning from -3.0 to +3.0).

Under IRT, questions are not created equal. Each item in an IRT-calibrated bank is pre-tested on live responses before it counts, so that its mathematical parameters are known before it is used for scoring.

---

2. The Core Parameters of Item Response Theory

Modern psychometric test engines utilize three primary item parameters to characterize every question:

1. Difficulty Parameter ($b$): The location on the ability scale where a candidate has a 50% probability of answering the question correctly. A question with $b = -1.5$ is easy (even candidates with low verbal ability have a high probability of success), while an item with $b = +1.8$ is exceptionally difficult (only candidates with superior analytical ability are likely to answer correctly).
2. Discrimination Parameter ($a$): The steepness of the Item Characteristic Curve (ICC) at the difficulty threshold. A high discrimination value indicates that the question sharply differentiates between candidates who possess the required verbal reasoning skill and those who do not.
3. Pseudo-Guessing Parameter ($c$): The lower asymptote of the curve, representing the baseline probability that a candidate with minimal verbal ability could answer the question correctly purely through random guessing (e.g., 0.33 on a three-option True/False/Cannot Say question).

A CAT engine uses these parameters to calculate the Fisher Information Function for each available question, then selects the item that maximizes psychometric information at the candidate's current provisional ability level.

---

3. The Dynamic Branching Algorithm: How Question Selection Occurs

An adaptive branching process of the kind used for the CSVT operates through a continuous, cyclical computational loop:

Initialize Prior Ability Estimate (θ = 0.0)
       │
       ▼
Administer Calibrated Item
       │
       ▼
Capture Candidate Response (Correct / Incorrect)
       │
       ▼
Re-estimate Latent Ability (θ) & Calculate Standard Error (SEM)
       │
       ▼
Check Convergence Criteria (SEM ≤ Target OR Maximum Items Reached)
     ┌─┴─┐
    No   Yes ──► Finalize Score & Map to Norm Percentile
     │
     ▼
Query Bank for Unused Item Maximizing Fisher Information at Updated θ
     │
     └───► (Repeat Loop)

Step-by-Step Execution

• Item 1 (Initialization): The test commences by administering an item of moderate difficulty ($b \approx 0.0$), matching the population average.
• Branching Upon a Correct Response: If the candidate answers correctly, the algorithm updates the provisional ability estimate upwards (e.g., from $\theta = 0.0$ to $\theta = +0.6$). The engine then filters the question bank for an unadministered item whose difficulty aligns with $+0.6$, increasing the linguistic complexity, qualifier subtlety, or propositional density of the text.
• Branching Upon an Incorrect Response: If the candidate answers incorrectly, the algorithm revises the ability estimate downwards (e.g., to $\theta = -0.6$). The engine selects a subsequent question of lower difficulty to test whether the candidate can reliably identify more direct, explicit factual relationships.
• Precision Refinement: With every administered question, the algorithm incorporates new empirical data, continually narrowing the Standard Error of Measurement (SEM) surrounding the candidate's estimated ability.

---

4. Test Length Variability & Psychometric Convergence

Candidates frequently observe that the number of questions administered on the CSVT varies between applicants, or across different sittings. In fixed-form exams, test length is static (e.g., exactly 30 questions for all candidates). In a computer-adaptive test, test length is determined by psychometric convergence.

A CAT engine concludes the assessment when one of two conditions is fulfilled:

1. Variable-Length Stopping Rule (Measurement Precision): The test ends once the Standard Error of Measurement falls below a predefined psychometric threshold. A candidate who answers with highly consistent accuracy (or a consistent error pattern) lets the algorithm reach statistical confidence quickly, so the test finishes sooner.
2. Fixed Maximum Ceiling: If responses are inconsistent — alternating between correct answers on very difficult items and careless errors on easy ones — the standard error narrows slowly, and the test runs on until the platform's maximum item limit is reached.

What is actually published: The Civil Service does not publish the CSVT's stopping rule, its minimum or maximum item count, or the size of its question bank. Do not plan around a specific number of questions, and be sceptical of third-party sites that quote one. GOV.UK states only the practical consequence: "the tests can vary in length, but are typically shorter than fixed-length tests."

Because every question administered is tailored to the candidate's boundary of capability, an adaptive test can reach a reliable ability estimate in far fewer items than a conventional linear test. Time is never wasted asking questions that are either trivially easy or impossibly difficult for that specific individual.

---

5. Unscored Trial (Pretest) Items: Mechanics & Rationale

GOV.UK confirms that the CSVT embeds unscored trial items (also known as pretest or calibration items): "You will be asked to answer some trial items during the test. These items are not scored and do not form part of your assessment. Your answers to these questions help us calibrate new questions for future tests." Candidates are never told which items are scored and which are trial items.

Why Psychometricians Embed Trial Items

Before a newly authored question can be utilized in high-stakes civil service sifting, its mathematical parameters ($a$, $b$, and $c$) must be empirically verified across a live, diverse applicant cohort. Occupational psychologists must confirm that:

• The question's empirical difficulty matches its theoretical design.
• The item demonstrates high discrimination and does not introduce ambiguity.
• The item is free from Differential Item Functioning (DIF)—ensuring that candidates of equal verbal ability from different demographic, educational, or socioeconomic backgrounds have equal probabilities of answering correctly.

The Candidate Rule: Zero Discretion

Because trial items are visually, structurally, and functionally identical to live scored questions, candidates must never attempt to guess whether an item counts toward their score. Attempting to relax effort on questions that appear unfamiliar, awkwardly phrased, or experimental is disastrous: mistaking an active, high-difficulty scored item for a trial question will severely drag down the estimated ability parameter.

---

6. Psychometric Implications for Candidates: Why Harder Questions Are Favourable

The mathematical reality of IRT branching fundamentally alters test-taking psychology:

In classical testing, candidates often celebrate when questions feel easy. In an adaptive test like the CSVT, if questions feel effortless, it indicates that the algorithm has routed the candidate to lower-tier items with negative or near-zero ability estimates. Conversely, encountering convoluted policy excerpts with intricate conditional logic is strong evidence that the system is testing for top-percentile classification.