5.1 Case Study Evidence Synthesis: Contradictory Sources, Measurement Limits & Quantitative Checks
Key Takeaways
Check the group, measure, time period and source before you draw a conclusion or recommend action from figures in a practice case or FSAC brief.
Always check denominators alongside raw counts: a fall in raw complaints or a rise in raw survey replies can mask a worsening complaint rate or a falling response rate when the total cohort changes.
Detect double-counted savings by tracing whether two workstreams claim the same vacated estate, staff reduction, or licence cancellation.
Pilot results showing gains in some teams without controlling for extra staffing or seasonal shifts cannot support claims of universal causal impact.
Compare scope, authority and effective dates. A later document changes an earlier rule only if its instruction is authorised and applies to the decision.
Why Evidence Synthesis Drives Case Study and FSAC Performance
The online Case Study tests decisions in an evolving situation, and the assessment centre framework values critical analysis in the exercises where Think is assessed. Use the checks below to practise judging the evidence before deciding which action to take.
A headline can describe a real change and still suggest the wrong decision. Ask how the figure was calculated, who was counted, when it applied and whether it answers the question at hand.
Trap 1: Denominator and Population Shifts
Raw counts frequently move in the opposite direction from rates when the underlying population changes. A briefing pack may quote an encouraging drop in complaint numbers or an impressive rise in consultation responses without highlighting that the base cohort shifted.
Worked Example: Complaint Counts vs. Complaint Rates
Suppose a regional passport-support office runs a three-month process trial. Quarter 1 recorded 120 formal complaints out of 4,000 customer contacts. Quarter 2 recorded 96 formal complaints, a 20% fall in the count. However, a separate operations table notes that total contacts fell to 2,400 in Quarter 2 because two walk-in desks closed for refurbishment:
Raw complaints fell by 24, while complaints per contact rose by 1 percentage point, from 3% to 4%. Report both figures. Before concluding that service quality worsened or the trial caused the change, check whether the mix of contacts and the way complaints were recorded stayed comparable.
The identical trap appears in stakeholder engagement metrics. Receiving 420 survey replies out of 1,400 invited councils (a 30% response rate) represents lower proportional participation than receiving 360 replies out of 900 invited councils (40% response rate) the previous year.
Trap 2: Double-Counted Savings and Hidden Offsets
Public-sector business cases often combine proposals from separate teams. When two workstreams target overlapping resources, their projected savings cannot simply be added together.
Worked Example: Overlapping Estate and Hybrid-Working Savings
Imagine a departmental efficiency pack shows two proposals:
- Workstream A (Regional Hub Consolidation): Exits a leased annex building, projecting £180,000 in annual rent and utilities savings.
- Workstream B (Hybrid Desk-Sharing): Assumes 40% fewer permanent desks are needed and claims £180,000 in annual accommodation savings by releasing the same accommodation cost used in Workstream A. Assume the £180,000 saving is realised in full in Year 1 and neither proposal supplies any additional savings.
- Finance Footnote: Moving staff into the remaining main hub requires £35,000 in IT cabling and workstation upgrades during Year 1.
An uncritical reader sums Workstream A and Workstream B to report £360,000 in savings. In reality, the same physical saving has been counted twice, and once the £35,000 transition cost is deducted, net Year 1 savings are only £145,000. Check the duplicate with finance and report the one saving and one cost separately. If the lease saving starts partway through the year, calculate the amount actually realised before reporting a Year 1 net figure.
Trap 3: Measurement Limits and Causal Attribution
Operational pilots rarely take place in laboratory conditions. Before treating a pilot metric as proof that an intervention works everywhere, check for confounding variables, small sample sizes, and uneven implementation.
Suppose a 12-week case-management trial across five processing teams shows higher weekly completions in three teams and flat output in two teams. A background staffing sheet reveals that the three improving teams received temporary agency overtime during weeks 4 to 10, while the two flat teams operated at normal staffing and experienced a software outage in week 7. Claiming that the new workflow "boosted productivity across the division" overstates the evidence. Report the mixed results and the limits of what you can attribute to the workflow. A targeted follow-up could collect comparable staffing and outage data and use a suitable comparison before committing to expansion. The Magenta Book gives examples of evaluation designs for assessing alternative explanations.
Trap 4: Indicator vs. Outcome and Survey Representativeness
Distinguish outputs, such as delivery of an activity, from outcomes, changes that the activity aims to achieve. Check separately whether the evidence can attribute an outcome to the intervention.
- Output indicator: Distributing 15,000 digital-guidance leaflets or hosting 18 employer webinars proves activity occurred, not that employers understood the new compliance rule or submitted fewer erroneous returns.
- Policy outcome: A sustained drop in late-filing penalties or faster error-free clearance can indicate an outcome. The measure alone does not establish what caused the change.
- Non-random sampling bias: A voluntary survey shown only after online submission excludes people who abandoned the form and people who used other channels. Even among people who saw it, the responses may differ from those of the people who chose not to reply.
Important
Whenever a Case Study option proposes cutting an offline channel because "92% of online respondents reported high satisfaction," pause immediately. Online respondents are self-selected digital users; relying solely on their feedback risks excluding vulnerable or digitally excluded citizens.
Trap 5: Contradictory Documents and Date Boundaries
Sources in a practice case may differ in authority, dates or coverage. A timeline helps, but check whether each instruction applies to the decision.
- Check effective dates: An authorised May budget revision in force for the travel booking controls over an early February planning paper that used a higher allowance.
- Distinguish status: An unapproved draft recommendation from a junior working group does not carry the same authority as a signed ministerial or directorate instruction.
- Spot scope boundaries: A regional exemption granted for rural coastal branches does not automatically authorise urban offices to bypass the standard procedure.
| Analytical Trap | Common Pack Illusion | Verification Check | Proportionate Case Action |
|---|---|---|---|
| Denominator shift | Raw complaint count fell by 20% | Divide complaints by total contacts | Report the true rate change and investigate root causes |
| Double-counted saving | Two teams claim £180,000 each | Trace asset, headcount, or lease ID | Remove overlap, deduct setup costs, and quote net figure |
| Confounded pilot | Three of five teams improved | Check overtime, outages, and sample size | Report the mixed evidence and check the staffing contribution |
| Output vs. outcome | Thousands of leaflets issued | Check error rates or resolution times | Request outcome metrics alongside activity counts |
| Sampling bias | High satisfaction on portal pop-up | Ask who could not access the survey | Pair portal data with helpline and abandonment metrics |
| Stale document | Earlier memo permits full budget | Compare dates and sign-off status | Apply the latest authorised constraint in decisions |
Applying Quantitative Checks Under Pressure
Simple checks on ratios, net totals and dates can reveal an important problem with a recommendation. When choosing the most effective action or rating responses across the 24 situations, favour options that verify discrepancies transparently, protect analytical integrity, and state limits honestly rather than polishing weak numbers to please senior stakeholders.
A local grant programme reports that the number of rejected applications fell from 150 in Q1 to 120 in Q2. However, total applications submitted dropped from 1,500 in Q1 to 800 in Q2. Which conclusion is supported by the data?
Application quality improved markedly in Q2 because 30 fewer applications were rejected.
The rejection rate increased from 10% in Q1 to 15% in Q2, so a higher share of applicants are being rejected.
The programme saved administrative costs in Q2 without affecting applicant success.
Rejection figures cannot be compared across quarters unless total grant funding doubled.
In a practice pack, two workstreams each claim the same £90,000 saving from one software licence cancellation. The full saving will be realised in Year 1, and the only associated cost is £25,000 for data migration. What is the Year 1 net saving?
£180,000 gross saving before considering any future licence renewals.
£155,000, using both claims and subtracting the migration cost only once.
£90,000 net saving because migration costs fall under a separate IT budget.
£65,000 net saving after removing the duplicated licence saving and deducting £25,000 migration costs.
A 10-week pilot across four regional offices introduces a new triage script. Offices A and B show a 12% drop in call handling time, while Offices C and D show no change. You later read that Offices A and B also received upgraded call-routing software in week 2. What is the most accurate way to report this in a brief?
Note that faster call times in Offices A and B coincided with upgraded routing software, so the triage script's independent effect remains unproven.
Report that the triage script cut handling times across half the network and recommend immediate national mandate.
Exclude Offices C and D from the evaluation summary so senior leaders see a clear positive trend.
Conclude that Offices C and D failed to implement the script properly and discipline their team leads.
In a practice case, March guidance requires Deputy Director approval for regional travel above £500. An authorised 28 April instruction, effective immediately for this team, requires Director approval for discretionary travel above £100. On 3 May, what should you do with a £320 discretionary regional travel request?
Approve the £320 booking because it falls below the £500 threshold in the formal March guidance.
Split the booking into four £80 claims to avoid seeking Director approval.
Apply the 28 April Finance Director email and escalate the £320 request for Director approval before booking.
Wait until the next quarter without replying so the colleague misses the regional event.
Sections you finish are checked off in the contents.