10.3 Sampling Bias, Misleading Graphs, Base-Rate Fallacies, and Evaluating Media/Research Claims

Population vs. Sample and Sources of Sampling Bias

In statistical investigations, the population is the entire target group about which a school or researcher wishes to draw a conclusion (for example, all $1{,}200$ families enrolled at a P–12 college), whereas a sample is the subset actually surveyed. For sample statistics to generalise validly to the population, the sample must be representative—meaning every subgroup of the population is included without systematic distortion.

The gold standard is random sampling:

• Simple random sampling: Every member of the entire population has an equal chance of being selected (for example, using a random number generator on the complete school enrolment roll).
• Stratified random sampling: The population is first divided into non-overlapping strata (such as Year 7 through Year 12), and a simple random sample is drawn from each stratum in exact proportion to its enrolment size.

By contrast, non-random methods introduce sampling bias:

• Convenience sampling: Selecting individuals who are easiest to reach. For example, surveying only parents who attend a Tuesday $10\text{:}00\text{ am}$ P&C meeting systematically excludes parents who work full-time daytime shifts, while surveying students inside the library at lunchtime about extending library hours over-represents keen library users.
• Voluntary-response (self-selection) bias: Inviting respondents to opt in to an online newsletter poll or return an optional feedback slip. People with strong complaints or passionate views are far more likely to respond than satisfied or neutral community members.
• Undercoverage bias: Drawing a sample from an incomplete frame that omits key segments of the population—such as surveying only a Year 12 Extension Mathematics class to estimate whole-school attitudes toward daily homework.
• Small sample size fallacy: Treating a tiny sample (such as interviewing $5$ students at the bus bay) as conclusive evidence for a whole-school percentage.

---

The Base-Rate Fallacy: Raw Counts vs. Proportional Rates

One of the most frequent analytical traps in LANTITE Numeracy is confusing raw counts (absolute frequencies) with proportional rates (percentages or per-capita rates) when comparing groups of different total sizes.

Worked Example: Canteen Participation Across Unequal Cohorts

Suppose a secondary school records that $45$ Year 7 students and $35$ Year 11 students buy hot lunches from the canteen on Monday:

• Comparing raw counts: Year 7 has more students buying lunch ($45 > 35$).
• Comparing participation rates: Suppose the Year 7 cohort has $300$ students in total, whereas the Year 11 cohort has $140$ students in total. Converting each count to a proportion of its own base cohort gives:

$$\text{Year 7 rate} = \frac{45}{300} = 0.15 = 15\%$$ $$\text{Year 11 rate} = \frac{35}{140} = 0.25 = 25\%$$

Once base cohort sizes are accounted for, Year 11 has a substantially higher participation rate ($25\%$ vs. $15\%$) despite having fewer buyers in raw headcount.

The converse base-rate trap occurs when a graph displays only percentages without cohort totals. If a chart shows that $60\%$ of School A's graduates and $45\%$ of School B's graduates entered university, you cannot determine which school sent a greater number of students to university unless the total number of graduates at each school is provided.

---

Simpson's Paradox, Survivorship Bias, and Misleading Visuals

Even when data are collected accurately, summary statistics and visuals can mislead readers in three important ways:

1. Simpson's Paradox (Subgroup Aggregation Reversal): A comparison that holds within every individual subgroup can reverse direction when the subgroups are combined into a single aggregate percentage if the groups have very different underlying compositions. For example, suppose School X and School Y both have $100$ students tested: at School X, $\frac{48}{80} = 60\%$ of EAL/D students and $\frac{18}{20} = 90\%$ of non-EAL/D students meet a writing benchmark (overall $\frac{66}{100} = 66\%$). At School Y, only $\frac{10}{20} = 50\%$ of EAL/D students and $\frac{64}{80} = 80\%$ of non-EAL/D students meet the benchmark, yet its overall combined rate is $\frac{74}{100} = 74\%$. School X outperforms School Y in both subgroups ($60\% > 50\%$ and $90\% > 80\%$), but appears lower overall ($66\% < 74\%$) solely because $80\%$ of School X's cohort is in the EAL/D subgroup.
2. Survivorship Bias in Longitudinal Cohorts: Suppose a cohort's mean numeracy score rises from $62\%$ in Year 10 to $71\%$ in Year 12. Concluding that classroom instruction alone caused a $9$-percentage-point improvement ignores whether lower-scoring students left school after Year 10 for apprenticeships or vocational training, leaving a more selective surviving cohort in Year 12.
3. Misleading Graph Scales: Always inspect axis labels before interpreting visual height. A truncated vertical axis (starting a bar chart at $80\%$ instead of $0\%$) makes an $86\%$ bar appear three times as tall as an $82\%$ bar. Similarly, omitting uneven years on a horizontal time axis distorts the apparent gradient of growth.

---

Validity and Reliability of Data and Interpretations

ACER's framework names data and interpretation validity and reliability as in-scope statistics content. Keep the two ideas separate:

Common threats to a valid interpretation:

• Changed measures: comparing 2024 and 2026 pass rates after the test, marking scheme, or cut-off changed.
• Mismatched populations: drawing conclusions about all students from data on volunteers or a single class.
• Over-reach: claiming a cause from a correlation, or predicting far beyond the range of the data.

Reliability improves with larger samples, averages of several measurements, clear marking criteria (or two markers who agree), and results compared across several occasions rather than a single day. A result can be reliable without being valid: a bathroom scale that always reads $2\text{ kg}$ heavy is consistent but wrong, so check both properties before accepting a conclusion.

---

Evaluating Claims: "True", "False", or "Unable to Determine"

Approximately $20\text{--}30\%$ of the LANTITE Numeracy test assesses the process of interpreting, evaluating, communicating, and representing mathematics. ACER practice items present a stimulus table or graph and ask you to mark each statement True, False, or Unable to determine (some items say Not possible to determine):

Worked Example: Classifying Claims from a School Excursion Summary

Suppose a table reports only the percentage of students attending a museum excursion across two campuses: Campus East ($80\%$ attendance) and Campus West ($70\%$ attendance), with a combined two-campus attendance rate of $74\%$:

• Claim 1: "The attendance rate at Campus East was $10$ percentage points higher than at Campus West." $\rightarrow$ True ($80\% - 70\% = 10$ percentage points).
• Claim 2: "Campus East and Campus West have the same total number of enrolled students." $\rightarrow$ False (if both campuses had equal enrolments, the combined rate would be the simple midpoint $\frac{80\% + 70\%}{2} = 75\%$; because the weighted average is $74\%$, Campus West must have a larger total enrolment than Campus East).
• Claim 3: "Exactly $160$ students from Campus East attended the excursion." $\rightarrow$ Unable to determine (no headcount or total enrolment number is given in the table).