6.6 Statistical Study Design, Margin of Error & Generalization
Key Takeaways
- Random sampling allows researchers to generalize study findings to the broader target population from which the sample was drawn, eliminating selection bias.
- Random assignment of subjects to treatment and control groups allows researchers to establish cause-and-effect (causal) relationships by balancing confounding variables.
- Observational studies observe existing behaviors without assigning treatments; they can establish correlation or association, but NEVER cause-and-effect.
- The margin of error (MoE) defines a confidence interval ($\text{Estimate} \pm \text{MoE}$) within which the true population parameter is expected to fall.
- Increasing the sample size $n$ decreases the margin of error inversely with the square root of $n$ ($\text{MoE} \propto \frac{1}{\sqrt{n}}$); to halve the margin of error, the sample size must be quadrupled ($4n$).
6.6 Statistical Study Design, Margin of Error & Generalization
Statistical inference questions on the Digital SAT evaluate whether conclusions drawn from scientific studies, clinical trials, or polling data are logically valid. Achieving full accuracy requires understanding the fundamental principles of data collection methodology, sample generalizability, experimental causality, and margin of error dynamics.
1. The Core Scientific Matrix: Sampling vs. Assignment
Every statistical study is defined by two independent design choices:
- How participants are selected: Random Sampling vs. Non-Random Sampling.
- How treatments are assigned: Random Assignment vs. Self-Selection (Observational).
+-----------------------------------------------------------------------------+
| THE STATISTICAL INFERENCE MATRIX |
| |
| RANDOM ASSIGNMENT? |
| YES NO |
| +-----------------------+-----------------------+ |
| YES | [ IDEAL EXPERIMENT ] | [ OBSERVATIONAL ] | |
| | - Causality PROVED | - Association ONLY | |
| RANDOM | - Generalizable to | - Generalizable to | |
| SAMPLING? | entire population | entire population | |
| +-----------------------+-----------------------+ |
| NO | [ LAB TRIAL ] | [ BIASED SURVEY ] | |
| | - Causality PROVED | - Association ONLY | |
| | - ONLY for sample | - CANNOT generalize | |
| | participants | to any population | |
| +-----------------------+-----------------------+ |
+-----------------------------------------------------------------------------+
The Two Golden Rules of Statistical Inference
- Random Sampling $\implies$ Generalization: Allows conclusions to be extended from the sample to the broader population from which the sample was randomly selected.
- Random Assignment $\implies$ Causation: Allows researchers to conclude that the treatment directly caused the observed difference in outcomes.
2. Observational Studies vs. Controlled Experiments
+-----------------------------------------------------------------------------+
| STUDY METHODOLOGY COMPARISON TABLE |
| |
| Dimension Observational Study Randomized Experiment |
| -------------------- ----------------------- --------------------- |
| Treatment Control None (Subjects choose) Assigned by researcher |
| Confounding Variables Present & Uncontrolled Balanced across groups |
| Valid Conclusion Correlation / Association Cause-and-Effect |
| Control Group Often absent Required (with placebo) |
+-----------------------------------------------------------------------------+
Confounding Variables & Why Observational Studies Cannot Prove Causation
A confounding (lurking) variable is an unmeasured third factor that influences both the explanatory variable and the response variable.
- Example: A study finds that people who drink 3 cups of coffee per day have lower rates of heart disease. However, coffee drinkers might also exercise more, have higher incomes, or follow healthier diets. Because treatments were not randomly assigned, coffee cannot be proven as the direct cause.
3. Sampling Frame & Population Generalizability
A study's findings can only be generalized to the specific population from which the sample was randomly drawn.
+-----------------------------------------------------------------------------+
| SAMPLING FRAME GENERALIZATION RULES |
| |
| Sampling Frame (Who was sampled) Valid Population for Inference |
| -------------------------------------- ------------------------------ |
| 600 registered voters in Ohio Registered voters in Ohio |
| (NOT all US voters; NOT non-voters
| 200 freshman students at UCLA Freshman students at UCLA |
| (NOT all college students in US) |
| 500 volunteers from a fitness app ONLY the 500 volunteers |
| (Voluntary response = NO pop) |
+-----------------------------------------------------------------------------+
Common Sampling Biases
- Selection / Convenience Bias: Sampling individuals who are easiest to reach (e.g., surveying people exiting a luxury gym about community fitness).
- Voluntary Response Bias: Allowing subjects to self-select into a poll (e.g., online comment polls where only extreme opinions participate).
- Non-Response Bias: When individuals who decline to respond differ systematically from those who do.
4. Margin of Error & Confidence Intervals
Surveys cannot poll every single member of a population; they survey a random sample to calculate an estimate (sample statistic $\hat{p}$). The margin of error (MoE) quantifies the expected random sampling variability.
+-----------------------------------------------------------------------------+
| MARGIN OF ERROR INTERVAL INTERPRETATION |
| |
| Polling Result: 54% support a proposition with a Margin of Error of ±3%. |
| |
| [ CONFIDENCE INTERVAL ]: 54% - 3% to 54% + 3% --> [ 51% to 57% ] |
| |
| Valid Inference: It is plausible that the true population proportion of |
| all voters who support the proposition is between 51% and 57%. |
| |
| Majority Claim: Because the entire interval [51%, 57%] is strictly |
| greater than 50%, the poll provides statistical evidence of a majority. |
+-----------------------------------------------------------------------------+
[!IMPORTANT] What Margin of Error Covers vs. What It Does NOT:
- Covers: Natural random sampling variation (chance differences between random samples).
- Does NOT Cover: Flawed survey design, leading questions, non-random sampling bias, measurement errors, or fraudulent responses.
5. Sample Size Dynamics and Margin of Error
The margin of error is inversely proportional to the square root of the sample size ($n$):
+-----------------------------------------------------------------------------+
| SAMPLE SIZE VS. MARGIN OF ERROR DYNAMICS |
| |
| Sample Size (n) Multiplier on n Effect on Margin of Error |
| ------------------- ------------------- ------------------------- |
| n = 100 Base sample size MoE = ±10% |
| n = 400 4x sample size MoE = 10% / √4 = ±5% (Halved!)|
| n = 900 9x sample size MoE = 10% / √9 = ±3.33% (1/3) |
| n = 1,600 16x sample size MoE = 10% / √16 = ±2.5% (1/4) |
+-----------------------------------------------------------------------------+
The Quadrupling Rule
To cut the margin of error in half ($\frac{1}{2}\times$), you must quadruple ($4\times$) the sample size.
Population Size Independence
As long as the total population is substantially larger than the sample (at least 10 to 20 times larger), the size of the population has no meaningful impact on the margin of error. A random sample of $n = 1{,}000$ people has virtually the same margin of error whether drawn from a city of $100{,}000$ or a nation of $330{,}000{,}000$.
6. Classic SAT Flawed Argument Traps
- The Overgeneralization Trap: Applying findings from a specific demographic (e.g., college athletes) to the entire general population.
- The Individual Determinism Fallacy: Claiming that because a group average increased, every individual in that group must experience the increase.
- The Observational Causation Fallacy: Concluding that an observed statistical correlation in a lifestyle survey proves a direct causal mechanism.
7. Desmos Playbook for Margin of Error & Interval Calculations
+-----------------------------------------------------------------------------+
| DESMOS MARGIN OF ERROR PLAYBOOK |
| |
| 1. Confidence Interval Bounds: |
| p = 0.54 |
| moe = 0.03 |
| [p - moe, p + moe] --> Desmos outputs [0.51, 0.57] |
| |
| 2. Sample Size Scaling: |
| n1 = 400 |
| moe1 = 0.05 |
| target_moe = 0.025 |
| n2 = n1 * (moe1 / target_moe)^2 --> Desmos outputs 1600 |
+-----------------------------------------------------------------------------+
[!TIP] Evaluating Margin of Error Scaling in Desmos: When asked what new sample size is required to achieve a target margin of error, evaluate
n_new = n_old * (moe_old / moe_new)^2directly in Desmos.
A researcher wants to estimate the proportion of residents in a city of 250,000 people who support constructing a new municipal bike path. The researcher randomly selects 600 registered voters from the city's voter registry and finds that 68% support the bike path, with a margin of error of 4%. To which of the following populations can the study's results be most appropriately generalized?
A university study tracked 1,200 randomly selected adults over a 5-year period and found that individuals who consumed at least 2 cups of green tea daily had a significantly lower incidence of cardiovascular disease than those who did not drink green tea. Which of the following is the most appropriate conclusion from this study?
In a public opinion poll of 400 randomly selected adults, 58% supported a proposed tax credit, with a margin of error of 5.0 percentage points. If the polling organization wants to conduct a follow-up survey of the same population to reduce the margin of error to 2.5 percentage points at the same level of confidence, approximately how many randomly selected adults should be surveyed?