23.4 Sampling Distributions & Simulations
Key Takeaways
- The sampling distribution of x̄ has mean μ and standard error σ/√n; for proportions the standard error is √(p(1−p)/n)
- The CLT lets you treat x̄ as approximately normal for n≥30 even when the parent is non-normal; for proportions require np≥10 and n(1−p)≥10
- Simulations approximate sampling distributions by repeatedly drawing samples and computing the statistic; the law of large numbers governs convergence
- Bias is systematic error that survives averaging; precision is random error that shrinks with larger n
- Simple random, stratified, cluster, and systematic sampling each trade cost for precision differently; non-response and undercoverage create bias that more data cannot fix
From a Single Sample to a Sampling Distribution
On the PA-CAT the blueprint group Anticipating Patterns (Bulletin Table 11, PA-CAT Bulletin of Information, rev. 20240815) explicitly includes sampling distributions and simulations. The conceptual leap is this: in real life you collect one sample and compute one statistic, but statistics asks what would happen if you repeated the sampling process many times. The distribution of that statistic over all possible samples is its sampling distribution. Once you can describe that distribution, you can quantify how far your one estimate is likely to be from the truth — which is exactly what a confidence interval and a hypothesis test do.
Sampling Distribution of the Sample Mean
If you draw simple random samples of size n from a population with mean μ and SD σ, the sampling distribution of the sample mean x̄ has:
- Mean: μ (x̄ is an unbiased estimator of μ).
- Standard error: SE(x̄) = σ / √n.
- Shape: approximately normal when n is large (Central Limit Theorem), or exactly normal when the parent is normal.
The key contrast is between the SD of the population, which describes spread among individuals, and the SE of the mean, which describes spread among sample means. A population SD of 15 mmHg for systolic blood pressure tells you how individuals vary; an SE of 15/√100 = 1.5 mmHg tells you how much sample means of size 100 vary. The PA-CAT will hand you both numbers and expect you to use the right one.
Sampling Distribution of the Sample Proportion
For a categorical outcome with true proportion p, the sample proportion p̂ from samples of size n has:
- Mean: p (unbiased).
- Standard error: SE(p̂) = √( p(1−p) / n ).
- Shape: approximately normal when np ≥ 10 and n(1−p) ≥ 10.
Worked Example
Suppose 30% of patients are non-adherent to a medication regimen. In samples of n = 200, the sampling distribution of p̂ has mean 0.30 and SE = √(0.30·0.70 / 200) = √(0.21/200) ≈ √0.00105 ≈ 0.0324. Because np = 60 and n(1−p) = 140 are both ≥ 10, the normal approximation is reasonable. The probability that p̂ exceeds 0.36 (i.e., 72 or more of 200) is P(Z > (0.36 − 0.30)/0.0324) = P(Z > 1.85) ≈ 0.032.
Standard Error Versus Margin of Error
The standard error is the SD of the statistic. The margin of error multiplies the SE by a critical z-value to build a confidence interval: ME = z* × SE. For a 95% interval z* ≈ 1.96. The PA-CAT usually asks for the SE itself, not the interval, but you should know that doubling the margin of error requires quadrupling n because SE shrinks with √n. To halve the ME, quadruple the sample size.
Simulations: Building a Sampling Distribution Empirically
When the math is intractable or you want to check an assumption, a simulation approximates the sampling distribution by repeatedly drawing samples and computing the statistic. The recipe:
- Specify a population model (e.g., a normal or skewed distribution, or resample from observed data).
- Draw a sample of size n and compute the statistic.
- Repeat many times (commonly 1,000–10,000 iterations).
- Plot the resulting statistic values; summarize their mean and SD.
A bootstrap simulation resamples with replacement from the observed data to estimate the sampling distribution without assuming a specific parent. The PA-CAT will not ask you to code a bootstrap, but it may describe one and ask what it demonstrates — typically that the empirical distribution of resampled means is centered near x̄ with SD close to s/√n.
Worked Simulation Example
A researcher simulates drawing 5,000 samples of size 50 from a right-skewed wait-time distribution with mean 35 minutes. The 5,000 sample means form a roughly symmetric, bell-shaped histogram centered at 35 with SD ≈ s/√50. The simulation demonstrates the CLT: even though individual wait times are skewed, the sample-mean distribution is approximately normal.
Bias Versus Precision
Two properties of an estimator matter on every PA-CAT Statistics item.
| Property | Definition | Reduced By |
|---|---|---|
| Bias | Systematic difference between the estimator's mean and the true parameter | Better design, not bigger n |
| Precision | How tightly repeated estimates cluster together | Larger n |
A biased estimator can be very precise yet still wrong: a miscalibrated scale that always reads 3 kg high is precise but biased. Increasing n reduces random error (improves precision) but never fixes systematic error. This is why study design — sampling method, question wording, instrument calibration — matters more than raw sample size for validity.
Sampling Methods
The PA-CAT expects you to recognize four common sampling designs and their bias profiles.
- Simple Random Sample (SRS): every set of n subjects has equal chance. Unbiased but logistically hard for large, dispersed populations.
- Stratified Random Sample: divide the population into homogeneous strata (e.g., age bands), then SRS within each. Reduces variability and guarantees representation of each stratum.
- Cluster Sample: randomly select clusters (e.g., hospitals) and sample all or part of each. Cheaper for geographically spread populations; less efficient than SRS of the same n.
- Systematic Sample: pick every k-th subject from a list. Easy, but vulnerable to periodicities in the list.
A multistage design combines these (e.g., cluster hospitals, then stratify by ward, then SRS patients). The key bias questions: is there undercoverage (some groups omitted), non-response (selected subjects who do not respond), response bias (subjects answer inaccurately), or selection bias (the sampling frame itself excludes part of the population)? These errors do not shrink with n.
Random Assignment Versus Random Sampling
Two different uses of randomness appear on the PA-CAT:
- Random sampling → generalizes from sample to population (external validity).
- Random assignment → establishes cause and effect within an experiment (internal validity).
A study can have one without the other. A randomized controlled trial at a single hospital uses random assignment (good internal validity) but a convenience sample of that hospital's patients (limited external validity). A national probability survey uses random sampling (good external validity) but cannot establish causation because it is observational.
Simulation Tie-In: Sampling Distributions You Can See
Modern statistics education increasingly uses applets that animate sampling distributions. The pedagogical pattern the PA-CAT rewards:
- Start with a parent population shape (normal, skewed, bimodal).
- Choose n = 5 and watch the simulated x̄ distribution — still skewed.
- Increase to n = 30 — the distribution becomes bell-shaped.
- Increase to n = 100 — the spread visibly shrinks by roughly √(100/30) ≈ 1.83×.
This pattern makes three lessons concrete at once: the mean stays at μ, the shape becomes normal (CLT), and the spread shrinks as 1/√n. The same applet run with p̂ from a categorical population shows the normal approximation emerging as np and n(1−p) cross 10.
Connecting to Inference
Everything in this section sets up the Statistical Inference blueprint group. A confidence interval is x̄ ± z* × SE(x̄); a hypothesis test compares a standardized statistic (z = (x̄ − μ₀) / SE) to a reference distribution. The standard error you computed here is the same one that flows into every test statistic and every margin of error. Treat sampling distributions and simulations as the load-bearing wall: if your SE is wrong — because you used σ instead of σ/√n, or used the wrong n — every downstream inference is wrong too.
A population proportion is 0.30. What is the standard error of p-hat for samples of size 200?
A researcher resamples with replacement 5,000 times from an observed data set and computes the mean each time. This procedure is best described as a:
Which statement correctly distinguishes bias from precision?