24.1 Sampling, Experimentation & Data Collection
Key Takeaways
- The PA-CAT allocates 5% of its 240 items (about 12 scored questions) to Statistics; this section covers the Sampling & Experimentation group from Bulletin Table 11.
- Simple random sampling is the unbiased benchmark; stratified, cluster, and systematic methods trade precision for cost or subgroup coverage.
- Selection, nonresponse, and response bias each distort estimates in different ways and cannot be fixed by a larger sample size.
- Only randomized experiments support causal conclusions; observational studies can establish association but not causation.
- Randomization, control, blocking, and blinding are the four principles of sound experimental design; confounding is balanced in expectation by randomization.
Why Statistics Matters on the PA-CAT
The PA-CAT (Physician Assistant College Admission Test), developed by Exam Master Corporation, allocates 5% of its 240 multiple-choice items to Statistics (PA-CAT Bulletin of Information, rev. 20240815, Table 11). Roughly 12 scored items assess biostatistics, and two thematic groups appear repeatedly: Sampling & Experimentation, and Statistical Inference. This section addresses the first group—how data are gathered—because the validity of every later inference depends on the quality of the underlying data collection.
Census vs Sample
A census attempts to measure every individual in the population of interest. A sample measures a subset. In clinical research, a true census is rarely feasible. For example, estimating the prevalence of hypertension among all adults in the United States would require measuring blood pressure in roughly 260 million people; instead, the National Health and Nutrition Examination Survey (NHANES) draws a representative sample. A sample is preferred when measurement is destructive (e.g., biopsy), when the population is large or dynamic, or when time and cost are limiting. The defining goal is external validity: a well-drawn sample allows conclusions about the population without measuring everyone.
Probability Sampling Methods
Simple random sampling (SRS) gives every possible subset of size n an equal chance of selection. Each individual is chosen by a random mechanism (random number generator, random digit table). SRS is the conceptual benchmark but can be logistically difficult and may, by chance, underrepresent small subgroups.
Stratified random sampling divides the population into homogeneous strata (e.g., age bands, sex, geographic region) and takes an SRS within each stratum. Stratification guarantees representation of key subgroups and reduces sampling variability when within-stratum measurements are similar. A PA-CAT-style example: to estimate vaccination rates, stratify by state so that small states are not lost in a national SRS.
Cluster sampling randomly selects naturally occurring groups (clinics, schools, counties) and samples all or a subsample of individuals within selected clusters. Clusters are heterogeneous internally; between-cluster variability is the price of convenience. Cluster designs are common in multi-site clinical trials because they reduce travel and administrative cost.
Systematic sampling selects every kth individual from an ordered list (e.g., every 20th patient registering at a clinic). It is simple but vulnerable to periodicity: if the list cycles by day of week, systematic sampling can overrepresent Mondays.
| Method | Unit selected | Key advantage | Key risk |
|---|---|---|---|
| SRS | Individual | Unbiased, simple | May miss small subgroups |
| Stratified | Individual within stratum | Guarantees subgroup representation | Requires known strata |
| Cluster | Group | Cost-efficient | Less precise if clusters differ |
| Systematic | Every kth | Easy to implement | Bias from periodicity |
Bias: Selection, Nonresponse, Response
Selection bias arises when the sampling frame systematically excludes part of the population. A classic example is the 1936 Literary Digest poll, which sampled telephone owners during the Depression and mispredicted the presidential election. In medicine, a survey distributed through a hospital patient portal excludes patients without internet access.
Nonresponse bias occurs when selected individuals refuse or cannot be reached, and nonrespondents differ systematically from respondents. If a survey on pain management experience is completed only by patients with strong opinions, the estimated satisfaction is biased.
Response bias arises from how a question is asked or answered: leading wording, social desirability, recall error, or interviewer effects. Asking patients "You don't still smoke, do you?" invites underreporting. Anonymous self-administered questionnaires reduce social-desirability bias. Importantly, increasing the sample size does not fix bias—a larger biased sample is just a more precisely wrong estimate.
Observational vs Experimental Studies
An observational study records variables without imposing treatments. A cohort study follows a defined group forward in time; a case-control study compares cases (with disease) to controls (without) and looks backward at exposures; a cross-sectional study measures exposure and outcome at one time point. Observational studies can establish association but not causation because the investigator does not control confounders.
An experiment (clinical trial) assigns a treatment to units and observes the response. Randomized controlled trials (RCTs) are the gold standard for causal inference. The PA-CAT expects students to distinguish association from causation and to recognize that only randomized experiments justify causal language.
Design of Experiments
Four principles underpin a sound experiment:
- Randomization — randomly assign subjects to treatment and control groups so that known and unknown confounders are balanced in expectation. Randomization is what licenses causal conclusions.
- Control — include a comparison group (placebo, standard of care) to isolate the treatment effect. Without a control, improvement could reflect regression to the mean, placebo effect, or disease natural history.
- Blocking — group subjects with a known nuisance variable (e.g., study site, baseline severity) and randomize within blocks, removing that source of variability. Blocking is the experimental analog of stratification.
- Blinding — withhold group assignment from subjects (single-blind), investigators (double-blind), or outcome assessors. Blinding minimizes expectation bias and differential outcome assessment.
A placebo is an inert treatment that mimics the active intervention. A sham procedure plays the same role in surgical trials. Double-blinding is standard in pharmaceutical trials; surgical trials often use assessor-blinding instead.
Confounding
A confounding variable is associated with both the exposure and the outcome, distorting the apparent treatment effect. For example, coffee drinkers may have higher lung cancer rates not because of coffee but because smoking is more common among coffee drinkers. Randomization balances measured and unmeasured confounders in expectation; in observational studies, confounders must be addressed by restriction, matching, stratified analysis, or multivariable regression. Effect modification (interaction) is different: the treatment effect genuinely differs across levels of a third variable (e.g., a drug works better in younger patients). Effect modification is a finding to report, not a bias to eliminate.
Key Vocabulary for the Exam
- Population: the complete collection of units of interest.
- Sample: the subset actually measured.
- Sampling frame: the list from which the sample is drawn.
- Parameter: a fixed but usually unknown population quantity (Greek letters: μ, σ, p).
- Statistic: a quantity computed from the sample (x̄, s, p̂) used to estimate the parameter.
- Inferential statistics: using sample statistics to make statements about population parameters.
Which sampling method divides the population into homogeneous subgroups and takes a simple random sample within each?
A hospital posts a patient satisfaction survey on its online portal. Patients without internet access are excluded. This is an example of:
Which principle of experimental design is the experimental analog of stratified sampling?