7.3 Study Design, Sampling Methods & Sources of Bias

7.3 Study Design, Sampling Methods & Sources of Bias

The validity of any statistical conclusion is inextricably bound to the method by which data were collected. In secondary mathematics curricula, including AP Statistics and Integrated Mathematics pathways, students must learn that sophisticated mathematical calculations cannot rescue data corrupted by flawed study designs. Accomplished teachers guide students through the structural differences between observational studies and experiments, the mathematics of sampling methods, the taxonomy of bias, and the experimental safeguards required to establish valid causal inferences.

1. Observational Studies vs. Controlled Experiments

The foundational distinction in empirical research is between passive observation and active experimental intervention.

Observational Studies

In an observational study, researchers observe individuals and record variables of interest but do not attempt to influence the responses or assign treatments.

• Retrospective Studies: Investigate existing records or historical conditions to identify associations between past exposures and present outcomes (e.g., case-control medical studies).
• Prospective Studies: Identify a cohort of subjects in the present and track them forward in time, collecting data periodically as natural events unfold.
• Fundamental Limitation: Observational studies can establish association (correlation), but they cannot establish causation due to the uncontrolled presence of confounding variables.

Controlled Experiments

In a controlled experiment, researchers deliberately impose specific experimental conditions (treatments) on experimental units (human subjects, animals, or physical objects) to observe their effect on one or more response variables.

• Factor: An explanatory variable manipulated by the experimenter (e.g., dosage of a medication).
• Levels: The specific values or concentrations assigned for a factor (e.g., 0 mg, 50 mg, 100 mg).
• Treatments: Any specific experimental condition formed by a combination of factor levels.
• Causal Capability: Well-designed randomized experiments can establish cause-and-effect relationships because random assignment distributes confounding variables evenly across treatment groups.

Confounding Variables

A confounding variable is an extraneous variable related to both the explanatory variable and the response variable, such that its influence on the response cannot be untangled from the effect of the explanatory variable.

                  [ Confounding Variable (Z) ]
                         /            \
                        /              \
                       v                v
    [ Explanatory Variable (X) ] ----> [ Response Variable (Y) ]
                 Spurious / Distorted Association

Example: An observational study notes that people who consume high amounts of coffee have higher rates of cardiovascular disease. Smoking status acts as a confounding variable: coffee consumption is positively associated with smoking, and smoking independently causes cardiovascular damage.

2. Probability Sampling Methods

Sampling theory seeks to obtain a representative subset of a population $N$ using a sample of size $n$, ensuring that sample statistics provide unbiased estimates of population parameters.

1. Simple Random Sample (SRS)

A Simple Random Sample of size $n$ is selected in such a manner that every individual in the population has an equal probability of being chosen, and every possible subset of $n$ individuals has an equal probability of being selected as the sample.

• Procedure: Compile an exhaustive sampling frame of all $N$ individuals numbered $1$ to $N$. Use a random number generator or random digit table to draw $n$ unique integers without replacement.

2. Stratified Random Sampling

In stratified sampling, the population is divided into mutually exclusive, non-overlapping subgroups called strata that are internally homogeneous with respect to a characteristic known to influence the response variable (e.g., grouping high school students by grade level: 9th, 10th, 11th, 12th).

• Procedure: Conduct an independent SRS within every stratum, and combine the resulting samples.
• Key Advantage: Guarantees representation of key subgroups and substantially reduces sampling variability (standard error) compared to an SRS of the same total size: "Homogeneous within, heterogeneous between."

3. Cluster Sampling

In cluster sampling, the population is divided into naturally occurring, geographically or organizationally convenient subgroups called clusters that are internally heterogeneous (each cluster is ideally a miniature microcosm of the entire population, e.g., homeroom classrooms or municipal voting precincts).

• Procedure: Select a random sample of clusters using an SRS. Measure all individuals within the selected clusters.
• Key Advantage: Cost-effective and logistically practical when an exhaustive population frame of individuals is unavailable, though it typically produces higher sampling variability than stratified sampling: "Heterogeneous within, homogeneous between."

4. Systematic Sampling

In systematic sampling, individuals are selected from an ordered frame by choosing every $k$-th individual after a randomly determined starting point between $1$ and $k$ (where $k \approx N/n$).

• Caution: If the ordering of the sampling frame contains hidden periodic cycles matching the interval $k$, severe systematic bias will occur.

Non-Probability Sampling (Flawed Methods)

• Convenience Sampling: Selecting individuals who are easiest to reach (e.g., surveying shoppers at a single shopping mall entrance). Produces severe selection bias.
• Voluntary Response Sampling: Allowing individuals to self-select into the sample (e.g., internet polls, call-in radio shows). Almost always over-represents individuals with extreme negative opinions.

3. Sources of Bias in Surveys and Studies

Bias is the systematic tendency of a study design to favor certain outcomes, causing sample estimates to diverge systematically from the true population parameter.

[!WARNING] The Large Sample Size Fallacy: A pervasive misconception among secondary students is that increasing the sample size $n$ eliminates bias. Increasing sample size does not reduce or correct systematic bias. A biased sample of $n = 1,000,000$ merely yields an exquisitely precise, completely erroneous estimate. Sample size reduces random sampling error (variance), not systematic error (bias).

4. Principles of Experimental Design

To establish causality, an experiment must isolate the effect of the treatment from all other sources of variability. Accomplished teachers emphasize the Four Foundational Pillars of Experimental Design:

1. Comparison / Control: Include at least two treatment groups (or a treatment group and an inactive control group) to control for external lurking variables, environmental fluctuations, and historical drift.
2. Random Assignment (Randomization): Use a chance mechanism (e.g., random number generator, coin flip) to assign experimental units to treatment groups. Random assignment balances known and unknown confounding variables across groups, establishing baseline equivalence.
3. Replication: Apply each treatment to a sufficiently large number of experimental units so that genuine treatment effects can be distinguished from random chance variability. (Replication also refers to repeating the entire experiment across different populations).
4. Blocking: Group experimental units into blocks based on a characteristic known to influence the response variable before administering treatments. Random assignment of treatments is then conducted independently within each block.

Experimental Architectures

• Completely Randomized Design (CRD): All experimental units are allocated across treatment groups entirely by chance.
• Randomized Block Design (RBD): Units are first stratified into blocks (e.g., blocking patients by baseline blood pressure: low, medium, high), and then randomly assigned to treatments within each block. Blocking removes block-to-block variation from the unexplained residual error: "Control what you can, block what you can predict, and randomize the rest."
• Matched Pairs Design: A specialized form of blocking where either:
  1. Two closely matched subjects (e.g., identical twins or subjects paired by identical age, weight, and health) are assigned randomly to treatment A or B.
  2. Each subject acts as their own control, receiving both treatments in a randomly assigned sequential order (crossover design).

Control Safeguards: Blinding and Placebos

• The Placebo Effect: The psychological phenomenon where subjects exhibit measurable clinical improvement simply because they believe they are receiving a beneficial treatment.
• Single-Blind: The experimental units do not know which treatment they are receiving (prevents psychological expectation bias).
• Double-Blind: Neither the experimental units nor the individuals measuring/evaluating responses know which treatment was administered (prevents experimenter bias and diagnostic expectancy).

5. Pedagogical Scenarios: The Scope of Statistical Inference

A critical pedagogical framework in grades 9–12 mathematics is the Scope of Inference 2 × 2 Decision Matrix, which governs what conclusions are mathematically legitimate:

+-----------------------------------------------------------------------------------+
|                         THE SCOPE OF INFERENCE MATRIX                             |
+-----------------------+-----------------------------------------------------------+
|                       |            WERE SUBJECTS RANDOMLY ASSIGNED TO GROUPS?     |
|                       +-----------------------------+-----------------------------+
|                       |            YES              |             NO              |
+---+-------------------+-----------------------------+-----------------------------+
| W |                   | Causal Inference Allowed?   | Causal Inference Allowed?   |
| E |        YES        |            YES              |             NO              |
| R | (Random Sample    | Generalize to Population?   | Generalize to Population?   |
| E |  from Population) |            YES              |            YES              |
| S |                   | (Ideal Randomized Experiment| (Ideal Observational Study  |
| U |                   |  with Representative Sample)|  with Representative Sample)|
| B +-------------------+-----------------------------+-----------------------------+
| J |                   | Causal Inference Allowed?   | Causal Inference Allowed?   |
| E |        NO         |            YES              |             NO              |
| C | (Volunteers or    | Generalize to Population?   | Generalize to Population?   |
| T |  Convenience)     |             NO              |             NO              |
| S |                   | (Standard Clinical Trial    | (Anecdotal / Voluntary      |
| ? |                   |  with Volunteers)           |  Polls - Severely Limited)  |
+---+-------------------+-----------------------------+-----------------------------+

Guiding Secondary Students in Research Design

When secondary students propose statistical investigations, accomplished teachers prompt them to articulate:

1. The Statistical Question: Is the goal to estimate a population parameter (requires random sampling) or to test the efficacy of an intervention (requires random assignment)?
2. Confounding Variable Identification: What extraneous variables could plausibly influence the response, and how can they be blocked or controlled?
3. Ethical and Logistical Constraints: Is it ethical to randomly assign the factor (e.g., smoking or sleep deprivation cannot be ethically assigned to minors; observational studies must be used)?