11.2 Sampling Variability, Simulation, and Margin of Error

11.2 Sampling Variability, Simulation, and Margin of Error

Quick Answer: A parameter (such as population mean $\mu$ or population proportion $p$) is a fixed numerical characteristic of an entire population, while a statistic (such as sample mean $\bar{x}$ or sample proportion $\hat{p}$) is calculated from sample data and varies from sample to sample—a phenomenon called sampling variability. In simulation-based inference, computer models simulate hundreds of random trials under a null hypothesis. For an approximately normal simulation distribution, the Margin of Error at the $95%$ confidence level is approximately two standard errors ($ME \approx 2 \cdot SE$), establishing a $95%$ confidence interval: $[\text{statistic} - 2\cdot SE, , \text{statistic} + 2\cdot SE]$. If an observed experimental result falls outside this interval, the result is statistically significant.

---

1. Parameters vs. Statistics & Sampling Variability (S-IC.4)

A foundational concept in inferential statistics is the distinction between population parameters and sample statistics:

• Population Parameter: A fixed, usually unknown numerical value summarizing an entire population. Parameters are conventionally symbolized using Greek letters (e.g., $\mu$ for population mean, $\sigma$ for population standard deviation, or $p$ for population proportion).
• Sample Statistic: A numerical value computed directly from an observed random sample. Statistics are symbolized using Latin letters (e.g., $\bar{x}$ for sample mean, $s$ for sample standard deviation, or $\hat{p}$ for sample proportion). Statistics serve as point estimators for unknown population parameters.

Parameter vs. Statistic Reference Table

Sampling Variability

If five independent researchers each draw a random sample of $n = 100$ students from a university of $20,000$ to estimate the proportion of students who commute, their sample proportions might be $0.38, 0.43, 0.41, 0.37,$ and $0.42$. None of these statistics is necessarily incorrect. Rather, this natural fluctuation among sample statistics obtained from different random samples drawn from the same population is known as sampling variability.

---

2. Sampling Distributions and Standard Error

Imagine taking every possible random sample of size $n$ from a population, calculating the statistic (e.g., $\bar{x}$ or $\hat{p}$) for each sample, and plotting all those values on a dotplot. The resulting distribution is the sampling distribution of the statistic.

                               Sampling Distribution
                                   (Mean = μ or p)
                                         ▲
                                        /│\
                                       / │ \
                                      /  │  \
                                     /   │   \
                         ───────────/────┼────\───────────
                                    -2SE │ +2SE
                                         │
                                ◄────── 95% ──────►

• Center of the Sampling Distribution: For random samples, the mean of all possible sample statistics equals the true population parameter. We say the statistic is an unbiased estimator of the parameter ($E(\bar{x}) = \mu$ and $E(\hat{p}) = p$).
• Standard Error ($SE$): The standard deviation of a sampling distribution is formally termed the standard error. It measures the typical distance that a sample statistic deviates from the true population parameter due to chance variation.
• The Effect of Sample Size: Increasing sample size decreases sampling variability. Standard error is inversely proportional to the square root of $n$ ($SE \propto \frac{1}{\sqrt{n}}$). Quadrupling the sample size ($n \to 4n$) cuts the standard error in half ($\frac{1}{\sqrt{4}} = \frac{1}{2}$), producing a tighter, more precise sampling distribution.

---

3. Simulation-Based Inference and Null Distributions (S-IC.2)

In Algebra II, rather than relying on theoretical calculus-based probability formulas, students use simulation-based inference. Computer or calculator algorithms simulate random sampling or random assignment hundreds of times under an assumed baseline condition called the null hypothesis ($H_0$).

Why Simulate?

1. Establish Baseline Chance: Simulation demonstrates what outcomes occur strictly through random chance.
2. Empirical Distribution: By repeating a random process (e.g., 200, 500, or 1,000 trials), the computer produces an empirical distribution of simulated statistics.
3. Assess Plausibility: By comparing an observed experimental result against this simulated distribution, researchers can visually and mathematically assess whether the observed result is plausible under chance alone, or whether it is so extreme that chance can be rejected.

---

4. Margin of Error and the 95% Confidence Interval (AII-S.IC.4)

[!NOTE] NYSED reworded this standard for Algebra II as: given a simulation model based on a sample proportion or mean, construct the 95% interval centered on the statistic (plus or minus two standard deviations) and determine if a suggested parameter is plausible. Every element of that sentence is a scoring point - the interval must be centered on the statistic, the multiplier is two, and the verdict must name the suggested parameter.

Because of sampling variability, a sample statistic $\bar{x}$ or $\hat{p}$ will rarely equal the exact population parameter. To provide an honest estimate, statisticians report a confidence interval, which attaches a margin of error to the point estimate.

The Two-Standard-Error Rule for Margin of Error

For mound-shaped, symmetric, approximately normal sampling distributions, the Empirical Rule dictates that approximately $95.4%$ (conventionally rounded to $95%$) of all sample statistics fall within two standard deviations of the center.

On the Regents Examination in Algebra II, the Margin of Error ($ME$) at a $95%$ confidence level is defined as:

$ME \approx 2 \cdot SE \quad \text{or} \quad ME \approx 2 \cdot s_{\text{sim}}$

Where $s_{\text{sim}}$ is the standard deviation of the simulated sampling distribution.

Constructing the 95% Confidence Interval

A confidence interval is centered at the observed sample statistic:

$\text{Confidence Interval} = \text{Statistic} \pm ME = [\text{Statistic} - 2\cdot SE, \; \text{Statistic} + 2\cdot SE]$

• For a Population Mean $\mu$: $[\bar{x} - 2 \cdot SE, \; \bar{x} + 2 \cdot SE]$
• For a Population Proportion $p$: $[\hat{p} - 2 \cdot SE, \; \hat{p} + 2 \cdot SE]$

Interpreting a 95% Confidence Interval

A correct Regents interpretation must focus on the parameter: "We are $95%$ confident that the true population mean (or proportion) lies within the interval from [lower bound] to [upper bound]."

[!CAUTION] The Individual Data Trap: A confidence interval does not indicate that $95%$ of individual values in the population fall within the interval. It estimates the location of the single, fixed population parameter (mean or proportion).

---

5. Evaluating an Observed Result Against a Simulation (AII-S.IC.2)

[!NOTE] Standard note. S-IC.5, the old "use data from a randomized experiment to compare two treatments" standard, was removed from Algebra II under the Next Generation standards. What remains, and what is assessed, is AII-S.IC.2, reworded by NYSED as: determine if a value for a sample proportion or sample mean is likely to occur based on a given simulation. NYSED fixes the decision rule for this course: "if the statistic falls within two standard deviations of the mean (95% interval centered on the population parameter), then the statistic is considered likely (plausible, usual)." The two-treatment scenario below is simply a familiar context for applying that one rule - you will be handed a simulation distribution and asked whether an observed value is plausible.

Regents constructed-response items frequently present a randomized experiment comparing two treatments and ask whether the difference between treatment means is statistically significant.

The Re-Randomization Simulation Protocol

Consider an experiment where Group A receives a new study technique and achieves a mean score of $\bar{x}_A = 86$, while Group B studies conventionally and achieves $\bar{x}_B = 80$. The observed difference is:

$\text{Observed Difference} = \bar{x}_A - \bar{x}_B = 86 - 80 = 6.0$

Is this 6-point advantage caused by the new technique, or did Group A just happen to receive stronger students by the luck of random assignment?

To decide, a computer runs a re-randomization simulation:

1. Null Hypothesis ($H_0$): Assume the technique has no effect; every student would have scored the exact same number regardless of group assignment.
2. Shuffle and Redistribute: Pool all test scores together, shuffle them randomly, and arbitrarily divide them into two simulated groups of the original sizes.
3. Calculate Simulated Difference: Compute $\bar{x}{\text{sim A}} - \bar{x}{\text{sim B}}$.
4. Repeat: Repeat this process hundreds of times (e.g., 250 trials) to construct a simulation distribution centered near $0$.

                          Simulation Distribution of Differences
                                (Centered at ~0 under H0)
                                         ▲
                                        /│\
                                       / │ \
                                      /  │  \
                                     /   │   \
                      ──────────────/────┼────\───────────┼──────►
                                    -2SD │ +2SD         Observed
                                         │             Difference
                                ◄── Plausible (95%) ──► (Significant!)

Decision Rule for Statistical Significance

Calculate the $95%$ interval of chance variation for the simulation distribution:

$[\text{Mean}_{\text{sim}} - 2 \cdot SD_{\text{sim}}, \; \text{Mean}_{\text{sim}} + 2 \cdot SD_{\text{sim}}]$

• Case 1: Observed Difference Falls WITHIN the 95% Interval
  • The observed difference is plausible under chance alone.
  • The result is not statistically significant.
  • Conclusion: There is insufficient evidence to conclude the treatment had an effect.
• Case 2: Observed Difference Falls OUTSIDE the 95% Interval
  • An outcome this extreme occurs less than $5%$ of the time strictly by chance.
  • The result is statistically significant.
  • Conclusion: Random assignment alone is an implausible explanation; the treatment caused the observed difference.

---

6. Worked Problems

Worked Problem 1: Survey Margin of Error & Evaluating a Stated Claim

Problem: A random sample of $250$ municipal residents is surveyed regarding whether they support constructing a new community bike trail. In the sample, $160$ residents respond in favor (sample proportion $\hat{p} = \frac{160}{250} = 0.64$). A computer runs $500$ simulated samples of size $250$ assuming a population proportion of $0.64$, producing a simulated sampling distribution with a mean of $0.640$ and a standard deviation of $0.030$.

1. Algebraically determine the margin of error and state the $95%$ confidence interval for the proportion of all residents who support the bike trail.
2. A city council member claims that only $55%$ ($0.55$) of all residents support the project. Based on your confidence interval, evaluate whether the council member's claim is plausible. Justify your answer.

• Step 1: Calculate the margin of error. Using the two-standard-error rule: $ME = 2 \cdot s_{\text{sim}} = 2(0.030) = 0.060$

• Step 2: Construct the 95% confidence interval. $\text{Confidence Interval} = \hat{p} \pm ME = [0.640 - 0.060, \; 0.640 + 0.060] = [0.580, \; 0.700]$ We are $95%$ confident that the true proportion of all municipal residents supporting the bike trail is between $0.580$ and $0.700$ (or $58%$ and $70%$).

• Step 3: Evaluate the council member's claim. The council member claims the true proportion is $p = 0.55$. Because $0.55$ falls outside (strictly below) the $95%$ confidence interval of $[0.580, 0.700]$, this claim is not plausible. The sample data provide strong evidence that the true community support exceeds $55%$.

Worked Problem 2: Evaluating Experimental Significance via Re-Randomization

Problem: An agricultural scientist tests whether an organic soil additive enhances the height of hydroponic lettuce. Thirty seedlings are randomly divided into two groups of $15$. Group 1 receives the additive and achieves a mean height of $\bar{x}_1 = 21.6 \text{ cm}$. Group 2 receives no additive and achieves a mean height of $\bar{x}_2 = 18.2 \text{ cm}$. The observed difference in treatment means is:

$\text{Observed Difference} = 21.6 - 18.2 = 3.4 \text{ cm}$

To determine if the 3.4 cm increase is statistically significant, the scientist performs $200$ re-randomization simulations. The resulting distribution of simulated differences has a mean of $0.05 \text{ cm}$ and a standard deviation of $1.20 \text{ cm}$.

1. Determine the $95%$ interval of chance variation for the simulated differences.
2. Determine whether the soil additive had a statistically significant effect on lettuce growth. Justify your response.

• Step 1: Compute the 95% simulation interval. $\text{Interval} = \text{Mean}_{\text{sim}} \pm 2 \cdot SD_{\text{sim}} = 0.05 \pm 2(1.20) = 0.05 \pm 2.40$ $\text{Interval} = [0.05 - 2.40, \; 0.05 + 2.40] = [-2.35, \; 2.45]$

• Step 2: Compare the observed difference to the interval. The observed difference between treatment means is $3.4 \text{ cm}$. $3.4 > 2.45$ Because $3.4 \text{ cm}$ falls outside the $95%$ interval of chance variation ($[-2.35, 2.45]$), an observed difference of this magnitude is extremely unlikely to occur by random assignment alone. Therefore, the effect of the soil additive is statistically significant, supporting the conclusion that the additive caused the enhanced plant growth.

---

7. Common Regents Pitfalls & Exam Strategies

• Pitfall 1: Forgetting to Multiply the Standard Error by 2. A frequent error on constructed responses is reporting the margin of error as $1 \cdot s_{\text{sim}}$ rather than $2 \cdot s_{\text{sim}}$. The $95%$ confidence standard mandates multiplying by $2$.
• Pitfall 2: Confusing Sample Size with Number of Simulations. If a simulation draws $1,000$ samples of size $n = 50$, the sample size governing the standard error is $n = 50$. The number $1,000$ merely represents how many dots appear on the simulated sampling distribution plot.
• Pitfall 3: Incomplete Written Justifications on Regents Exams. If asked whether a value is plausible or significant, you must explicitly state both the calculated interval bounds and the location of the test value relative to those bounds (e.g., "Because $3.4$ is outside the interval $[-2.35, 2.45]$, the difference is statistically significant"). Merely writing "yes" or "no" yields zero credit.