15.2 Inferential Statistics: Hypothesis Testing, Alpha Levels, Power, and Effect Sizes

Inferential Statistics: Hypothesis Testing, Alpha Levels, Power, and Effect Sizes

While descriptive statistics summarize observed samples, inferential statistics allow psychologists to draw probabilistic conclusions about broader populations based on sample data. Because researchers rarely have access to entire populations, they must quantify the likelihood that observed sample differences reflect genuine psychological phenomena rather than mere sampling error.

1. Probability, Sampling Distributions, and the Central Limit Theorem

To make valid inferences, researchers rely on theoretical sampling distributions—the probability distribution of a specific statistic (such as the sample mean $\bar{X}$) computed from an infinite number of random samples of size $N$ drawn from the same population.

The Central Limit Theorem (CLT)

The Central Limit Theorem is the foundational mathematical theorem of inferential statistics. It states that for any population with mean $\mu$ and finite standard deviation $\sigma$, as sample size $N$ increases, the sampling distribution of the sample mean approaches a normal distribution, regardless of the shape of the parent population (whether skewed, bimodal, or uniform).

Parent Population (Heavily Skewed)      Sampling Distribution of the Mean
       /\                                             (Normal Bell Curve)
      /  \                                                   /\
     /    \                                                 /  \
    /      \__________                                     /    \
   ────────────────────>                                  ────────
     Raw Scores (X)                                     Sample Means (X̄)
                                                        Approaches Normality as N ≥ 30

Core Properties of the Sampling Distribution of the Mean

1. Mean of the Sampling Distribution (Expected Value): The average of all sample means in the sampling distribution is identically equal to the population mean $\mu$:

$\mu_{\bar{X}} = \mu$

This property establishes that the sample mean $\bar{X}$ is an unbiased estimator of $\mu$.

1. Standard Error of the Mean ($SE$ or $\sigma_{\bar{X}}$): The standard deviation of the sampling distribution of the mean is termed the standard error. It quantifies the expected margin of sampling error between a sample mean and the population mean:

$\sigma_{\bar{X}} = \frac{\sigma}{\sqrt{N}} \quad \text{or estimated as} \quad s_{\bar{X}} = \frac{s}{\sqrt{N}}$

The Inverse Square Root Law

Because the denominator of the standard error formula contains $\sqrt{N}$, reducing the standard error by half requires quadrupling the sample size ($4N$):

• If $N = 25$ and $\sigma = 20$, $SE = \frac{20}{\sqrt{25}} = \frac{20}{5} = 4.0$.
• To reduce $SE$ to $2.0$, sample size must increase to $N = 100$ ($SE = \frac{20}{\sqrt{100}} = \frac{20}{10} = 2.0$).
• To reduce $SE$ to $1.0$, sample size must increase to $N = 400$ ($SE = \frac{20}{\sqrt{400}} = \frac{20}{20} = 1.0$).

This principle illustrates the law of diminishing returns in empirical sample size planning.

2. The Logic of Null Hypothesis Significance Testing (NHST)

Null Hypothesis Significance Testing (NHST) merges the probabilistic decision frameworks of Ronald Fisher, Jerzy Neyman, and Egon Pearson. In this framework, an empirical claim is tested by evaluating the plausibility of its direct opposite.

Hypotheses Formulation

• Null Hypothesis ($H_0$): States that in the population, there is no true difference, no experimental effect, or no relationship between variables. Any observed difference in the sample is attributed entirely to random sampling error:

$H_0: \mu_{\text{treatment}} - \mu_{\text{control}} = 0$

• Alternative Hypothesis ($H_1$ or $H_A$): The experimental or research hypothesis predicting that a genuine difference, effect, or relationship exists in the population:

$H_1: \mu_{\text{treatment}} - \mu_{\text{control}} \neq 0 \quad \text{(non-directional / two-tailed)}$

$H_1: \mu_{\text{treatment}} > \mu_{\text{control}} \quad \text{(directional / one-tailed)}$

The Alpha Level ($\alpha$) and Critical Regions

Before collecting data, the experimenter specifies the significance level ($\alpha$), which represents the maximum probability of committing a Type I error that the researcher is willing to tolerate. By historical convention in psychology, $\alpha$ is typically set at $.05$, $.01$, or $.001$.

• The $\alpha$ level demarcates the critical region (rejection region) in the sampling distribution.
• If the calculated test statistic falls within this critical region, the researcher rejects $H_0$ and concludes the effect is statistically significant.
• If the test statistic falls outside the critical region, the researcher fails to reject $H_0$ (never "accepts $H_0$", because absence of evidence is not evidence of absence).

Deconstructing the p-Value

The $p$-value is the exact probability of obtaining a test statistic at least as extreme as the one observed in the sample, assuming that the null hypothesis is true:

$p = P(\text{Data as or more extreme than observed} \mid H_0 \text{ is True})$

3. Decision Matrix, Error Typology, and Statistical Power

Every statistical decision in NHST is subject to uncertainty, yielding four possible outcomes structured across a $2 \times 2$ truth matrix:

The Fundamental Trade-Off Between Type I and Type II Errors

There is an inherent trade-off between $\alpha$ and $\beta$:

• If an experimenter becomes extremely conservative to prevent false positives and lowers $\alpha$ from $.05$ to $.01$ or $.001$, the critical threshold moves further out into the tail of the distribution.
• This shift widens the non-rejection region, making it harder to reject $H_0$. Consequently, the risk of a false negative (Type II error, $\beta$) directly increases, which depresses statistical power ($1 - \beta$).

Sampling Distribution under H0               Sampling Distribution under H1
             │                                              │
             ▼                                              ▼
          ┌─────┐                                        ┌─────┐
         /   │   \                                      /   │   \
        /    │    \                                    /    │    \
       /     │     \      Critical Boundary (α)       /     │     \
      /      │      \               │                /      │      \
  ───┴───────┼───────┴──────────────┼───────────────┴───────┼───────┴───
             μ0                     │                       μ1
      [ Correct Decision ]          │               [ Statistical Power ]
         Area = 1 - α               │                  Area = 1 - β
                             [ Type I Error α ]
                                    │
                     [ Type II Error β (under H1 left of boundary) ]

Statistical Power ($1 - \beta$)

Statistical power ($1 - \beta$) is the probability of correctly rejecting a false null hypothesis—that is, the probability of detecting a true experimental effect when one genuinely exists in the population. Jacob Cohen established that psychological studies should aim for a minimum statistical power of $.80$ (corresponding to an acceptable Type II error rate of $\beta = .20$).

The Determinants of Statistical Power

Statistical power is governed by six interacting factors:

1. Sample Size ($N$): Increasing $N$ reduces the standard error ($SE = \sigma / \sqrt{N}$), narrowing both sampling distributions and reducing their overlap. This is the primary mechanism researchers use to boost power during study design.
2. Magnitude of the Effect Size: Larger true differences between population means produce greater physical separation between the $H_0$ and $H_1$ sampling distributions, increasing the proportion of the $H_1$ curve that clears the critical boundary.
3. Significance Level ($\alpha$): Setting a more liberal $\alpha$ (e.g., $.05$ instead of $.01$) shifts the critical boundary toward the center of the $H_0$ distribution, increasing the area under $H_1$ to the right of the cutoff and elevating power.
4. Directionality of Test (One-Tailed vs. Two-Tailed): A directional (one-tailed) test places the entire $\alpha$ level into a single tail, moving the critical threshold closer to the distribution center and increasing power for effects in that predicted direction.
5. Measurement Reliability and Error Variance: Reducing extraneous noise and utilizing highly reliable measurement tools shrinks sample variance ($s^2$), narrowing the standard error and boosting power.
6. Experimental Design (Within-Subjects vs. Between-Subjects): Repeated-measures designs partial out individual baseline differences from the error term, substantially increasing power compared to between-subjects designs with identical sample sizes.

4. Directional (One-Tailed) vs. Non-Directional (Two-Tailed) Tests

When formulating statistical hypotheses, researchers choose between non-directional and directional tests based on their theoretical expectations.

Non-Directional (Two-Tailed) Tests

• Hypotheses: $H_0: \mu_1 = \mu_2$ versus $H_1: \mu_1 \neq \mu_2$.
• Critical Region Allocation: The nominal alpha level is split equally between the two extreme tails of the sampling distribution (e.g., for $\alpha = .05$, $\alpha / 2 = .025$ in each tail).
• Critical Values: For a standard normal $z$-test with $\alpha = .05$, the critical cutoffs are $z_{\text{crit}} = \pm 1.96$.
• Methodological Advantage: A two-tailed test detects significant effects regardless of whether the treatment group performs significantly better or significantly worse than the control group. It is the accepted scientific default in psychological research.

Directional (One-Tailed) Tests

• Hypotheses: $H_0: \mu_1 \le \mu_2$ versus $H_1: \mu_1 > \mu_2$ (or vice versa).
• Critical Region Allocation: The entire $\alpha$ level is concentrated into the single tail corresponding to the predicted direction (e.g., all $\alpha = .05$ in the upper tail).
• Critical Values: For a standard normal $z$-test with $\alpha = .05$, the critical cutoff is $z_{\text{crit}} = +1.645$.
• Methodological Consideration: Because $1.645 < 1.96$, a one-tailed test has greater statistical power to detect an effect in the predicted direction. However, if the experimental manipulation produces a massive effect in the opposite direction, the researcher cannot reject the null hypothesis, regardless of how extreme the data are.

5. Effect Size Metrics and Confidence Intervals

A major limitation of Null Hypothesis Significance Testing is that statistical significance ($p < .05$) is not equivalent to practical or theoretical importance. Because the standard error shrinks as sample size grows ($SE = s / \sqrt{N}$), an extraordinarily large sample ($N = 50,000$) can render an utterly trivial difference (e.g., a memory enhancement of $0.05$ words) statistically significant at $p < .001$.

To decouple the magnitude of a psychological phenomenon from sample size, researchers report standardized effect sizes and confidence intervals.

Cohen's d (Standardized Mean Difference)

Developed by Jacob Cohen, $d$ expresses the difference between two sample means in units of pooled standard deviation:

$d = \frac{\bar{X}_1 - \bar{X}_2}{s_{\text{pooled}}}$

$\text{Where } s_{\text{pooled}} = \sqrt{\frac{(n_1 - 1)s_1^2 + (n_2 - 1)s_2^2}{n_1 + n_2 - 2}}$

Unlike the $t$-statistic, Cohen's $d$ is independent of sample size. Cohen established widely recognized interpretive benchmarks for psychological research:

• Small Effect ($d = 0.20$): The difference between means equals $0.2$ standard deviations. The groups overlap by approximately $85\%$. Cohen's example: the difference in mean height between 15- and 16-year-old girls.
• Medium Effect ($d = 0.50$): The difference equals half a standard deviation. Overlap is approximately $67\%$. Noticeable to a trained observer without formal measurement.
• Large Effect ($d = 0.80$): The difference equals $0.8$ standard deviations. Overlap drops to approximately $53\%$. Cohen's example: the difference in mean height between 13- and 18-year-old girls.

Variance-Accounted-For Effect Sizes

Other effect size indices quantify the proportion of total variance in the dependent variable that is explained by the independent variable:

• Coefficient of Determination ($r^2$): In correlational designs, $r^2$ represents the proportion of variance shared between two continuous variables.
• Eta-Squared ($\eta^2$): In Analysis of Variance (ANOVA), $\eta^2$ represents the proportion of total variation attributed to the experimental treatment:

$\eta^2 = \frac{SS_{\text{between}}}{SS_{\text{total}}}$

• Partial Eta-Squared ($\eta_p^2$): In factorial ANOVA, $\eta_p^2$ isolates the effect of interest by removing variance explained by other factors from the denominator:

$\eta_p^2 = \frac{SS_{\text{effect}}}{SS_{\text{effect}} + SS_{\text{error}}}$

Confidence Intervals (CIs)

A confidence interval provides an estimated range of values calculated from sample statistics that is likely to include the unknown population parameter at a specified confidence level (typically $95\%$ or $99\%$):

$95\%\text{ CI} = \bar{X} \pm (t_{\text{critical}} \times s_{\bar{X}})$

• Frequentist Interpretation: A $95\%$ confidence interval does not mean there is a $95\%$ probability that the true population mean lies within that specific interval. Rather, if an experiment were repeated infinitely using identical methods, $95\%$ of the computed intervals would capture the true population parameter $\mu$.
• Relation to Hypothesis Testing: Confidence intervals provide a direct test of the null hypothesis. If a $95\%$ confidence interval for the difference between two means contains zero (e.g., $95\%\text{ CI } [-1.20, +3.40]$), the difference is not statistically significant at $\alpha = .05$. Conversely, if zero falls outside the interval (e.g., $95\%\text{ CI } [+0.85, +4.10]$), the null hypothesis is rejected at $p < .05$.