7.4 Validity Threats and Causal Inference
Key Takeaways
- Internal validity concerns whether the observed effect can be attributed to the intervention or independent variable.
- External validity concerns whether findings generalize across people, settings, times, and procedures.
- Construct validity concerns whether the operational definition captures the intended psychological construct.
- Statistical conclusion validity concerns whether the statistical inference is accurate given power, assumptions, and error.
Identify Which Inference Is Under Attack
Validity in research is not one issue. A study can have strong internal validity and weak external validity, or strong statistical analysis and weak construct measurement. EPPP questions often describe a flaw and ask for the name of the threat. The fastest path is to ask which inference the flaw damages: causation, generalization, construct meaning, or statistical conclusion.
Internal validity concerns whether the independent variable caused the observed effect. Threats include history, maturation, testing, instrumentation, regression to the mean, selection, attrition, and diffusion of treatment. For example, if a clinic introduces a new intervention at the same time a major community event changes client stress, history is a plausible alternative explanation.
| Threat family | What it weakens | Example cue |
|---|---|---|
| History | Causal inference | An outside event occurs between pretest and posttest. |
| Maturation | Causal inference | Participants change naturally over time. |
| Testing | Causal inference | Taking the pretest changes posttest performance. |
| Instrumentation | Measurement comparability | The measure, rater, or scoring procedure changes during the study. |
| Attrition | Group comparability | Participants drop out unevenly across conditions. |
| External validity | Generalization | The sample or setting differs from the intended population. |
| Construct validity | Meaning of variables | The operational definition does not capture the target construct. |
Regression to the mean appears when extreme scores tend to move closer to the average on retesting. This is especially important when participants are selected because they have extreme initial scores. If a program enrolls only people with very high distress, some improvement may occur because unusually high scores often become less extreme over time, even without a strong treatment effect.
Selection threats occur when groups differ before the intervention. This is common in quasi-experimental designs using intact groups. If one clinic uses a new treatment and another clinic uses usual care, differences between clinics may explain outcome differences. Matching, statistical control, and careful comparison can help, but they are not the same as random assignment.
External validity asks whether findings travel. A therapy outcome study conducted with highly selected adults in a university clinic may not generalize to adolescents, older adults, rural clients, court-referred clients, or clients with complex medical comorbidity. The correct EPPP answer may preserve the finding while limiting the population to which it applies.
Construct validity asks whether the study measured or manipulated what it claimed. If a study defines social support only as number of social media contacts, the operational definition may miss quality, availability, reciprocity, and perceived support. The problem is not just statistics; the construct itself has been narrowed poorly.
Statistical conclusion validity concerns whether the data analysis supports the inference. Low power can miss a real effect. Violated assumptions can distort results. Multiple unplanned tests can increase false positives. Outliers can change results. The exam may ask which issue is present, but it may also ask how to improve the study. The best answer fixes the threat directly rather than adding unrelated features.
When validity options look similar, name the inference in your head. If the problem is whether the treatment caused change, think internal validity. If the problem is whether the result applies elsewhere, think external validity. If the problem is whether the variable represents the construct, think construct validity. If the problem is whether the statistical test supports the conclusion, think statistical conclusion validity.
A study begins just before a natural disaster, and participant anxiety changes by posttest. Which threat is most likely?
A treatment study uses only highly selected university clinic clients, and the question asks whether results apply to rural community clinics. What validity issue is most central?
What is regression to the mean?