Data Collection Instruments, Validated Methods, and Survey Design

Quick Answer: Validated instruments are measurement tools with documented evidence for reliability and validity in a target population; the CPH exam tests your ability to identify them (SF-36, PHQ-9, BRFSS modules, AUDIT) and to design new data collection tools that follow sound question-writing, sampling, and pretesting practices.

Validity and Reliability Foundations

Before using any instrument, confirm two properties. Validity is whether the instrument measures what it claims to measure: face validity (looks right to experts), content validity (covers the construct's domains), construct validity (correlates with related measures as expected), and criterion validity (agrees with a gold standard, expressed as sensitivity and specificity for screening tools). Reliability is consistency of measurement: test-retest reliability (stable answers on repeat administration), inter-rater reliability (agreement between observers, often measured by kappa), and internal consistency (items in a scale move together, measured by Cronbach's alpha, with 0.70 or above generally acceptable for group-level use).

Widely Used Validated Instruments

The exam names specific instruments you should recognize:

• SF-36 and SF-12: generic health-related quality-of-life scales.
• PHQ-9: nine-item depression screen with established cut points (a score of 10 or above suggests moderate depression).
• GAD-7: seven-item generalized anxiety screen.
• AUDIT and AUDIT-C: World Health Organization alcohol use disorder screens.
• Beck Depression Inventory (BDI): 21-item self-report depression severity scale.
• USDA Food Security Module: household food insecurity scale used in federal nutrition monitoring.
• BRFSS optional modules: state-selected validated question sets on topics such as adverse childhood experiences, cognitive decline, and food insecurity.

Using a validated instrument is preferable to writing your own because the psychometric work is already done, results are comparable across studies, and reviewers trust established tools. The trade-off is that the instrument may not fit your construct exactly, and translated or cultural adaptations require their own validation evidence in the new population before cut points can be applied.

Designing a New Data Collection Tool

When no validated instrument fits, you design your own. The process: define the construct, draft items in plain language at or below a 6th-grade reading level, avoid double-barreled questions ("Do you smoke and drink?"), avoid leading or loaded wording, provide mutually exclusive and exhaustive response options, order sensitive items last, and pretest with cognitive interviewing to confirm respondents interpret items as intended.

Survey Sampling Essentials

A survey is only as good as its sampling frame. Probability samples (simple random, stratified, cluster, systematic) allow statistical inference to the target population. Non-probability samples (convenience, purposive, snowball) are acceptable for formative or qualitative work but cannot generalize without strong assumptions. Sample size is driven by the desired precision (margin of error), the expected prevalence, and the design effect from clustering and weighting. Non-response bias is addressed through weighting, follow-up contacts, and reporting the response rate using AAPOR standards.

Mode of Administration

Mode shapes who responds and how. Telephone surveys (BRFSS) reach broad populations but exclude households without phones and are increasingly affected by cell-phone-only households. Web surveys are cheap and fast but skew toward the digitally connected. In-person surveys achieve the highest response rates and allow physical examination (NHANES) but are costly. Mail surveys work for address-based frames. Mixed-mode designs combine modes to improve coverage while managing mode effects on responses, and the analysis must adjust for mode to avoid biased estimates.

Worked Scenario

A local health department needs to assess depression among Spanish-speaking agricultural workers. The team selects the PHQ-9, which has a validated Spanish translation, and administers it in person during a farmworker health fair. Because the fair crowd is a convenience sample, results describe attendees, not the full worker population; the report must state that limitation. If the department wanted county-level estimates, it would need a probability frame such as employer rosters stratified by crop type, or a household survey in agricultural census tracts. The validated instrument cannot rescue an unrepresentative sample.

Exam Traps

• Treating "validated" as a marketing label. A single published study in one population does not generalize the instrument to all populations; cross-cultural validity must be demonstrated.
• Confusing sensitivity and specificity with reliability. Screening test metrics describe criterion validity against a gold standard, not consistency of repeated measurement.
• Assuming longer instruments are more valid. Adding items can lower response rates and does not automatically improve measurement.
• Forgetting that mode changes answers. The same question asked by phone versus self-administered web can yield different distributions due to social desirability.
• Treating Cronbach's alpha as a measure of validity. Alpha reflects internal consistency only; a scale can be highly consistent and still measure the wrong construct.