5.4 Data Collection and Management

Start With the Question, Not the Tool

Data collection begins with the evaluation question, not the easiest instrument. A survey suits knowledge and attitudes; observation suits skills and fidelity; administrative records efficiently capture attendance, referrals, immunization status, or service use; interviews and focus groups explain why a program did or did not fit participants. Triangulating two or more methods strengthens confidence when any single source is weak.

Survey Quality

Surveys reach many people and produce comparable responses, but they require careful wording, reading-level review, translation checks, accessible formats, and pilot testing. Avoid these recurring flaws:

• Leading questions that suggest a preferred answer.
• Double-barreled items that ask two things at once ("How satisfied and active were you?").
• Vague recall periods ("usually" instead of "in the past 7 days").
• Response options that do not fit the population or omit "prefer not to answer."

Observation and Existing Data

A structured observation checklist with trained, calibrated observers beats casual impressions for assessing skills or implementation fidelity. Existing data (clinic records, school logs, referral systems, public surveillance) can reduce burden, but they may carry missing fields, inconsistent definitions, delayed availability, or access restrictions, so confirm quality before relying on them.

Data Management

A basic data-management plan protects accuracy and people. It specifies who collects data, where files live, how identifiers are separated from responses (for example, a code key stored separately), how paper forms are locked, how electronic files are encrypted and password protected, how errors are checked through double entry or range checks, and when records are destroyed. De-identification is critical when reporting small groups in which individuals could be recognized.

Missing Data

Handle missing data honestly. Ignoring it biases results when missingness is patterned: people with lower literacy may skip written items, and people with transportation barriers may miss posttests. The best CHES response is to document the missingness, examine its pattern, improve collection procedures, and avoid overstating conclusions, never to silently delete records or fabricate values.

Sampling Basics

Who provides the data shapes what the data can claim. Probability sampling gives every member of the population a known chance of selection and supports generalization: simple random, systematic (every kth person), stratified (sample within subgroups to guarantee representation), and cluster (sample whole groups such as classrooms). Nonprobability sampling is faster but limits generalization: convenience (whoever is available), purposive (chosen for a characteristic), quota, and snowball (referrals from participants).

Most community health evaluations use convenience or purposive samples, which is acceptable as long as the report does not overclaim generalizability. The sampling frame is the actual list from which people are drawn; if it omits part of the population, such as households without phones, the resulting coverage bias cannot be fixed by a larger sample.

Quantitative vs Qualitative Methods

Match the method to the question, not the habit. Quantitative methods (closed-ended surveys, structured observation counts, record extraction) answer how many, how often, and how much, and they produce comparable numbers. Qualitative methods (open interviews, focus groups, open-ended responses) answer why and how, surfacing context, barriers, and meaning. A mixed-methods design uses both, for example a survey to size a problem and interviews to explain it.

Exam items that ask why attendance fell or how participants experienced a program point to qualitative collection; items that ask what percentage improved point to quantitative collection.

Workflow and Neutral Collectors

A strong plan anticipates workflow: when participants complete forms, who answers questions, where completed forms go, and what happens when someone arrives late. In community settings these small details affect response rates and confidentiality. Train data collectors to respond neutrally: explaining instructions is appropriate, but coaching participants toward a preferred answer is not, because it inflates social-desirability bias, especially when participants personally know the program staff.

Where possible, separate the evaluator role from the facilitator role, or collect responses anonymously, so participants do not feel they are grading a person who taught them.

Response Rate and Nonresponse

The response rate is the share of invited people who actually provide data, and a low rate threatens the credibility of any survey. The deeper risk is nonresponse bias: if the people who skip the posttest differ systematically from those who complete it, the results misrepresent the group. Practical boosters include short instruments, multiple reminders, accessible formats and languages, completing forms during a session rather than mailing them home, and modest incentives where allowed.

The exam may show a 30% response rate and ask the best interpretation; the strongest answer notes the limited generalizability and the need to compare responders with the full enrolled group rather than declaring the program a success or failure outright.

Triangulation Strengthens Confidence

No single source is perfect, so evaluators triangulate, combining methods, data sources, or evaluators to cross-check findings. If attendance logs, a participant survey, and facilitator observation all point to weak dose received, the conclusion is far stronger than any one source alone. If they disagree, the discrepancy itself is useful information that prompts a closer look at data quality. When an item offers a choice between relying on one convenient source and corroborating across several, the corroborated, triangulated approach is the defensible answer, balanced against participant burden and resources.