11.4 Designing Data Collection Instruments: Questionnaires, Surveys & Structured Interviews

Choosing the Instrument Before Writing the Questions

Practitioners reach for a survey by reflex. The content outline treats instrument design as a distinct skill because the instrument determines what can be learned, and a well-written survey aimed at the wrong question produces confident, useless data.

The strongest evaluation designs triangulate: operational data establishes what changed, a survey establishes how widely, and interviews establish why.

Writing Items That Do Not Contaminate the Answer

Four defects account for most unusable talent development instruments.

Double-barrelled items ask two things and accept one answer. "The training was relevant and well delivered" cannot be answered by someone who found it relevant and badly delivered. Split every item containing "and" or "or" and check whether both halves could diverge.

Leading items embed the desired answer. "How much did the new coaching programme improve your confidence?" presupposes improvement. The neutral form asks about direction and magnitude without assuming either: "Since the coaching programme, my confidence in handling difficult customer conversations has..." with a scale running from decreased significantly to increased significantly.

Loaded items attach a value judgment to one option. "Do you support the sensible new approval limits?" is not salvageable by scale design.

Recall items beyond reliable memory ask respondents to reconstruct what they cannot. "How many hours of informal learning did you undertake last quarter?" returns a number, but that number is a construction rather than a measurement. Shorten the window or ask about frequency bands.

A fifth, subtler defect is assumed vocabulary. Terms like "capability," "enablement," or "learning journey" mean something specific to the TD function and something vaguer to everyone else. Pilot every instrument with a handful of real respondents and ask them to explain, in their own words, what each item is asking.

Response Scales

• Length. Five to seven points is the working range. Three points loses discrimination; eleven points implies precision respondents do not possess.
• Labelling. Fully labelled scales, where every point carries a word, produce more consistent responses than scales that label only the endpoints, because respondents interpret unlabelled middle points differently from one another.
• Neutral midpoint. Include one only when "neither" is a genuine, meaningful answer. Forcing a choice on a topic where neutrality is real manufactures opinion; offering a midpoint on a topic where it is not invites satisficing.
• Agreement scales are not always right. Agreement formats invite acquiescence bias — the tendency to agree regardless of content. Where possible, ask about frequency (never / rarely / sometimes / often / always) or about a concrete behaviour, both of which are anchored to something observable.
• Consistency. Keep direction and length constant across the instrument. Reversing polarity mid-survey to "catch inattentive respondents" mostly catches careful ones reading quickly.

Sampling and Response Rate

A high response rate on a biased sample is worse than a modest response rate on a representative one, because volume disguises non-response bias rather than correcting it. Before fielding, decide which subgroups must be comparable — region, shift, tenure band, role — and check realized responses against workforce composition afterwards. If night shift is 22% of the population and 4% of respondents, the result describes day shift.

Practical levers on response: keep the instrument short and say honestly how long it takes; explain what was changed after the last survey, which is the single strongest driver of repeat participation; field it inside working time rather than expecting unpaid completion; and never promise anonymity the system cannot deliver.

Structured and Behavioural Interviews

A structured interview uses a fixed question set asked in the same order of every participant, with planned probes and a consistent recording method. The structure is what makes responses comparable; without it, an interviewer's shifting interests produce a set of anecdotes that cannot be aggregated.

Behavioural probing is the core technique. Ask for a specific past instance rather than a general policy: "Tell me about the last time you had to apply the escalation procedure. Walk me through what you did." Then probe for situation, action, and outcome. Generalized questions — "How do you usually handle escalations?" — return the respondent's self-image; specific incident questions return behaviour.

Three disciplines separate a usable interview from a pleasant conversation:

1. Silence after the answer. The first response is the rehearsed one; the substantive material arrives in the second, after a pause the interviewer does not fill.
2. Separate observation from interpretation in notes. Record what was said, not what it meant. Interpretation happens during analysis, across interviews, not in the moment.
3. Stop at saturation, not at a target. When successive interviews stop producing new themes, further interviews add cost without information.

Confidentiality Determines Data Quality

Honest answers depend on credible protection, and credibility is structural rather than rhetorical. State the minimum reporting threshold — results suppressed for any group below a stated headcount — before fielding, not after someone asks. Be explicit about who sees raw responses and verbatim comments, since free-text comments are frequently identifiable even when the numbers are not. And never re-identify a respondent to follow up on a concerning comment, however well-intentioned: one such follow-up ends honest responding in that population for years.

Exam Trap: When a scenario reports a surprising or implausible survey result, the intended answer is usually an instrument or sampling defect — leading wording, a skewed respondent profile, a scale that forced a position — rather than a call for a bigger survey or a different statistical treatment.