How to Write Good Survey Questions: A Practical Framework
Samee
Good questionnaires are built one question at a time, and one bad question can sink a decision. Here's a made-up example you've probably lived a version of. A DTC coffee brand is weighing a subscription tier. The insights lead asks customers: “How interested would you be in our new, money-saving subscription with free delivery?” Interest comes back sky high. The tier launches. Three months later, sign-ups are a trickle.
The sample was fine. The question was the problem. “New”, “money-saving” and “free delivery” did the selling, and “how interested” measured curiosity, not willingness to pay. A leading, double-barrelled, vague question produced a confident wrong answer.
Below is a six-check system for survey questions, the PROBES test, that you can run on every item before launch. For the full menu of formats, start with our complete guide to survey question types.
Key Takeaways
- Good questionnaires ask one clear, neutral, specific thing per question, in an order that doesn't steer later answers.
- Run every draft through PROBES: Plain, Rid of slant, One idea, Bounded, Easy to answer honestly, Sequenced.
- Swap agree/disagree statements for rating questions that name the dimension.
- Order alone moves results: Pew recorded a 15 point swing from changing which question came first.
- Never launch cold: cognitive interviews, then a soft launch.
What separates good questionnaires from bad ones?
Good questionnaires are understood the same way by every respondent and are easy to answer honestly. Bad ones make people guess your meaning or nudge them towards the answer you hoped for.
Methodologists describe answering as four stages: understand the question, retrieve memories, form a judgement, then map it onto the options. The model comes from Tourangeau, Rips and Rasinski's The Psychology of Survey Response, and Krosnick and Presser use it in their review of question and questionnaire design.
Every bad question breaks a stage. Jargon breaks comprehension. “In the past” breaks retrieval. A loaded adjective bends judgement. A scale with no fitting option breaks the response. Each check below targets one failure point.
The PROBES test: six checks for every survey question
PROBES stands for Plain, Rid of slant, One idea, Bounded, Easy to answer honestly and Sequenced. A question that passes all six is ready for a pilot.
Each check has a rule, a reason, a red-pen rewrite and a one-minute test.
P: Plain words
Rule: if a 14-year-old couldn't read it aloud without stumbling, rewrite it.
Krosnick and Presser's conventional wisdom starts with simple, familiar words and syntax, and no ambiguous terms or double negatives. Your team's vocabulary (“SKU”, “omnichannel”) isn't your customers'. Confused respondents rarely skip. They guess.
Quick test: read it aloud to someone outside your team. If they ask “what do you mean by…”, you have your answer.
R: Rid of slant
Rule: the question should never reveal what you want to hear.
Small wording changes move big numbers. In Pew Research Center's guide to writing survey questions, 68% favoured military action in Iraq to end Saddam Hussein's rule. Adding “even if it meant that U.S. forces might suffer thousands of casualties” cut support to 43%. AAPOR's best practices for survey research say it plainly: avoid language that pushes respondents towards an answer.
BEFORE: LEADING How much did you love our new eco-friendly packaging?
- Loved it a lot
- Loved it a little
- Didn't notice
Red pen: “love” and “eco-friendly” sell the answer, and there is no negative option at all.
AFTER: NEUTRAL AND BALANCED
What is your overall opinion of our new packaging?
- Very positive
- Somewhat positive
- Neither positive nor negative
- Somewhat negative
- Very negative
Neutral stem, balanced scale, every point labelled.
Quick test: swap in a competitor's name. If it now reads like an attack ad or a pitch, it's slanted.
O: One idea per question
Rule: one question, one concept, one answer.
A double-barrelled question asks about two things but accepts one answer. Pew splits confidence in a president “to handle domestic and foreign policy” into two items, and AAPOR says to ask about one concept at a time. Someone who loves your recipes but hates your prices can't answer a combined question honestly.
Quick test: circle every “and” and “or”. Each one is a suspect. If someone could feel differently about the two halves, split them.
B: Bounded and specific
Rule: pin down the timeframe, the unit and the exact thing you mean.
Vague words wreck retrieval. “Regularly” means weekly to one person and yearly to another. Krosnick and Presser also want options exhaustive and mutually exclusive, so no “18 to 25” next to “25 to 34”. Our guide to demographic survey questions has clean age and income bands.
BEFORE: UNBOUNDED
Do you regularly buy organic food?
- Yes
- No Red pen: no timeframe, no definition of “regularly”, and a yes/no that hides frequency.
AFTER: BOUNDED
In the past 30 days, how many times did you buy organic food?
- None \
- 1 to 2 times
- 3 to 5 times
- 6 or more times
- Not sure
Clear window, countable behaviour, ranges that don't overlap.
Quick test: ask two colleagues what your fuzziest word means. Two different answers means you need a boundary.
E: Easy to answer honestly
Rule: the answer format must fit the question, and no honest answer should feel embarrassing.
Start with format. Our breakdown of open-ended vs closed-ended questions shows when free text beats a list. For rating scales, Krosnick and Presser say 7 points is probably optimal in many cases and reliability is higher when every point has a verbal label; our Likert scale examples cover the 5 vs 7 point choice.
Then two quieter biases. Acquiescence is the pull to agree: Pew finds less educated and less informed respondents agree more, which distorts group comparisons. Social desirability inflates voting, church attendance and giving, and hides things like alcohol use. Pew's fixes: give a face-saving reason and make infrequent behaviour easy to admit. Format matters too: in a 2018 experiment, Pew found forced-choice questions produced estimates 8 points higher on average than select-all lists, and judged them more accurate.
EXPERT PERSPECTIVE
Krosnick and Presser argue that agree/disagree, true/false and yes/no formats invite acquiescence, and that reversing items doesn't fix it. Ask rating questions that show the full dimension (“how satisfied or dissatisfied”) instead.
Paraphrasing Jon A. Krosnick and Stanley Presser, Question and Questionnaire Design, Handbook of Survey Research (2010)
Quick test: would you answer truthfully with your manager watching? If not, soften the stem or add “prefer not to say”.
S: Sequenced
Rule: every question is read in the shadow of the one before it.
A well-worded question can still mislead in the wrong place. Quick test: read it straight after the previous one. Does that plant an idea, a brand or a mood?
Bias spotter: how do you catch biased survey questions?
Most response bias in questionnaire design comes from six repeat offenders. Learn their tells.
HOW MINDPROBE HELPS
Writing from a blank page is where bias creeps in. MindProbe's pre-built templates for NPS, CSAT, brand tracking and concept testing give you a structured starting point, and every question stays fully editable so you can run it through PROBES.
How should you order survey questions?
Go easy to hard and general to specific, group related questions, and put sensitive and demographic items last. Good questionnaires treat order as part of the wording.
In Pew's November 2008 data, 81% said Republican leaders should work with Obama when that question followed one about Democratic leaders, versus 66% when it came first. Pew also asks open questions before related closed ones, because a list shown earlier gets repeated back. AAPOR advises general before specific, sensitive topics later. Use this funnel:
- Screener. Confirm they're in scope.
- Warm-up. One or two easy, relevant questions.
- Unaided, open-ended. “Which brands come to mind?” before any list or logo.
- General attitudes. Category before brand, habits before opinions.
- Specific evaluation. Your product, features, concepts, prices.
- Sensitive topics. Spending, health, circumstances, with an opt-out.
- Demographics last. Short, with “prefer not to say”.
Randomise unordered lists but keep ordinal scales in order. In MindProbe, logic and branching route people past questions that don't apply, so non-users never rate features they've never seen.
Should you let AI draft your survey questions?
Yes, for the first draft. AI writes a starting set fast; good questionnaires still need a human red pen.
MindProbe's question builder drafts questions from your research goal for you to edit, reorder or delete, across 30+ question types, from ranking to close-ended questions. Speed makes it easy to ship unexamined questions, so MindProbe recommends running PROBES on every draft and cutting anything not tied to a decision.
How do you pilot test a questionnaire before launch?
Run cognitive interviews with a few people like your respondents, fix what they misread, then soft launch to a small slice of your sample before sending the rest.
Pew calls pretesting “an essential step”. AAPOR adds: pretest with people like your sample, and pilot the full process. MindProbe recommends this routine:
- Cognitive interviews (5 to 8 people). Have them think aloud. Probe: “What does ‘regularly’ mean to you here?”
- Fix and re-test. Rewrite anything two or more people misread.
- Soft launch to about 5 to 10% of your sample. Check completion time, drop-off, straight-lining and skipped items.
- Mine “Other, please specify”. Repeated write-ins become new options.
- Lock and launch. Freeze wording so waves stay comparable.
HOW MINDPROBE HELPS
Soft launch it in MindProbe: share the survey link with a small slice of your sample, watch the first answers arrive in the real-time dashboard, fix what looks off, then send to everyone else. If you need respondents fast, field the pilot to an on-demand consumer panel.
Pre-launch checklist: 12 checks for a good questionnaire
PRINT ME: PRE-LAUNCH CHECKLIST
☐ 1. Every question maps to a written objective.
☐ 2. Customer words only: no jargon or double negatives.
☐ 3. No emotive adjectives or hinted answers.
☐ 4. Every “and”/“or” checked for double-barrelling.
☐ 5. No assumed behaviour; filters come first.
☐ 6. Every behaviour question has a timeframe.
☐ 7. Options are exhaustive and don't overlap.
☐ 8. Scales are balanced, fully labelled, same direction.
☐ 9. Agree/disagree grids replaced where possible.
☐ 10. Open questions before related lists; lists randomised.
☐ 11. Sensitive and demographic items last, with opt-outs.
☐ 12. Logic tested; cognitive interviews and soft launch done.
Frequently Asked Questions
What makes a good questionnaire?
Good questionnaires ask only what a decision needs, in plain neutral words, one idea per question, with fitting options, ordered general to specific with sensitive items last.
What is a double-barrelled question?
One that asks about two things but allows one answer, like “How satisfied are you with our price and quality?” Split it in two.
How do you avoid bias in survey questions?
Cut selling words and assumptions, prefer rating questions to agree/disagree statements, use face-saving wording for sensitive behaviour, and ask general questions before specific ones.
How many questions should a survey have?
As few as your objectives need. Cut anything not tied to a decision, and watch soft-launch drop-off for survey fatigue.
Where should demographic questions go in a survey?
At the end, unless you screen on them. Keep them short, with non-overlapping ranges and “prefer not to say”.
Draft fast, then red-pen like a pro
Get AI-generated question drafts to edit, 30+ question types and logic and branching, then spend your time on the PROBES checks. Start blank or from a survey template.
References
- Krosnick, J. A. and Presser, S. (2010). Question and Questionnaire Design. In Handbook of Survey Research, 2nd edn (eds Marsden and Wright). Emerald. PDF
- Tourangeau, R., Rips, L. J. and Rasinski, K. (2000). The Psychology of Survey Response. Cambridge University Press. cambridge.org
- Pew Research Center. Writing Survey Questions. pewresearch.org
- Lau, A. and Kennedy, C. (2019). When Online Survey Respondents Only ‘Select Some That Apply’. Pew Research Center. pewresearch.org
- AAPOR. Best Practices for Survey Research. aapor.org