{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-06-29T13:58:31.028Z"},"content":[{"type":"documentation","id":"d4443885-953f-4e06-8026-9922ec16ae1b","slug":"questionnaire-design-guide","title":"Questionnaire Design: The Complete Guide to Writing Questions That Get Honest Answers","url":"https://www.koji.so/docs/questionnaire-design-guide","summary":"A research-backed, step-by-step questionnaire design guide: starting from objectives and constructs, choosing question types, writing neutral and answerable questions, constructing response scales (number of points, labeling), ordering for flow, keeping length short, pre-testing, and designing against acquiescence, social-desirability, leading, order, and satisficing biases. Includes how Koji administers a questionnaire conversationally with adaptive follow-ups.","content":"## What is questionnaire design? (Answer first)\n\n**Questionnaire design is the discipline of constructing a set of questions that measures what you actually intend to measure — accurately, without bias, and with as little burden on the respondent as possible.** A questionnaire is the *instrument*; a survey is the broader process of distributing it and analyzing the results. Good design is the difference between data you can trust and data that quietly misleads you, because the way a question is worded, scaled, and ordered changes the answer.\n\nHow much does it change the answer? In a classic Pew Research Center split-sample experiment, simply adding the phrase *\"even if it meant that U.S. forces might suffer thousands of casualties\"* to a question flipped the result from **68% in favor / 25% opposed to 43% in favor / 48% opposed** — a 25-point swing from one clause. Wording is not a detail; it is the measurement.\n\n> **Bottom line:** Every questionnaire is a series of design decisions — objectives, question type, wording, scales, order, and pre-testing — and each one can introduce or remove bias. Below is the full process. And when a questionnaire is the wrong tool entirely, an AI-moderated conversation that adapts to each answer often gets you closer to the truth.\n\n> \"Writing clear and neutral survey questions is much more difficult than it might seem. We spend *a lot* of time thinking about the phrasing and ordering of our survey questions. Paying close attention to these seemingly minor factors makes a huge difference.\" — **Courtney Kennedy**, VP of Methods and Innovation, Pew Research Center\n\n## Step 1 — Start with objectives and constructs, not questions\n\nBefore you write a single question, name the decision the questionnaire will inform and the *constructs* (the abstract things — satisfaction, trust, effort, intent) you need to measure. A common professional practice is to first explore a topic with open-ended questions to learn how people actually talk about it, then convert that language into closed-ended items. Skipping this step is how you end up with twenty questions that don't add up to an answer.\n\n## Step 2 — Choose the right question type\n\n- **Closed-ended** questions (scales, single/multiple choice, yes/no) produce comparable, quantifiable data and are fast to answer.\n- **Open-ended** questions capture unprompted depth and the reasons behind a rating.\n\nNeither is universally better — the strongest questionnaires mix both. Pew has shown the choice matters: when respondents were *offered* \"the economy\" as an option, 58% selected it, versus only 35% who *volunteered* it unprompted. Closed options shape what people say, so use open-ended questions wherever you'd otherwise be guessing at the answer set.\n\n## Step 3 — Write each question to be neutral and answerable\n\nMost measurement error is born here. The rules:\n\n- **No leading or loaded words.** \"Welfare\" pulls different answers than \"assistance to the poor.\" A single adjective can move results 12 points — Pew found 60% said plenty of *\"jobs\"* were available versus 48% for plenty of *\"good jobs\"* (Pew Research Center, 2019).\n- **No double-barreled questions.** As NN/g's Maddie Brown defines it, **\"A double-barreled question asks respondents to answer two things at once.\"** \"How satisfied are you with our price and support?\" can't be answered cleanly — split it. Watch for the word \"and.\"\n- **No jargon, no double negatives, no absolutes** (\"always,\" \"never\").\n- **Make options mutually exclusive and exhaustive** (no overlapping age bands; include \"Other\" or \"Prefer not to say\" where needed).\n- **Ask about recent past behavior, not future predictions.** People are poor forecasters of their own behavior — \"How likely are you to use this feature?\" is far weaker than \"When did you last do X?\" This is NN/g's most distinctive rule.\n\n## Step 4 — Construct response scales deliberately\n\nScale design has real, measured effects on data quality:\n\n- **Number of points.** Reliability and validity are weak for 2–4 point scales, rise toward a sweet spot around 7, and *test–retest reliability declines above 10 points* (Preston & Colman, *Acta Psychologica*, 2000). Respondents in that study most preferred 10-, then 7- and 9-point scales.\n- **Label every point, not just the ends.** A fully labeled 7-point scale reached **.719 reliability versus .506 when only the endpoints were labeled** (Maitland, *Survey Practice*, citing Alwin 2007).\n- **Match the scale to the construct.** Bipolar constructs (satisfaction: dissatisfied↔satisfied) suit a 7-point scale with a neutral midpoint; unipolar constructs (how effective?) suit a 5-point scale from \"not at all\" to \"extremely.\"\n- **Balance the scale** (equal positive and negative options) and prefer item-specific wording over generic agree/disagree, which invites acquiescence bias.\n\n## Step 5 — Order questions for flow\n\nOrder changes answers through context effects. Pew documents that a question placed earlier can shift a later one by 10 points via assimilation or contrast. Best practices:\n\n- **Funnel technique:** broad and easy first, narrow and specific later.\n- **Sensitive questions late**, once you've earned a little trust; **demographics last**.\n- **Watch primacy/recency:** in long option lists, items at the top (visual) or end (audio) get picked more — randomize where appropriate.\n- **Keep wording identical across waves** if you're tracking a trend over time.\n\n## Step 6 — Keep it short\n\nLength is the silent killer of data quality. Across 26,000+ surveys, SurveyMonkey found **10-question surveys averaged 89% completion versus 79% for 40-question surveys**, and that engagement per question roughly halves as a survey drags on — respondents spend ~75 seconds on the first question but under 20 by question 30. An academic study (Sauermann et al., 2018) found a 13-question survey hit **63% completion versus 37% for a 72-question version**. Shorter questionnaires don't just feel kinder; they produce more honest, less satisficed data.\n\n## Step 7 — Pre-test before you launch\n\nPretesting is not optional. Run your draft past 5+ people using **cognitive interviewing / think-aloud** (\"tell me what this question is asking you\") and a **soft-launch pilot** to a small slice of your sample. You will discover ambiguous wording, broken skip logic, and questions that mean something different to respondents than you intended — every time.\n\n## The biases to design against\n\n| Bias | What it does | Design fix |\n|---|---|---|\n| Acquiescence | Tendency to agree | Item-specific scales, not agree/disagree |\n| Social desirability | Over-report \"good\" answers | Self-administered mode, neutral wording, anonymity |\n| Leading/loaded | Wording pushes an answer | Balanced, neutral phrasing |\n| Order effects | Earlier Qs prime later ones | Funnel + randomization |\n| Satisficing/straight-lining | Low-effort answering | Shorter survey, attention checks |\n\nMode matters too: Pew found self-administered (web) answers differ from interviewer-administered ones by about **5 percentage points on average** across 60 questions, largely due to social-desirability pressure.\n\n## When a questionnaire is the wrong tool — the modern alternative\n\nHere is the uncomfortable truth a static questionnaire can't escape: it asks the same fixed questions of everyone and can never follow up. The most interesting answer — the \"it depends,\" the unexpected workaround — slips through because there's no one there to ask \"why?\"\n\nThis is where AI-native research changes the economics. With Koji, you design the instrument once and an **AI interviewer administers it conversationally** — reading questions in a natural voice or text, and probing each answer with 1–3 adaptive follow-up questions exactly where a human researcher would. You still get the structured, quantifiable data of a questionnaire, plus the depth of an interview, without manually running hundreds of calls. Teams adopting AI-assisted research consistently report far faster time-to-insight than the design-distribute-wait-export cycle of legacy survey tools.\n\nKoji's [structured questions](/docs/structured-questions-guide) give you six instrument types in one study — **open_ended, scale, single_choice, multiple_choice, ranking, and yes_no** — each with the design properties above baked in (configurable scale points and labels, mutually exclusive options, optional \"Other\"). Because every question carries a stable ID, scale distributions and open-ended themes are aggregated together automatically in the report, so you don't export a CSV and start over in a spreadsheet.\n\n## Before and after: fixing three flawed questions\n\nThe fastest way to internalize these rules is to see them applied.\n\n**1. The double-barreled question**\n- ❌ \"How satisfied are you with the speed and reliability of the app?\"\n- ✅ Split into two: \"How satisfied are you with the app's speed?\" and \"How satisfied are you with the app's reliability?\" — because a respondent can love one and hate the other.\n\n**2. The leading question**\n- ❌ \"How much did you enjoy our award-winning onboarding experience?\"\n- ✅ \"How would you describe your onboarding experience?\" — the \"award-winning\" framing and the assumption of enjoyment both push the answer upward.\n\n**3. The vague, unanswerable scale**\n- ❌ \"Rate our service: 1–10.\" (1 = what? 10 = what?)\n- ✅ \"How would you rate our support team's helpfulness?\" on a labeled 5-point scale from \"Not at all helpful\" to \"Extremely helpful\" — labeled, item-specific, and matched to a unipolar construct.\n\n### A pre-launch checklist\n\nBefore you send any questionnaire, confirm: every question maps to a research objective; no question is leading, loaded, or double-barreled; answer options are mutually exclusive and exhaustive; scales are labeled and balanced; the order funnels broad to narrow with sensitive items late; the whole thing takes under ten minutes; and you've piloted it with at least five people. If a question doesn't earn its place against a decision you need to make, cut it — every question you remove raises the quality of the answers to the ones that remain.\n\n## Related Resources\n\n- [Survey Design Best Practices](/docs/survey-design-best-practices)\n- [Survey Question Types Explained](/docs/survey-question-types)\n- [Survey Question Wording Guide](/docs/survey-question-wording-guide)\n- [Likert Scale Research Guide](/docs/likert-scale-research-guide)\n- [Avoiding Leading Questions](/docs/avoiding-leading-questions)\n- [Open-Ended vs. Closed-Ended Questions](/docs/open-ended-vs-closed-ended-questions)\n- [Structured Questions Guide](/docs/structured-questions-guide)","category":"Research Methods","lastModified":"2026-06-28T03:28:00.721305+00:00","metaTitle":"Questionnaire Design: How to Write Questions That Get Honest Answers","metaDescription":"A complete questionnaire design guide: define your constructs, write neutral questions, choose response scales, order for flow, keep it short, pre-test, and design against the biases that ruin survey data — with research citations.","keywords":["questionnaire design","how to design a questionnaire","survey question design","questionnaire vs survey","response scales","leading questions","double-barreled questions","pre-testing surveys","question order bias","number of scale points"],"aiSummary":"A research-backed, step-by-step questionnaire design guide: starting from objectives and constructs, choosing question types, writing neutral and answerable questions, constructing response scales (number of points, labeling), ordering for flow, keeping length short, pre-testing, and designing against acquiescence, social-desirability, leading, order, and satisficing biases. Includes how Koji administers a questionnaire conversationally with adaptive follow-ups.","aiPrerequisites":["A research goal or decision the questionnaire will inform","A target audience to survey","Basic familiarity with survey concepts"],"aiLearningOutcomes":["Translate research objectives into measurable constructs","Write neutral, answerable questions free of common wording errors","Choose and label response scales that maximize reliability","Order questions to minimize context and order-effect bias","Pre-test a questionnaire and design against the major response biases"],"aiDifficulty":"beginner","aiEstimatedTime":"13 min read"}],"pagination":{"total":1,"returned":1,"offset":0}}