{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-11-08T22:05:30.587Z"},"content":[{"type":"documentation","id":"275da9bf-daee-410e-8a01-a7f5ed312ceb","slug":"select-all-that-apply-vs-pick-top-3","title":"Select All That Apply or Pick Your Top 3? How to Cap a Multi-Select Question","url":"https://www.koji.so/docs/select-all-that-apply-vs-pick-top-3","summary":"Use select all that apply only for a rough list, since it under-counts: in a 2019 Pew experiment, yes/no questions per item gave estimates about 8 points higher and showed no order effect, against 3 points for select-all. Ask yes/no per item when you need prevalence, and a capped pick (exactly 3, shuffled, with a why follow-up) when you need priority. The three formats measure different things and can't be compared.","content":"# Select All That Apply or Pick Your Top 3? How to Cap a Multi-Select Question\n\n**Short answer:** use \"select all that apply\" only when you want a rough list of everything people recognise, and expect it to under-count. When you need to know how many people have or do each thing, ask a yes/no question per item instead. When you need to know what matters most, cap the question (\"pick up to 3\", or \"pick exactly 3\") and ask why straight after. The three formats measure different things, so their numbers can't be compared with each other.\n\n**Try it this week:** in a Koji study, add a multiple choice question, open its pick-limit menu and choose **Exact number** (3) or **Range** (1 to 3), set the option order to **Randomize except last 1** so \"None of these\" stays at the bottom, and keep one follow-up so every person explains their top pick. The rest of this guide explains when each choice is the right one.\n\n## Three formats, three different measures\n\nA multi-select question looks like one tool. It's really three, and each answers a different question.\n\n| Format | What it measures | Typical wording | Use it when |\n|---|---|---|---|\n| Select all that apply | Endorsement: what comes to mind and seems true at a glance | \"Which of these tools do you use? Select all that apply.\" | You want a quick, rough map of a long list and can live with under-counting |\n| Yes/no per item (forced choice) | Prevalence: how many people have, do or want each item | \"Do you use Slack for this? Yes / No\" for each item | The share for each item is the number you'll report |\n| Capped pick (top 3, up to 3) | Priority: what matters most when people have to trade off | \"Which 3 of these matter most to you?\" | You need a ranking you can act on, like a roadmap or a message test |\n\n![Table comparing three multi-select formats: select all measures endorsement, yes/no per item measures prevalence, a capped top 3 measures priority.](/visuals/multi-select-question-formats-compared/multi-select-question-formats-compared.png \"Pick the format by the number you need, not by habit.\")\n\nThe mistake most teams make is using the first format to answer the second or third question. A select-all list where every option scores between 40% and 70% tells you very little about what to build first.\n\n## What the research says about select-all lists\n\n### They under-count\n\nThe best evidence comes from survey methodologists who ran the same items in both formats, side by side.\n\n- In a Pew Research Center experiment with 4,581 U.S. adults on its American Trends Panel (fielded July 30 to August 12, 2018), estimates of personal experiences averaged **8 percentage points higher** when each item was asked as its own yes/no question than when the same items sat in a select-all checklist (Pew Research Center, \"When Online Survey Respondents Only 'Select Some That Apply'\", 2019).\n- The gaps on single items were large. Being denied health insurance came out at **19% with forced choice and 13% with select-all**. Losing a job came out at **63% versus 51%** (same Pew report).\n- Smyth, Dillman, Christian and Stern ran 16 experimental comparisons across two web surveys and a paper survey in 2002 and 2003. In their results, people who answered yes/no for each item **endorsed more options and spent more time** answering than people given a check-all list (Public Opinion Quarterly, 2006).\n\nThe authors explain the mechanism plainly: the forced-choice format \"asks respondents to provide an answer (yes or no) for each response option, a task that should encourage respondents to consider and come to a judgment about each item individually.\" A blank box in a select-all list can mean \"no\", \"didn't read it\" or \"couldn't be bothered\", and you can't tell which.\n\n![Bar chart from Pew Research Center comparing select-all and forced-choice answers: health insurance denial 13% vs 19%, job loss 51% vs 63%.](/visuals/select-all-vs-forced-choice-pew-results/select-all-vs-forced-choice-pew-results.png \"Same items, same survey, a different format, and the shares move by up to 12 points. · Source: Pew Research Center, select-all vs forced-choice experiment, 2019\")\n\n### They are more sensitive to option order\n\nPew also looked across 21 of its online surveys and found an average order effect of **3 percentage points** for select-all questions and **0 points** for forced-choice questions. People skimming a checklist tick what they read first. With 300 respondents a 3-point swing is noise. With the 20 to 30 people most interview studies have, it can decide which item comes out on top.\n\n### What they get right\n\nPew notes that the rank order of items was largely similar across the two formats, with differences mostly inside sampling error. So a select-all list is fine for a rough \"which of these are common\" read. It's the absolute shares that are off.\n\nThat evidence led Pew to a policy: it \"has adopted a policy of using a forced-choice format instead of a select-all-that-apply list in its online surveys whenever possible.\"\n\n## What a cap changes, and what nobody has measured\n\nA capped question (\"pick your top 3\") is a different measure again. It forces a trade-off, so it shows priority rather than prevalence. That makes it the right format for \"which problem should we solve first\", \"which message is clearest\" or \"which feature would you give up last\".\n\nBe honest about the evidence here. The Pew and Smyth studies compare select-all with forced choice. We found no study that isolates the effect of the cap itself, for example comparing \"pick up to 3\" with \"pick exactly 3\" on the same items. The advice below on caps is practice, not measured fact.\n\n**Rules for capped questions that hold up in practice:**\n\n1. **Pick the cap from the list length.** Three is a sensible default for 7 to 12 options. For 5 or fewer options a cap of 3 barely forces a choice, so use a single choice or a ranking question.\n2. **Exact or \"up to\"?** \"Exactly 3\" gives every person the same weight and makes counts easy to read. \"Up to 3\" is kinder when some people genuinely care about one thing, but those people then count less in the totals. Use exact when you'll report a ranking, up to when you'll read the answers person by person.\n3. **Always ask why.** A capped pick tells you what. The follow-up tells you whether two people who both picked \"Integrations\" meant the same thing.\n4. **Report capped counts as capped.** \"48% put pricing in their top 3\" is a different claim from \"48% care about pricing\". Never put a capped result next to a select-all result from another study as if they measured the same thing.\n5. **Shuffle the options.** A cap doesn't remove order effects, so randomize when the options have no natural order.\n\n## Which format should you use?\n\n![Flowchart for choosing a multi-select question format: forced yes/no for prevalence, a capped top 3 for priority, select-all for a rough list.](/visuals/multi-select-question-format-flowchart/multi-select-question-format-flowchart.png \"Start with the number you'll report, then pick the format that measures it.\")\n\nWork from the sentence you'll write in the report:\n\n- \"X% of customers use Y\" → yes/no per item.\n- \"The top three reasons were...\" → capped pick, with a why follow-up.\n- \"People mentioned a wide range of tools, mostly A, B and C\" → select-all is fine.\n\nIf the list has more than about 10 items and you need prevalence, a page of yes/no questions gets long. Split the list by theme, or screen first and only ask the yes/no items that apply.\n\n## How to word each format\n\n**Select all that apply**\n- Say it in the question: \"Select all that apply.\"\n- Keep it to 10 options or fewer, and put \"None of these\" last.\n- Don't expect absolute shares to hold; compare items with each other, not with other studies.\n\n**Yes/no per item**\n- One short question per item, all with the same stem: \"In the last month, have you used...?\"\n- Use a time window so \"yes\" means something you can count.\n- Allow \"Not sure\" only when people genuinely might not know.\n\n**Capped pick**\n- Put the cap in the question and the instruction: \"Which 3 matter most to you? Pick exactly 3.\"\n- Name the decision the pick feeds: \"...when choosing a tool like this.\"\n- Ask the why straight after, while the choice is fresh.\n\n## A worked example: 8 options, 25 people\n\nThis example is illustrative, to show how the formats read differently. The numbers aren't from a real study.\n\n![A product team sorting sticky notes on a whiteboard to decide which customer problems to fix first](https://images.unsplash.com/photo-1676276383599-478e31770aac?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3w5NDUzMDN8MHwxfHNlYXJjaHw5fHx0ZWFtJTIwc29ydGluZyUyMHN0aWNreSUyMG5vdGVzJTIwd2hpdGVib2FyZHxlbnwxfDB8fHwxNzk0MTIwNjEwfDA&ixlib=rb-4.1.0&q=80&w=1080)\n\n_Photo by [Walls.io](https://unsplash.com/@walls_io?utm_source=koji&utm_medium=referral) on [Unsplash](https://unsplash.com/?utm_source=koji&utm_medium=referral)_\n\nA product manager wants to know which of 8 onboarding problems to fix first and has 25 customers to ask.\n\n- **Select-all version:** 6 of the 8 problems get ticked by 9 to 16 people. Everything looks important and the order between neighbours is within a couple of people, so a 3-point order effect could swap them.\n- **Capped version (exactly 3, shuffled, with a why):** each person makes 3 picks, 75 in all. Two problems get 18 and 14 picks, and the rest spread thin. The why answers show that the second problem is two different issues under one label, which no checkbox would have shown.\n\nThe capped version gives the manager a decision. The select-all version gives them a list.\n\n## How Koji helps\n\nKoji runs these formats inside an AI-moderated interview, so every answer can come with its reason. What you can set today:\n\n- **Pick limits on multiple choice.** Choose Any number, an Exact number or a Range. In text interviews the answer widget enforces it and shows the hint (\"Choose 3\", \"Choose up to 3\", \"Choose 2 to 4\"). In voice interviews Koji asks the person to adjust when they give too many or too few.\n- **Option order per person.** Order as shown, Randomize order, or Randomize except the last 1 to 3 options, so \"Other\" or \"None of these\" stays at the bottom. Each participant keeps their own order if they refresh, and reports always list options in your original order.\n- **A why after the pick.** Each question has a follow-up setting (none, or 1 to 3 follow-ups) with optional guidance, so Koji asks why someone picked what they picked and the report shows the quotes next to the counts.\n- **Yes/no and single choice questions** for the forced-choice version, and ranking questions when you need a full order.\n\n![Koji question editor showing open ended, 1 to 10 scale and single choice questions with follow-up settings](/email/shots/shot-question-types.jpg \"Each question carries its own type, options and follow-up setting.\")\n\nA survey tool can randomize and cap too. The difference is the follow-up: you see why people chose, not only what. You can also ask the research assistant in plain words (\"make question 4 pick exactly three and shuffle the options, keep the last one in place\") and it sets the same options.\n\n## Common mistakes\n\n- **Reporting select-all shares as prevalence.** They under-count. If the share is the headline, ask yes/no.\n- **Comparing a capped count with an uncapped one.** They measure different things.\n- **Putting \"None of these\" in the shuffle.** Keep escape options last.\n- **A cap of 3 on a list of 4.** That's barely a choice. Use single choice or ranking.\n- **No follow-up.** A pick without a reason is easy to misread.\n\n## Related Resources\n\n- [Structured Questions Guide](/docs/structured-questions-guide)\n- [Survey Randomization Guide](/docs/survey-randomization-guide)\n- [Forced-Choice Survey Questions](/docs/forced-choice-survey-questions)\n- [Multiple Choice Questions in AI Interviews](/docs/multiple-choice-questions-ai-interviews)\n- [Question Order Bias](/docs/question-order-bias-guide)\n- [Choice and Ranking Questions](/docs/choice-ranking-questions-guide)\n\nFor founders deciding which customer problem to tackle first, the playbook [Which problem to solve first](/blog/which-problem-to-solve-first-rank-customer-pains) applies the capped-pick method end to end.\n\n## Sources\n\n- Pew Research Center, \"When Online Survey Respondents Only 'Select Some That Apply'\", May 2019 (American Trends Panel, 4,581 U.S. adults, July 30 to August 12, 2018).\n- Jolene D. Smyth, Don A. Dillman, Leah Melani Christian and Michael J. Stern, \"Comparing Check-All and Forced-Choice Question Formats in Web Surveys\", Public Opinion Quarterly 70(1), 66 to 77, 2006.\n","category":"Research Methods","lastModified":"2026-11-08T06:50:10.812573+00:00","metaTitle":"Select All That Apply vs Pick Your Top 3: Multi-Select Guide","metaDescription":"Select all that apply under-counts against yes/no questions by about 8 points (Pew, 2019). When to use each multi-select format and how to cap one.","keywords":["select all that apply vs select top 3","check all that apply vs forced choice","multi-select question best practice","limit number of choices survey","pick your top 3 question","forced choice survey","survey question design"],"aiSummary":"Use select all that apply only for a rough list, since it under-counts: in a 2019 Pew experiment, yes/no questions per item gave estimates about 8 points higher and showed no order effect, against 3 points for select-all. Ask yes/no per item when you need prevalence, and a capped pick (exactly 3, shuffled, with a why follow-up) when you need priority. The three formats measure different things and can't be compared.","aiPrerequisites":["A list of options you want to ask about","A rough idea of what decision the answers feed"],"aiLearningOutcomes":["Tell apart endorsement, prevalence and priority questions","Choose between select all, yes/no per item and a capped pick","Word each multi-select format so answers are readable","Read capped counts without comparing them to uncapped ones","Set pick limits and option order in an AI interview"],"aiDifficulty":"intermediate","aiEstimatedTime":"11 minutes"}],"pagination":{"total":1,"returned":1,"offset":0}}