{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-11-04T19:31:04.228Z"},"content":[{"type":"blog","id":"b406d946-5c50-4568-921a-eadbadab29b8","slug":"which-problem-to-solve-first-rank-customer-pains","title":"Which Problem Should You Solve First? Rank Customer Pains Without Leading Them","url":"https://www.koji.so/blog/which-problem-to-solve-first-rank-customer-pains","summary":"A founder guide to choosing which customer problem to solve first. Hear 8 to 12 stories, write a list of 5 to 8 pains, ask 15 to 30 people to pick their top three from a shuffled list, ask why, and check what they did about it. Read the result against chance (3 divided by the number of options), split close results by customer type, and confirm with a commitment such as a waitlist or pilot. Cites Pew Research on order effects, CB Insights on startup failure and Camuffo et al. on testing hypotheses.","content":"You have talked to a dozen people and you now hold a page of pains: invoices paid late, hours lost to spreadsheets, a tool nobody trusts. They all sound real, and you can only build for one. The short answer: hear the stories first, turn them into a list that is shuffled for every person, ask each person to pick their top three and say why, then check how many of them did something about it. By the end of this guide you can run that round with 15 to 30 people this week and read the result without fooling yourself.\n\n## When you need this\n\nYou are past the first few conversations and before you write code. You have a list of problems you heard, and you are tempted to rank them by the one you find most exciting. Founders ask it in plain words, for example [how do I talk to customers without them just telling me what I want to hear?](https://echai.ventures/startingup/customer-research/how-do-i-talk-to-customers-without-them-just-telling-me-what-i-want-to-hear) [8]\n\nThe cost of guessing is large. CB Insights looked at 385 VC-backed companies that shut down since 2023 and found that 43% pointed to poor product-market fit as a reason, with running out of capital the most common at 70%, which the report treats as the last step rather than the cause [2]. Percentages add up to more than 100 because companies named several reasons. Poor fit is often what a wrong pick of problem looks like two years later.\n\n![Bar chart of reasons 385 shut-down startups gave: ran out of capital 70 percent, poor product-market fit 43, bad timing 29, weak unit economics 19](/visuals/why-startups-fail-reasons-385-shutdowns/why-startups-fail-reasons-385-shutdowns.png \"Poor fit is the reason you can still act on before you build. · Source: CB Insights Research, Why Startups Fail, 431 shutdowns since 2023\")\n\nTeams that test before they commit do better. In a randomised trial of 116 Italian startups, founders taught to treat their ideas as hypotheses and test them performed better than a control group [4]. A later set of four trials with 759 firms found the same pattern, with fewer pivots among the teams that tested first [5]. Ranking problems with customers is one small, cheap version of that habit.\n\n## The method in brief\n\nA ranking round has two jobs, and they need two different tools.\n\n- **Open stories** tell you what the pains are, in the customer's words. This is the Mom Test style of interview: past behaviour, not opinions about your idea. See [the Mom Test](/docs/mom-test-methodology) and [problem validation](/docs/problem-validation-guide).\n- **A shuffled top-three pick** tells you how often each pain matters across many people. It is a short structured question, covered in [choice and ranking questions](/docs/choice-ranking-questions-guide).\n\nDo them in that order. Ash Maurya, who wrote Running Lean, advises opening with \"Walk me through your last experience with your current solution\" instead of asking about someone's biggest problem [3]. If you start with a list, you only learn which of your own guesses people tick.\n\nWhy shuffle? Order changes answers. In a Pew Research experiment on its online panel with 4,581 US adults, select-all lists showed an average order effect of 3 percentage points across 21 surveys, against 0 for forced-choice questions [1]. In one test, 57% endorsed the first-listed trait and 42% the last-listed one [1]. With 25 people, a gap of a few points can flip your number one, so give every person a different order. The longer write-up is in [survey randomization](/docs/survey-randomization-guide).\n\nWhy a top three and not a rating of each pain? Pew found that select-all lists and forced choice rank items in much the same order, and that either works if rank order is all you want [1]. Rating every pain on a 1 to 5 scale tends to make everything look important. A fixed pick of three forces a trade-off, which is what your customer does with their time and money anyway.\n\n## Step by step\n\n![Process of five steps to rank customer problems: hear stories, write the list, shuffle and pick three, ask why, and check what people did about it](/visuals/rank-customer-problems-five-steps/rank-customer-problems-five-steps.png \"Stories first, then the shuffled pick, then the proof. · Source: Koji synthesis\")\n\n**1. Hear 8 to 12 stories first (days 1 to 4).** Talk to people who had the problem recently, not people who might. For a group with similar jobs, themes repeat fast: in a classic study, 92% of the themes appeared within 12 interviews, 70% within the first six [6]. That study used one fairly uniform group, so mix very different customers and you need more. How to find them is in [finding 20 people to interview](/blog/find-people-to-interview-with-no-audience).\n\n**2. Write the list from their words (day 5).** Take the pains that came up more than once and write each as the customer would say it: \"I chase invoices for weeks\", not \"Accounts receivable friction\". Keep it to 5 to 8 items. Add \"Something else\" at the end and keep it there.\n\n**3. Run the pick with 15 to 30 people (days 6 to 10).** Ask each person to choose the three that cost them the most in the last three months. Shuffle the list for each person and leave \"Something else\" at the bottom.\n\n**4. Ask why, for each pick.** The number tells you how many. The reason tells you whether to believe it.\n\n**5. Check what they did (same call).** Ask what they tried, what it cost and who else was in the room. A pain with no workaround, no money spent and no time given is a mild annoyance.\n\n### The question bank\n\n![Question bank with nine questions in three parts for ranking customer problems: hear the story, rank the pains, and check the commitment](/visuals/rank-customer-problems-interview-questions/rank-customer-problems-interview-questions.png \"Save this list for your next round. Change the options, keep the order of the questions.\")\n\n## A worked example\n\nSay you are a two-person team exploring software for freelance bookkeepers. You have done ten story calls. Seven pains came up more than once. You write them as the bookkeepers said them and run the pick with 24 bookkeepers. These numbers are made up to show the reading, not real data.\n\n![Two co-founders at a desk comparing notes on a laptop to decide which customer problem to build for first](https://images.unsplash.com/photo-1664223308156-3d374ea8d7eb?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3w5NDUzMDN8MHwxfHNlYXJjaHwzfHx0d28lMjBwZW9wbGUlMjBsYXB0b3AlMjBub3RlcyUyMGRlc2t8ZW58MXwwfHx8MTc5MzgwMzgxMXww&ixlib=rb-4.1.0&q=80&w=1080)\n\n_Photo by [GRIN](https://unsplash.com/@grin_co?utm_source=koji&utm_medium=referral) on [Unsplash](https://unsplash.com/?utm_source=koji&utm_medium=referral)_\n\n| Pain | Picked in top three | Said why |\n|---|---|---|\n| Chasing clients for receipts | 17 of 24 | \"Every month I send the same email four times\" |\n| Reconciling bank feeds | 13 of 24 | Cost hours, but they have a macro for it |\n| Explaining reports to clients | 11 of 24 | Varied a lot by client type |\n| Onboarding a new client | 9 of 24 | Mostly a one-off |\n| Chasing the client's accountant | 7 of 24 | Rare |\n| Tax deadline stress | 7 of 24 | Seasonal |\n| Something else | 4 of 24 | Mixed |\n\nFirst, remember what chance looks like. With seven options and three picks each, a pain that nobody cared about would still be picked by about 43% of people (3 divided by 7), so 13 of 24 is barely above noise. Receipts at 17 of 24, 71%, is well above it. With 24 people the margin around that figure is about 20 points either way (the usual 95% interval for a share, my arithmetic), so you can say receipts leads, and you cannot say whether bank feeds or reports comes second.\n\nSecond, read the reasons. Receipts has a repeated, costly workaround (\"the same email four times\"). Bank feeds has a macro, so it is a solved pain for many. You choose receipts and go back to five of the 17 who picked it, to ask what they have tried and what they would pay to stop doing it.\n\n## Reading the results and deciding\n\nThese are working rules, not findings from a study. Adjust them to your market.\n\n- **Compare with chance, not zero.** Chance is 3 divided by the number of options. A pain needs to clear that by a wide margin.\n- **One clear leader** (picked by roughly twice the chance rate, with specific reasons): take it into a solution conversation. See [solution interviews](/docs/solution-interviews-guide).\n- **Two or three close:** do not average them. Cut the picks by who the person is. A split often means two groups with two different problems, and you pick the group (see [ranking by segment](/docs/segment-ranking-sample-size-artifact)).\n- **A leader with no stories behind it:** go back and ask the people who picked it for their last time. If they cannot produce one, it is an opinion, not a pain.\n- **Nothing clears chance:** your list missed the real pain. Read the \"Something else\" answers and run the story calls again.\n\n![Flowchart for reading a top-three problem ranking: compare with chance, check the stories, split close results by customer type, then test commitment](/visuals/read-customer-problem-ranking-flowchart/read-customer-problem-ranking-flowchart.png \"Follow the arrows before you pick what to build. · Source: Koji synthesis\")\n\n## Run it this week with Koji\n\nEverything above works with a spreadsheet and a calendar. Koji makes the pick part faster, and it keeps the reasons next to the numbers.\n\n![Koji question editor showing open ended, 1 to 10 scale and single choice questions with follow-up settings](/email/shots/shot-question-types.jpg \"The question editor: add a multiple choice question after your open ones and set how many people may pick.\")\n\n1. **Start with an open question.** \"Tell me about the last time [the problem] happened.\" Let the AI interviewer follow up on what they say.\n2. **Add a multiple choice question** with your 5 to 8 pains as the options.\n3. **Set the pick limit** to Exact number, 3. In a text interview the widget enforces it. In a voice interview the interviewer asks the person to adjust if they name too many or too few.\n4. **Set the option order** to \"Randomize except last\", keep the last one, and leave \"Something else\" at the bottom. Each person sees their own order and keeps it if they refresh. Reports show your original order, so nothing is scrambled when you read them.\n5. **Add a follow-up** asking why for each pick and what they tried.\n6. **Send the study link** to 15 to 30 people and read the report: the share who picked each pain, with the answers behind each number.\n\nThese settings are available on every plan, including the free one. If you are not sure how to word the options, ask the research assistant to draft the question from your story notes.\n\n## Common mistakes\n\n- Ranking before you have stories, so the list is only your own guesses.\n- Listing options in the order you believe in them. You get your own bias back.\n- Letting people tick everything. Without a cap, every pain scores high.\n- Reading a gap of two or three people out of 24 as a winner.\n- Mixing very different customers in one count.\n- Treating the pick as demand. People tell you what hurts; they have not told you what they will pay to fix.\n\n## When research will mislead you\n\nA ranking shows the pains people recognise on a list. It cannot show the pain you forgot to list, which is why the story calls come first and \"Something else\" stays. It also measures stated priority, and stated priority is not purchase. Pair the top pick with a commitment: a waitlist signup with a work email, a paid pilot, a letter of intent, or a booked follow-up call. See [waitlist validation](/docs/waitlist-validation-guide) and [fake door testing](/docs/fake-door-testing-guide).\n\nFounders also ask whether an AI persona can do the ranking for them. A 2026 review of synthetic respondents from NIM concludes that this data fits early concept exploration better than decisions that need precise insight [7]. Use it to draft your options, then ask real people to pick.\n\n## Where this fits\n\nThis is the step after you have people to talk to: [how to find 20 people to interview](/blog/find-people-to-interview-with-no-audience) comes first. Once one pain leads, the next question is whether they want what you plan to build, which is [testing your landing page before you launch](/blog/test-your-landing-page-before-you-launch). The whole route is laid out in [the founder's research playbook](/learn/founders#idea-stage), and there is more for this stage at [koji.so for founders](/for/founders).\n\n## FAQ\n\n**How many people do I need to rank problems?**\nFifteen to thirty for a first read, after 8 to 12 story calls. Below 15 a single answer moves the result too much. Different customer types in one count need more people.\n\n**Should I let people rank all the options or pick three?**\nPick three. A fixed pick forces a trade-off. Pew found that either format ranks items in a similar order, so use the one your people can answer fast [1].\n\n**Why shuffle the options if I only have 20 people?**\nBecause with 20 people a few points of order effect can change your top pick. Shuffling costs nothing.\n\n**What if my top pick is a problem I do not want to solve?**\nThen you learned that before you built it. Check whether another group ranks a different pain first. Ranking tells you what customers rate, not what you must build.\n\n**Is this the same as a survey?**\nThe pick is survey-shaped, but the reasons make it an interview. You get the count and the why in one sitting.\n\n**When will this mislead me?**\nWhen the list is missing the real pain, when the sample is not your customer, or when you treat a stated priority as a promise to pay.\n\n**Next step:** write down your list of pains in the customers' words, and send the pick to your first 15 people this week.\n\n## References\n\n1. Pew Research Center. \"When Online Survey Respondents Only 'Select Some That Apply'.\" Pew Research Center Methods, 9 May 2019. https://www.pewresearch.org/methods/2019/05/09/when-online-survey-respondents-only-select-some-that-apply/\n2. CB Insights Research. \"Why Startups Fail: Top 9 Reasons.\" CB Insights Research (current edition, covers 431 VC-backed shutdowns since 2023). https://www.cbinsights.com/research/report/startup-failure-reasons-top/\n3. Maurya, Ash. \"How to Validate Any Startup Idea in 90 Days, LEAN 1-2-3.\" Lean Foundry newsletter, 31 July 2025. https://leanfoundry.com/newsletter/jul-31-2025\n4. Camuffo, A., Cordova, A., Gambardella, A. and Spina, C. \"A Scientific Approach to Entrepreneurial Decision Making: Evidence from a Randomized Control Trial.\" Management Science 66(2), 2020 (working paper version: CEPR DP12421). https://cepr.org/publications/DP12421\n5. Camuffo, A., Gambardella, A., Messinese, D., Novelli, E., Paolucci, E. and Spina, C. \"A scientific approach to entrepreneurial decision-making: Large-scale replication and extension.\" Strategic Management Journal 45(6), 2024. https://repositorio.ie.edu/entities/publication/6aae4961-2c1d-45ed-97eb-e8bee19a2471/full\n6. Guest, G., Bunce, A. and Johnson, L. \"How Many Interviews Are Enough? An Experiment with Data Saturation and Variability.\" Field Methods 18(1), 2006. https://bibbase.org/network/publication/guest-bunce-johnson-howmanyinterviewsareenoughanexperimentwithdatasaturationandvariability-2006\n7. NIM. \"Leaving Insight to Digital Twins? Promise, Progress and Limits of Synthetic Respondents.\" NIM publications, 2026. https://nim.org/en/publications/detail/leaving-insight-to-digital-twins\n8. Echai Ventures. \"How do I talk to customers without them just telling me what I want to hear?\" Echai Ventures, 2026. https://echai.ventures/startingup/customer-research/how-do-i-talk-to-customers-without-them-just-telling-me-what-i-want-to-hear\n\n**Further reading**\n\n- [The Mom Test](/docs/mom-test-methodology): the interview rules to follow before any ranking.\n- [Survey randomization](/docs/survey-randomization-guide): the longer treatment of shuffling options.\n- [Choice and ranking questions](/docs/choice-ranking-questions-guide): how the question types work in an interview.\n- Pew's report [1]: the clearest experiment on how question format changes answers.\n","category":"Founders","lastModified":"2026-11-04T14:50:11.368831+00:00","metaTitle":"Which Problem to Solve First: Rank Customer Pains","metaDescription":"Heard several customer pains? Hear the stories, then have 15 to 30 people pick a top three from a shuffled list. How to read it against chance.","keywords":["which problem to solve first","prioritize customer problems","rank customer pain points","startup problem validation","customer discovery","top three pick survey"],"aiSummary":"A founder guide to choosing which customer problem to solve first. Hear 8 to 12 stories, write a list of 5 to 8 pains, ask 15 to 30 people to pick their top three from a shuffled list, ask why, and check what they did about it. Read the result against chance (3 divided by the number of options), split close results by customer type, and confirm with a commitment such as a waitlist or pilot. Cites Pew Research on order effects, CB Insights on startup failure and Camuffo et al. on testing hypotheses.","aiKeywords":["prioritize customer problems","rank customer pain points","which problem to solve first","problem validation","customer discovery","shuffle answer options"],"aiContentType":"guide","faqItems":[{"answer":"Fifteen to thirty for a first read, after 8 to 12 story calls. Below 15 a single answer moves the result too much, and different customer types in one count need more people.","question":"How many people do I need to rank problems?"},{"answer":"Pick three. A fixed pick forces a trade-off. Pew Research found that select-all lists and forced choice rank items in a similar order, so use the format your people can answer fast.","question":"Should I let people rank all the options or pick three?"},{"answer":"With 20 people a few points of order effect can change your top pick. Shuffling the options for each person costs nothing and removes the first-listed advantage.","question":"Why shuffle the options if I only have 20 people?"},{"answer":"Then you learned it before building. Check whether another customer group ranks a different pain first. Ranking tells you what customers rate, not what you must build.","question":"What if my top pick is a problem I do not want to solve?"},{"answer":"The pick is survey-shaped, but asking why after each pick makes it an interview. You get the count and the reasons in one sitting.","question":"Is this the same as a survey?"},{"answer":"When the list is missing the real pain, when the sample is not your customer, or when you treat a stated priority as a promise to pay. Pair the top pick with a commitment such as a waitlist signup or a paid pilot.","question":"When will this mislead me?"}],"relatedTopics":["problem validation","customer discovery","survey design","prioritization"]}],"pagination":{"total":1,"returned":1,"offset":0}}