{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-11-10T20:00:31.321Z"},"content":[{"type":"blog","id":"36336b0e-3626-4796-92fb-1b94029705d6","slug":"validate-startup-idea-with-ai","title":"How to Validate a Startup Idea With AI (Without Fooling Yourself)","url":"https://www.koji.so/blog/validate-startup-idea-with-ai","summary":"A founder guide to validating a startup idea with AI. Use AI to list assumptions, draft questions and summarise notes, but get evidence from 10 to 15 real interviews in one narrow segment and a commitment test. Cites 2025 evidence that AI-generated survey answers miss badly on thin topics (19.8 points of error on coffee questions versus 3.0 on politics), a 21% average overstatement in stated willingness to pay, and a randomised trial favouring hypothesis testing. Includes a question bank, a worked example and decision rules.","content":"You can use AI to validate a startup idea, but only for the parts of the job that don't need a customer. A chatbot can sharpen your assumptions, draft your questions and summarise what people tell you. It cannot tell you whether anyone will pay, because it has never met your customers. By the end of this guide you'll know where AI helps, where it fools you, and how to get real evidence from about 10 to 15 people in a week.\n\n## When you need this\n\nYou have an idea, maybe a Notion page of it, and you've done the thing everyone does now: pasted it into a chatbot and asked \"is this a good idea?\" It said yes, with a SWOT analysis. You feel better, and you've learned nothing you can bet on.\n\nThe sign you need this guide: you can't name the last three people who described this problem to you in their own words. If you can't, you have an opinion, not evidence.\n\nThe cost of guessing is large and boring. You build for months, then find that the people you imagined were never looking for a fix. Most expensive mistakes are about demand, and demand lives with people, not models.\n\n## What the research says about AI and real customers\n\nTwo things are true at once, and you should hold both.\n\nFirst, the newest results on AI stand-ins for customers are better than they were. A 2025 preprint from Maier and colleagues asked language models for purchase intent on product concepts. When the models were asked for a 1 to 5 score directly, the match with real survey distributions was poor. When the models first wrote a free-text opinion and that text was converted to a score, the match jumped [2]. The chart below shows the gap on a 0 to 1 similarity scale.\n\n![Column chart: two chatbots matched real purchase-intent ratings at 0.26 and 0.39 when asked for a score, and 0.88 and 0.80 when asked for words first.](/visuals/llm-purchase-intent-direct-rating-vs-semantic-similarity/llm-purchase-intent-direct-rating-vs-semantic-similarity.png \"The method you use to ask matters more than which chatbot you ask. · Source: Maier et al., LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings, arXiv, 2025\")\n\nSecond, the same year, Verasight compared AI-generated survey answers with a nationally representative survey of more than 1,500 US adults. On political questions the average error was 3.0 percentage points. On coffee questions it was 19.8. On daily coffee drinking, the real figure was 56% and the AI-generated answers said 91% [1]. The model did well where the internet has talked about the topic for years, and badly where it hadn't.\n\nWhat that means for you: a new idea in a niche market is exactly the place where the model has the least to go on. The Maier study tested packaged consumer goods against real panel data, which is a different world from \"scheduling software for dental clinics\". Both studies are preprint or industry-report evidence rather than peer-reviewed trials, so read them as signals, not proof.\n\nThere is a second, older reason to want real people: stated intent runs high. A meta-analysis of 77 studies on willingness to pay found people overstate what they'd pay by about 21% on average, and indirect methods like conjoint overstated more than direct questions [3]. If real people overstate, a model trained on what people say overstates too.\n\nAnd there is evidence that testing beats thinking. In a randomised trial of Italian startups, founders trained to form hypotheses and test them abandoned more weak ideas, pivoted in a more focused way and earned more on average than those who relied on intuition [4]. The point is the testing, not the training course.\n\n## The method in brief\n\nUse AI as an assistant for your thinking and real conversations as your evidence. In practice that is three layers.\n\n1. **AI for the desk work.** List your assumptions, find the riskiest one, draft questions, spot leading wording, tidy transcripts and pull themes. Check any market-size number it gives you by hand.\n2. **Real people for the evidence.** 10 to 15 interviews with people who match your target, asked about their past behaviour. See [the Mom Test guide](/docs/mom-test-methodology) and [customer discovery interviews](/docs/customer-discovery-interviews) for the method.\n3. **A commitment test for the truth.** Something that costs the person something: a pilot date, a deposit, an introduction, a pre-order. [Smoke tests](/docs/smoke-test-product-validation) and [the guide to which validation test to run first](/docs/which-validation-test-to-run-first) cover the options.\n\nIf you want the longer argument on AI personas, see [synthetic users in research](/docs/synthetic-users-research-methodology). Here is the short version: use them to rehearse and brainstorm, never to decide.\n\n## Step by step: five days\n\n**Day 1. Write the idea as a bet.** One sentence: \"[Type of person] has [problem] often enough to pay about [amount] to fix it.\" Ask a chatbot to list the five assumptions that must be true, then pick the one that would hurt most if wrong. Don't ask it for a verdict.\n\n![A founder at a desk with a laptop and notebook, planning a week of customer interviews.](https://images.unsplash.com/photo-1559163499-413811fb2344?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3w5NDUzMDN8MHwxfHNlYXJjaHw1fHxmb3VuZGVyJTIwbGFwdG9wJTIwbm90ZWJvb2slMjBjb2ZmZWUlMjBkZXNrfGVufDF8MHx8fDE3OTQzMDA2MTB8MA&ixlib=rb-4.1.0&q=80&w=1080)\n\n_Photo by [Unsplash](https://unsplash.com/@unsplash?utm_source=koji&utm_medium=referral) on [Unsplash](https://unsplash.com/?utm_source=koji&utm_medium=referral)_\n\n**Day 1. Pick one narrow group.** Not \"small businesses\", but \"owners of two to five dentist clinics\". If you're unsure who, start with [finding your first customer segment](/blog/find-your-first-customer-segment-before-you-build).\n\n**Day 2. Find 15 people.** Your network, communities where the group already posts, LinkedIn messages that ask for 15 minutes and say what you want to learn, not what you want to sell. [How to find 20 people to interview with no audience](/blog/find-people-to-interview-with-no-audience) has the scripts.\n\n**Days 3 and 4. Interview.** Aim for 10 to 15 conversations. Ask about the last time the problem happened, what they did, what it cost and what they already pay for. [How many user interviews](/docs/how-many-user-interviews) explains why you'll start hearing repeats within a dozen or so in a narrow group, and why you should stop earlier if the first five say nothing.\n\n**Day 5. Decide.** Sort what you heard into the three signals below, then run one commitment test.\n\n## What to ask\n\nThese questions stay on what happened, not on what people would do. Take them as a starting bank and keep the wording plain.\n\n![Question bank: nine questions for validating a startup idea, in three groups: the last time, what they tried, and commitment.](/visuals/startup-idea-validation-interview-questions/startup-idea-validation-interview-questions.png \"Ask about the past and about commitment. Skip the would-you-use-this question. · Source: Koji synthesis, after Rob Fitzpatrick, The Mom Test\")\n\nDo not describe your idea until the last ten minutes. If you do, people try to be kind. [How to avoid leading questions](/docs/avoiding-leading-questions) shows what to cut.\n\n## A worked example\n\nSay you're a two-person team with an idea for scheduling software that cuts no-shows at dental clinics. A chatbot says the market is large, no-shows cost clinics a lot, and your idea is promising. None of that is evidence.\n\nYou write the bet: \"Practice managers at clinics with two to five dentists lose enough to no-shows to pay for a fix.\" The riskiest assumption turns out to be that practice managers, not owners, feel the pain. You find 14 practice managers through two dental forums and a friend's clinic, and interview them.\n\nWhat you hear: 9 of 14 describe a specific no-show last month and what it cost them. Of those 9, 6 already tried text reminders, and 4 pay for software that includes them. Only 2 say they'd look at another tool. Three offer an intro to the owner, which is a stronger signal than any compliment.\n\nWhat you decide: the pain is real, but most have a partial fix. Your idea needs to beat text reminders, so the next test is narrower: the 3 clinics without any reminder tool. You ask them for a 30-day pilot. This is made-up data to show the shape of the decision, not a result from a real study.\n\n## Reading the results and deciding\n\nLook for three signals in your notes:\n\n- **Behaviour:** people describe a specific recent episode, not \"it would be nice\".\n- **Pain with cost:** they can say what it cost in hours or money.\n- **Commitment:** they give something: time, an intro, data, a deposit, a pilot date.\n\nIf most of your 10 to 15 have the first two and a few give the third, narrow and test further. If you only hear compliments, you're in polite-answer territory: stop and change the segment or the problem. If the same people keep saying \"we've solved it with a spreadsheet\", ask what the spreadsheet fails at before you write any code.\n\nOne rule of thumb that I use, not a published threshold: treat any AI summary as a draft, and don't write down a finding unless you can point to two or more transcripts that say it.\n\n![Flowchart: is your AI validation real evidence? Three questions about real behaviour, repeat across people, and commitment.](/visuals/is-ai-idea-validation-real-evidence-flowchart/is-ai-idea-validation-real-evidence-flowchart.png \"Run any claim through these three questions before it goes in a pitch or plan.\")\n\n## Run it this week with Koji\n\nYou can do all of this by hand with a calendar, a call recorder and a spreadsheet, and many founders do. Koji is one way to get the interviews done without scheduling every call. It runs interviews with real people by text or voice, so you get 10 to 15 transcripts rather than synthetic answers. Here is the setup.\n\n1. **Start from the idea.** On the home screen, type your riskiest assumption as the research question. The research assistant drafts a brief with a goal and open questions. Edit it with the question bank above, and ask the assistant to remove any leading question.\n\n![Koji home screen asking \"What do you want to research?\" with a research question typed in and a Start study button.](/email/shots/shot-brief-from-one-message.jpg \"Type the question once and the assistant drafts the brief for you to edit.\")\n\n2. **Screen for your segment.** Add a screening question so only the right role gets through, for example practice managers.\n3. **Add a concept or a price.** Attach an image of your landing page or mockup to any question. Show it once for 3, 5, 10 or 15 seconds as a first-impression test, then let Koji ask what people remember. For pricing, the price test question takes a ladder of 3 to 10 prices and reads best from about 30 interviews. It's directional, so confirm with real payment.\n4. **Send it to real people.** Share the interview link with your list or community, or recruit from Koji's panel (launching a panel recruitment needs a paid plan).\n5. **Read the report with the evidence.** Each finding links to the quotes behind it. Read three full transcripts yourself before you trust the summary.\n\n### Check the interview before it goes out\n\nTake your own interview first, by text and by voice. A preview is free, doesn't count as a response and never shows up in the report, so you can fix the wording first.\n\n![Koji preview complete banner saying the preview is not counted and not charged.](/email/shots/shot-preview-complete.jpg \"A preview is saved with its own analysis and never counts as a response.\")\n\nIf you want to try it, start with one study on the free account: five interviews, one segment, the questions above.\n\n## Common mistakes\n\n- **Asking the chatbot to judge the idea.** It will find reasons it could work, because it is built to be helpful.\n- **Using AI answers as customers.** Fine for rehearsal. Not for a decision.\n- **Taking a compliment as demand.** \"Cool idea\" costs nothing to say.\n- **Interviewing friends and fellow founders.** They are kind, and they aren't buyers.\n- **Skipping the commitment test.** Interviews tell you about pain, not about paying.\n\n## When research will mislead you\n\nInterviews tell you what people say and remember, which is not the same as what they will do. The 21% average overstatement in stated willingness to pay is the reason to pair what you hear with a deposit, a pilot or a pre-order [3]. Small samples can hide a segment you didn't talk to. And an AI interviewer or summariser can miss tone, so read the transcripts. [Can you trust AI interviewers?](/docs/ai-interview-hallucinations-bias-mitigation) covers how Koji limits that.\n\n## Where this fits\n\nBefore this, decide [which problem to solve first](/blog/which-problem-to-solve-first-rank-customer-pains) and [find a first segment](/blog/find-your-first-customer-segment-before-you-build). After it, [price before you have customers](/blog/price-a-product-before-you-have-customers) and [test your landing page](/blog/test-your-landing-page-before-you-launch). The whole path is in [the founder's research playbook](/learn/founders#idea-stage), and [Koji for founders](/for/founders) shows the setup.\n\n## FAQ\n\n**Can ChatGPT validate my startup idea?**\nIt can help you list assumptions, draft interview questions and summarise notes. It can't validate demand, because it has no access to what your customers do or pay for. Treat its verdict as a prompt for what to test.\n\n**Are synthetic users good enough to validate an idea?**\nFor rehearsal, sometimes. For a decision, not yet. Error was 19.8 points on a topic with thin coverage in one 2025 comparison [1], and results depend on the method [2]. Check any synthetic result against real people.\n\n**How many real people do I need?**\nStart with 10 to 15 in one narrow segment. If the first five say nothing specific, change the segment before you do more.\n\n**Can an AI interview my customers?**\nYes, and the people answering are real. The interviewer is the AI; the evidence is human. Take a preview yourself first.\n\n**How long does it take?**\nAbout a week: a day to write the bet, a day or two to find people, two days of interviews and a day to decide.\n\n**What if everyone is polite?**\nAsk about the last time they hit the problem, and ask for a small commitment. Politeness without commitment is a no.\n\n## Next step\n\nWrite your idea as a one-sentence bet, pick the assumption that would hurt most, and book five conversations this week.\n\n## References\n\n1. Verasight, _Synthetic Sampling Report for Market Research_ (coffee), 2025. [verasight.io/reports/coffee-llm](https://www.verasight.io/reports/coffee-llm)\n2. Maier, B. F. et al., _LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings_, arXiv:2510.08338, 2025. [arxiv.org/abs/2510.08338](https://arxiv.org/abs/2510.08338)\n3. Schmidt, J. and Bijmolt, T., _Accurately measuring willingness to pay for consumer goods: a meta-analysis of the hypothetical bias_, Journal of the Academy of Marketing Science, 2020. [marketingcenter.de/en/publication/151167](https://www.marketingcenter.de/en/publication/151167)\n4. Camuffo, A., Cordova, A., Gambardella, A. and Spina, C., _A Scientific Approach to Entrepreneurial Decision Making: Evidence from a Randomized Control Trial_, Management Science, 2020. [doi.org/10.1287/mnsc.2018.3249](https://doi.org/10.1287/mnsc.2018.3249)\n\n**Further reading**\n\n- [The Mom Test method](/docs/mom-test-methodology): the questions that keep people from being polite.\n- [Synthetic users in research](/docs/synthetic-users-research-methodology): where AI personas help and where they mislead.\n- [Which validation test to run first](/docs/which-validation-test-to-run-first): interviews, fake door, landing page or pre-sales.\n","category":"Founders","lastModified":"2026-11-10T08:50:10.682678+00:00","metaTitle":"Validate a Startup Idea With AI, Without Fooling Yourself","metaDescription":"What AI can and can't do when you validate a startup idea, and how to get real evidence from 10 to 15 people in a week.","keywords":["validate startup idea with ai","chatgpt validate startup idea","synthetic users startup","ai customer discovery","startup idea validation","customer interviews founders"],"aiSummary":"A founder guide to validating a startup idea with AI. Use AI to list assumptions, draft questions and summarise notes, but get evidence from 10 to 15 real interviews in one narrow segment and a commitment test. Cites 2025 evidence that AI-generated survey answers miss badly on thin topics (19.8 points of error on coffee questions versus 3.0 on politics), a 21% average overstatement in stated willingness to pay, and a randomised trial favouring hypothesis testing. Includes a question bank, a worked example and decision rules.","aiKeywords":["validate startup idea with AI","synthetic users","AI customer discovery","ChatGPT idea validation","customer interviews","commitment test"],"aiContentType":"guide","faqItems":[{"answer":"It can help you list assumptions, draft interview questions and summarise notes. It can't validate demand, because it has no access to what your customers do or pay for. Treat its verdict as a prompt for what to test.","question":"Can ChatGPT validate my startup idea?"},{"answer":"For rehearsal, sometimes. For a decision, not yet. One 2025 comparison found 19.8 points of average error on a topic with thin coverage, and results depend on the method. Check any synthetic result against real people.","question":"Are synthetic users good enough to validate an idea?"},{"answer":"Start with 10 to 15 in one narrow segment. If the first five say nothing specific, change the segment before you do more.","question":"How many real people do I need?"},{"answer":"Yes, and the people answering are real. The interviewer is the AI; the evidence is human. Take a preview yourself first.","question":"Can an AI interview my customers?"},{"answer":"About a week: a day to write the bet, a day or two to find people, two days of interviews and a day to decide.","question":"How long does it take?"},{"answer":"Ask about the last time they hit the problem, and ask for a small commitment. Politeness without commitment is a no.","question":"What if everyone is polite?"}],"relatedTopics":["idea validation","AI for founders","synthetic users","customer discovery"]}],"pagination":{"total":1,"returned":1,"offset":0}}