{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-11-11T19:57:10.057Z"},"content":[{"type":"blog","id":"377905a5-dcdb-47e5-86a8-07f9e6fd7d04","slug":"feature-validation-methods-compared-product-managers","title":"Feature Validation Methods Compared: Interviews, Fake Doors, Prototypes and Price Tests for Product Managers","url":"https://www.koji.so/blog/feature-validation-methods-compared-product-managers","summary":"Product managers can validate a feature with six methods: problem interviews (is the problem real), concept tests (do people understand and want it), fake doors (do they click), prototype tests (can they use it), price tests (what would they pay) and staged rollouts or A/B tests (does it move the metric). Most features need two: one method that asks people and one that watches behaviour, typically problem interviews plus a fake door. Pendo found 80% of features are rarely or never used, and Microsoft experiments showed only about a third of ideas improved their target metric.","content":"**The short answer:** match the method to the question. Use **problem interviews** to learn whether the problem is real and frequent, a **concept test** to see whether people understand the idea and want it, a **fake door** to measure whether they click when it's in front of them, a **prototype test** to check they can use it, a **price test** to see what they would pay, and a **staged rollout or A/B test** to prove it moves the metric. Most product managers need two of these per feature, not six: one that asks people and one that watches what they do.\n\nThis guide is for product managers with a live product and real users, deciding what to build next. If you are still validating a startup idea, start with [which validation test to run first](/docs/which-validation-test-to-run-first) instead.\n\n## Why feature validation matters more than it feels\n\nShipping is the easy part. The hard part is shipping things people use:\n\n- Pendo's analysis of three months of usage across 615 of its customers' products found that **80%** of features are rarely or never used, and an average of **12%** of features generate 80% of daily usage ([Pendo, Feature Adoption Report, 2019](https://www.pendo.io/resources/the-2019-feature-adoption-report/)).\n- At Microsoft, controlled experiments showed that only about **one third** of ideas improved the metric they were designed to improve; a third made no difference and a third made it worse ([Ron Kohavi, Online Controlled Experiments: Introduction, Learnings, and Humbling Statistics, 2012](https://ai.stanford.edu/~ronnyk/2012-09ACMRecSysNR.pdf)).\n- In CB Insights' review of 111 startup post-mortems, **35%** named no market need as a cause of failure ([CB Insights, The Top Reasons Startups Fail, 2021](https://www.cbinsights.com/research/report/startup-failure-reasons-top/)).\n\nThe Pendo figure is from 2019 and covers a three-month window, so treat it as a direction, not a precise benchmark. The point holds: most features you are confident about will not move anything. Validation is how you find out which ones before engineering spends a quarter on them.\n\n## The six methods, and the question each one answers\n\nEach method answers one question well and others badly. Pick by the question you can't answer yet.\n\n![Table comparing six feature validation methods by the question each answers, effort, time and when it misleads.](/visuals/feature-validation-methods-comparison-table/feature-validation-methods-comparison-table.png \"Pick the method by the question you cannot answer yet\")\n\n| Method | Answers | Effort | Time | Misleads when |\n|---|---|---|---|---|\n| Problem interviews | Is the problem real, frequent and painful? | Low | Days | You pitch the feature instead of asking about the last time |\n| Concept test | Do people understand it and want it? | Low | Days | The mockup looks finished and people are polite |\n| Fake door | Do people click when it's in front of them? | Medium | 1 to 3 weeks | The button is hidden or the label oversells |\n| Prototype test | Can people use it? | Medium | 1 to 2 weeks | You test with people who would never need it |\n| Price test | What would they pay? | Low | Days | The people answering are not the buyers |\n| Staged rollout or A/B test | Does it move the metric? | High | Weeks | Traffic is too low to read the result |\n\n### 1. Problem interviews\n\nTalk to users about the last time they hit the problem: what they were doing, what they tried, what it cost them. Don't mention the feature. If people can't recall a recent example, the problem is not frequent enough to build for. Ten to fifteen interviews with the right segment usually show whether a pattern exists; [how many user interviews](/docs/how-many-user-interviews) covers sample sizes. The full method is in our [feature validation guide](/docs/feature-validation-guide).\n\n### 2. Concept test\n\nShow a mockup, a description or a short video of the feature and ask what they think it does, when they would use it and what it would replace. Concept tests catch misunderstandings early, which is cheaper than catching them after launch. A rough sketch gets more honest answers than a polished design, because people are less afraid to criticise it. See [concept testing methodology](/docs/concept-testing-methodology).\n\n### 3. Fake door\n\nAdd the entry point (a menu item, a button, a pricing tier) before the feature exists, and count how many people click. Clicking leads to a short \"we're building this, want early access?\" message. A fake door is the only cheap method that measures behaviour instead of opinion, but it measures curiosity, not lasting use. Keep it up for a fixed time, and decide the click rate that would count as a yes before you launch it. [Fake door testing](/docs/fake-door-testing-guide) covers ethics and setup.\n\n### 4. Prototype test\n\nAsk users to complete a task in a clickable prototype while you watch. This tells you whether the design works, not whether anyone needs it, so run it after the problem is confirmed. Five users find most of the usability problems in one round of testing; see [how many users for usability testing](/blog/how-many-users-usability-testing-2026) and [prototype testing and concept validation](/docs/prototype-testing-concept-validation).\n\n### 5. Price test\n\nIf the feature is a paid add-on or a reason to upgrade, test the price before you build the packaging. A Gabor-Granger test asks people whether they would buy at one price at a time and finds the price that earns the most. Make sure the people answering are the ones who would pay. [Price testing methods for startups compared](/blog/price-testing-methods-for-startups-compared) shows when to use which method.\n\n### 6. Staged rollout or A/B test\n\nShip to a slice of users and compare the metric against everyone else. It's the only method that proves impact, and the most expensive, because you have to build the feature first. Use it to confirm a decision the cheaper methods already pointed to, not to discover what to build. With low traffic, an A/B test can run for months without a clear result; a staged rollout with follow-up interviews is often more useful for B2B products.\n\n## Map: what people say versus what they do\n\nThe methods fall into three groups. Asking people (interviews, concept tests) is fast and tells you why. Watching behaviour (fake doors, prototype tests, rollouts) is slower and tells you what. Asking for commitment (price tests, pre-orders) sits in between.\n\n![Method map placing six feature validation methods by effort and by whether the evidence is what people say or what they do.](/visuals/feature-validation-methods-map-effort-evidence/feature-validation-methods-map-effort-evidence.png \"Pair one method that asks with one that watches · Source: Koji synthesis, after Christian Rohrer, Nielsen Norman Group (2014)\")\n\nThe pairing that works for most features: **problem interviews plus a fake door**. The interviews tell you whether the problem is real and in what words people describe it, and the fake door, labelled in those words, tells you whether people act on it. When both say yes, build a first version and roll it out in stages. When they disagree, believe the behaviour and go back to the interviews to find out why.\n\n## Which method to use when\n\n- **You have a feature request from a big customer.** Problem interviews with five to ten other customers in the same segment. One loud customer is not a market.\n- **You have an idea nobody asked for.** Concept test first. If people can't explain back what it does, fix the concept before measuring demand.\n- **You have a design and want to know if it works.** Prototype test with users who have the problem.\n- **You have to decide whether it's a paid add-on.** Price test with the people who hold the budget.\n- **You have to justify the roadmap to leadership.** A fake door or staged rollout gives you a number, and interview quotes tell the story behind it.\n\n![A product team at a whiteboard covered in sticky notes, sorting feature ideas before deciding which ones to test first](https://images.unsplash.com/photo-1620325867502-221cfb5faa5f?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3w5NDUzMDN8MHwxfHNlYXJjaHwzfHxwcm9kdWN0JTIwdGVhbSUyMHdoaXRlYm9hcmQlMjBzdGlja3klMjBub3Rlc3xlbnwxfDB8fHwxNzk0Mzc5ODExfDA&ixlib=rb-4.1.0&q=80&w=1080)\n\n_Photo by [Brands&People](https://unsplash.com/@brandsandpeople?utm_source=koji&utm_medium=referral) on [Unsplash](https://unsplash.com/?utm_source=koji&utm_medium=referral)_\n\n## When validation lies to you\n\n- **Politeness.** Users want to be helpful and will say they'd use almost anything. Ask about past behaviour, not future intent.\n- **The wrong sample.** Power users love power features. Screen for the segment the feature is for, not whoever replies first.\n- **Leading prototypes.** A beautiful mockup reads as \"already decided\". Show rough work and ask what's missing.\n- **Small numbers read as precise.** Twelve clicks out of 300 is a signal, not a 4% conversion rate.\n\n## How to run feature validation in Koji\n\nKoji runs the asking half of this pairing as AI interviews, by voice or text. Interviews run in parallel and participants take them when it suits them, so there are no calls to schedule.\n\n1. **Problem interviews with your own users.** Start a study, and the research assistant drafts the brief from your research question. Share one study link in your app or newsletter, or email users from Participants with a personal link each (paid plans). Add screening questions so only the segment you care about takes part.\n2. **Concept tests with images.** Attach a mockup, a screenshot or a sketch to any question, and Koji's follow-up questions are about what the image shows. Set the image to show once, full screen, for 3, 5, 10 or 15 seconds, then ask what people remember: a first-impression test in the same interview.\n3. **Price tests.** Add a price test question with a ladder of 3 to 10 prices. Koji finds the highest price each person would still buy at, asks why, and the report shows the share who would buy at each price with the quotes behind it. It reads best from about 30 interviews and is labelled directional. Setup is in [set up a price test in Koji](/docs/price-test-setup-guide).\n4. **Shuffled choice questions** for prioritising between candidate features, so the first option on the list isn't favoured.\n\n![Koji question editor showing open ended, 1 to 10 scale and single choice questions with follow-up settings](/email/shots/shot-question-types.jpg \"Mix open questions, scales and choices in one interview\")\n\nScreening questions, images on questions and price tests are available on every plan, including Free.\n\n## Validate the next feature this week\n\nPick the feature at the top of your roadmap and write down the one question you can't answer about it. If it's \"is the problem real?\", run ten problem interviews. If it's \"will they click?\", ship a fake door. Most of the time you need both.\n\n[Start a free Koji study](/signup) for the interview half, and see [Koji for product managers](/for/product-managers) for templates built for discovery, concept tests and pricing.\n","category":"Research","lastModified":"2026-11-11T07:20:14.188365+00:00","metaTitle":"Feature Validation Methods Compared for Product Managers","metaDescription":"Problem interviews, concept tests, fake doors, prototype tests, price tests and A/B tests compared: which question each answers, effort, time and when it misleads.","keywords":["feature validation methods","how to validate a feature before building","fake door vs prototype test","feature validation interviews","product manager validation","concept testing"],"aiSummary":"Product managers can validate a feature with six methods: problem interviews (is the problem real), concept tests (do people understand and want it), fake doors (do they click), prototype tests (can they use it), price tests (what would they pay) and staged rollouts or A/B tests (does it move the metric). Most features need two: one method that asks people and one that watches behaviour, typically problem interviews plus a fake door. Pendo found 80% of features are rarely or never used, and Microsoft experiments showed only about a third of ideas improved their target metric.","aiKeywords":["feature validation","fake door test","prototype testing","concept testing","price testing","product managers"],"aiContentType":"comparison","faqItems":[{"answer":"Pair one method that asks people with one that watches what they do. For most features that means problem interviews to confirm the problem and a fake door to see whether people click. Build once both say yes.","question":"What is the best way to validate a feature before building it?"},{"answer":"A fake door measures demand: you add the entry point before the feature exists and count clicks. A prototype test measures usability: people try to complete a task in a clickable design. Run the fake door to learn whether to build, the prototype to learn how.","question":"What is the difference between a fake door test and a prototype test?"},{"answer":"Ten to fifteen problem interviews with the right segment usually show whether a pattern exists, five users find most usability problems in a prototype round, and a price test reads best from about 30 interviews.","question":"How many users do I need to validate a feature?"},{"answer":"Users are polite and say they would use almost anything. Ask about the last time they hit the problem rather than whether they would use a feature, and screen for the segment the feature is for.","question":"Why do feature validation interviews mislead?"},{"answer":"Use an A/B test or staged rollout to confirm a decision the cheaper methods already pointed to and to prove impact on a metric. With low traffic it can run for months without a clear result.","question":"When should a product manager use an A/B test?"}],"relatedTopics":["Feature validation","Fake door testing","Prototype testing","Concept testing","Price testing"]}],"pagination":{"total":1,"returned":1,"offset":0}}