{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-05-18T13:56:30.203Z"},"content":[{"type":"documentation","id":"dbf64e67-b0f5-4294-ae21-a90455cffd5f","slug":"concept-testing-survey-guide","title":"How to Run Concept Testing Surveys Before You Build the Wrong Thing","url":"https://www.koji.so/docs/concept-testing-survey-guide","summary":"Comprehensive guide to concept testing surveys covering monadic, sequential monadic, comparative, and proto-monadic methodologies. Includes purchase intent scales, concept scoring frameworks, stimulus design, and how Koji AI interviews uncover hidden objections and emotional reactions.","content":"# How to Run Concept Testing Surveys Before You Build the Wrong Thing\n\nThe graveyard of failed products is full of ideas that seemed brilliant in the conference room. Google Glass. Amazon Fire Phone. Quibi. Each had enormous resources behind it. Each failed because the concept did not resonate with real users in real contexts.\n\nConcept testing is the discipline of evaluating ideas before you commit resources to building them. It answers the fundamental question: does this concept solve a real problem for real people in a way they find compelling? And it answers it before you spend months and millions on development.\n\nBut traditional concept testing has a critical flaw. Surveys measure surface-level reactions, stated preferences, and top-of-mind responses. They miss the deeper emotional responses, hidden objections, and contextual factors that actually determine whether someone will adopt a new product.\n\nKoji solves this by combining structured concept evaluation with conversational depth. The AI interviewer presents concepts with standardized measurement questions, then probes for emotional reactions, usage context, objections, and comparisons that reveal whether a concept has genuine potential or just polite approval.\n\n## Why Concept Testing Matters\n\nConcept testing sits between ideation and development. It is the critical validation step that most teams either skip entirely or do superficially.\n\n**Without concept testing, you risk:**\n- Building a product nobody wants (the most expensive mistake)\n- Launching a marketing message that does not resonate\n- Pricing at a level the market will not support\n- Missing a critical objection that kills adoption\n- Targeting the wrong audience with the right product\n\n**With proper concept testing, you gain:**\n- Validated demand before engineering commitment\n- Optimized messaging and positioning\n- Identified adoption barriers and objections\n- Comparative data on multiple concepts\n- Segment-level insights on who wants what\n\n## Concept Testing Methodologies\n\n### Monadic Testing\n\nIn monadic testing, each respondent evaluates only one concept. This eliminates comparison bias and gives you a pure reaction to each concept.\n\n**Advantages:**\n- No order effects or comparison bias\n- Captures the natural first reaction\n- Simulates real-world experience where users encounter one product at a time\n- Shorter survey per respondent\n\n**Disadvantages:**\n- Requires larger sample sizes (each concept needs its own group)\n- Cannot directly compare concepts within the same respondent\n- More expensive to recruit for\n\n**When to use monadic testing:**\n- Early-stage concepts where you need unbiased reactions\n- Testing messaging or positioning where framing effects are strong\n- When you have budget for larger sample sizes\n- When concepts are in the same category and comparison would create anchoring\n\n### Sequential Monadic Testing\n\nEach respondent evaluates multiple concepts, one at a time, in a randomized order. This combines some benefits of monadic testing with the efficiency of within-subjects design.\n\n**Advantages:**\n- Smaller sample size needed\n- Can collect relative preference data\n- More cost-efficient\n\n**Disadvantages:**\n- Order effects (first concept seen often rated higher)\n- Comparison effects alter ratings of subsequent concepts\n- Longer survey per respondent, leading to fatigue\n\n**When to use sequential monadic:**\n- When you need to compare 2-4 concepts directly\n- When budget constrains sample size\n- When you want both independent ratings AND relative preference\n\n### Comparative Testing\n\nRespondents see all concepts simultaneously and compare them directly.\n\n**Advantages:**\n- Most efficient for identifying a winner\n- Forces trade-off thinking\n\n**Disadvantages:**\n- Heavy comparison bias\n- Does not simulate real-world evaluation\n- Only tells you which is best among options, not if any are good enough\n\n### Proto-Monadic Testing\n\nA hybrid approach: respondents first evaluate one concept monadically (pure reaction), then see the other concepts for comparison. This gives you both unbiased individual reactions and comparative preference data.\n\nThis is ideal for Koji because the AI interviewer can seamlessly transition: \"Now I'd like to show you a few other concepts we're considering. Having seen them all, which one appeals to you most and why?\"\n\n## Concept Scoring Frameworks\n\n### The Five-Point Purchase Intent Scale\n\nThe most widely used concept evaluation metric:\n\n**Purchase Intent (Single Choice):**\n\"Based on what you've seen, how likely would you be to [buy/use/subscribe to] this product?\"\n- Definitely would\n- Probably would\n- Might or might not\n- Probably would not\n- Definitely would not\n\n**Interpreting Purchase Intent:**\n- **Top Box (Definitely would)**: Your core audience. Typically 10-20% for strong concepts.\n- **Top 2 Box (Definitely + Probably)**: Your addressable market. Strong concepts score 50%+.\n- Industry benchmarks vary dramatically by category. Compare to your own historical data when possible.\n- **Critical insight**: Purchase intent consistently over-predicts actual purchase behavior. Apply a discount factor (typically 0.3-0.5x for \"definitely would\" and 0.1-0.2x for \"probably would\").\n\nKoji's AI follow-up adds essential context: \"You said you would probably buy this. What would need to be true for that to become a definite yes?\" This captures the conditions and objections that purchase intent alone misses.\n\n### Concept Appeal Score\n\nA multi-dimensional evaluation of concept strength:\n\n**Uniqueness (Scale 1-5):**\n\"How different is this product from anything else available?\"\n\n**Relevance (Scale 1-5):**\n\"How relevant is this product to your needs?\"\n\n**Believability (Scale 1-5):**\n\"How believable are the claims made about this product?\"\n\n**Appeal Score = (Uniqueness + Relevance + Believability) / 3**\n\nStrong concepts score above 3.5. Concepts below 3.0 need fundamental rethinking. The diagnostic power is in the individual dimensions: a concept can be highly unique but not relevant (a solution looking for a problem) or highly relevant but not believable (claims that seem too good to be true).\n\n### Concept Strength Index\n\nA more comprehensive framework used by major CPG and tech companies:\n\n**Novelty (Scale 1-7):**\n\"This product offers something I can't get anywhere else\"\n\n**Desirability (Scale 1-7):**\n\"I would really like to have this product\"\n\n**Credibility (Scale 1-7):**\n\"I believe this product can deliver what it promises\"\n\n**Value (Scale 1-7):**\n\"This product would be worth the price\"\n\n**Concept Strength Index = (Novelty x 0.2) + (Desirability x 0.3) + (Credibility x 0.25) + (Value x 0.25)**\n\nWeight desirability highest because want is the strongest predictor of adoption.\n\n## Designing Concept Stimuli\n\n### What to Include in a Concept Description\n\nA concept stimulus should contain:\n1. **Headline**: The core promise in one sentence\n2. **Insight**: The problem or need being addressed\n3. **Benefit statement**: How the product solves the problem\n4. **Key features**: 3-5 supporting features (not exhaustive)\n5. **Price point** (if testing price): Include to anchor realistic evaluation\n6. **Visual** (if available): Mockup, rendering, or prototype screenshot\n\n### What to Avoid\n\n- **Overselling**: Keep the language neutral and factual\n- **Too much detail**: Concepts should be digestible in 30-60 seconds\n- **Jargon**: Use language your target audience uses naturally\n- **Multiple ideas in one concept**: Test one core proposition per concept\n- **Finished-looking designs**: These bias toward visual evaluation rather than concept evaluation\n\n### Concept Stimulus Example\n\n> **TaskFlow AI** — Your tasks, prioritized by AI\n>\n> *You have too many tasks and not enough clarity on what matters most.*\n>\n> TaskFlow AI analyzes your calendar, deadlines, dependencies, and energy patterns to automatically prioritize your daily task list. It learns what you procrastinate on, what blocks others, and what has the highest impact.\n>\n> Key features:\n> - AI-prioritized daily task list based on your patterns\n> - Automatic deadline and dependency detection\n> - Energy-based scheduling (hard tasks when you are most focused)\n> - One-click daily planning that adapts in real time\n>\n> Starting at $12/month\n\n## Structuring a Koji Concept Test\n\n### For Monadic Testing\n\n**Step 1 - Screener:**\nVerify the respondent is in your target audience with 2-3 qualification questions.\n\n**Step 2 - Context Setting:**\n\"I'm going to share a product concept with you. Please read it carefully and share your honest reaction. There are no right or wrong answers.\"\n\n**Step 3 - Concept Presentation:**\nPresent the concept stimulus (Koji supports rich text and images in the study guide).\n\n**Step 4 - Structured Evaluation:**\n\n**Overall Reaction (Scale 1-7):**\n\"What is your overall reaction to this concept?\"\n(1 = Very negative, 7 = Very positive)\n\n**Purchase Intent (Single Choice):**\n\"How likely would you be to try this product?\"\n- Definitely would try\n- Probably would try\n- Might or might not try\n- Probably would not try\n- Definitely would not try\n\n**Uniqueness (Scale 1-5):**\n\"How unique or different is this concept from what is currently available?\"\n\n**Relevance (Scale 1-5):**\n\"How relevant is this concept to your needs?\"\n\n**Step 5 - AI Qualitative Probing:**\n\nThis is where Koji transforms concept testing. The AI follows up on every structured response:\n\n- On high ratings: \"You gave this a 6 out of 7. What specifically excited you about it?\"\n- On low ratings: \"You rated the relevance as a 2. Is this solving a problem you do not actually have, or is the solution approach wrong?\"\n- On purchase intent: \"You said you probably would not try this. What is your main hesitation?\"\n- Probing for objections: \"Is there anything about this concept that concerns you or seems unrealistic?\"\n- Usage context: \"If this product existed today, when during your week would you use it?\"\n- Competitive framing: \"How does this compare to what you currently use for [task]?\"\n\n**Step 6 - Open Feedback:**\n\"Is there anything else about this concept you'd like to share?\"\n\n### For Sequential Monadic Testing\n\nFollow the same structure but present 2-4 concepts sequentially (randomize order). After all individual evaluations, add:\n\n**Preference (Ranking):**\n\"Now that you've seen all the concepts, please rank them from most to least appealing.\"\n\n**Preference Reasoning (Open-ended):**\n\"Why did you rank [top choice] first?\"\n\n## Analyzing Concept Testing Data\n\n### Quantitative Analysis\n\n1. **Calculate Top Box and Top 2 Box** purchase intent percentages\n2. **Calculate mean scores** for each evaluation dimension\n3. **Compare concepts** on a radar chart (novelty, desirability, credibility, value)\n4. **Identify the strongest concept** across multiple metrics, not just purchase intent\n5. **Segment the data**: Which concept wins with which audience?\n\n### Qualitative Analysis from Koji\n\nThe AI follow-up conversations reveal:\n\n- **Hidden objections**: \"I like the idea but I would never pay monthly for a task manager\"\n- **Missing features**: \"This would need to integrate with Slack for me to consider it\"\n- **Emotional reactions**: \"This sounds exactly like what I've been looking for\" vs. \"Meh, another productivity tool\"\n- **Use case mapping**: Specific workflows and scenarios where users envision using the product\n- **Competitive positioning**: How users mentally categorize the concept relative to alternatives\n\n### The Concept Optimization Matrix\n\nCreate a 2x2 matrix:\n- **X-axis**: Purchase intent (low to high)\n- **Y-axis**: Uniqueness (low to high)\n\n| | Low Purchase Intent | High Purchase Intent |\n|---|---|---|\n| **High Uniqueness** | Niche opportunity: unique but limited appeal. Investigate segment-specific potential. | Sweet spot: unique and wanted. Prioritize development. |\n| **Low Uniqueness** | Kill: neither unique nor wanted. Move on. | Me-too risk: wanted but not differentiated. Needs stronger positioning. |\n\n## Common Concept Testing Mistakes\n\n### Mistake 1: Testing Concepts That Are Too Abstract\nConcepts need enough specificity for meaningful evaluation. \"An AI tool that helps you work better\" is too vague. Include concrete features and a price point.\n\n### Mistake 2: Ignoring Non-Buyers\nThe \"definitely would not\" group is as informative as the \"definitely would\" group. Their objections reveal fatal flaws and boundary conditions for your market. Koji's AI specifically probes this group: \"What would this product need to change for you to consider it?\"\n\n### Mistake 3: Optimizing for Purchase Intent Alone\nA concept with 60% purchase intent and 20% uniqueness will face brutal competition. A concept with 40% purchase intent and 80% uniqueness might be a category creator. Look at the full profile.\n\n### Mistake 4: Testing with the Wrong Audience\nConcept testing with a general population when your product targets data engineers will give you meaningless results. Screen rigorously.\n\n### Mistake 5: Only Testing Concepts You Like\nInclude a \"throwaway\" concept that you expect to lose. If it wins, you have learned something invaluable. If it loses, it validates your instincts with data.\n\n## When to Run Concept Testing\n\n- **Before product development**: Test 3-5 concepts to identify the strongest direction\n- **Before marketing campaigns**: Test messaging and positioning variations\n- **Before line extensions**: Test new flavors, models, or tiers with existing customers\n- **Before market entry**: Test localized concepts in new geographic markets\n- **Before pivot decisions**: Test the new direction against the current one\n\n## The Bottom Line\n\nConcept testing is the cheapest insurance against building the wrong product. A well-designed concept test costs a fraction of a single sprint of engineering work, yet it can prevent months of wasted development.\n\nKoji elevates concept testing from a checkbox exercise to genuine discovery. Traditional surveys tell you that 45% of respondents \"probably would\" try the product. Koji tells you that AND why, along with the specific objections that would stop them, the features they would need to commit, the competitors they would compare you to, and the exact use case where they see the product fitting.\n\nThat is the difference between launching with hope and launching with conviction.\n\n---\n\n## Related Survey Guides\n\n- [Product-Market Fit Guide](/docs/product-market-fit-survey-guide) — Measure overall market fit\n- [Feature Prioritization Guide](/docs/feature-prioritization-survey-guide) — Prioritize what to build\n- [Beta Testing Feedback Guide](/docs/beta-testing-feedback-survey-guide) — Validate before launch\n- [Pricing Research Guide](/docs/pricing-research-survey-guide) — Test pricing alongside concepts\n- [Buyer Persona Guide](/docs/buyer-persona-survey-guide) — Understand your target audience\n\n*Use [structured questions](/docs/structured-questions-guide) to combine concept scoring scales with open-ended AI probing.*\n\n## Further reading on the blog\n\n- [Concept Testing: The Complete Guide for Product Teams (2026)](/blog/concept-testing-guide-2026) — Concept testing validates whether your idea is worth building before you build it. This guide covers methods, question templates, analysis a\n- [Best Online Survey Software in 2026: The Complete Buyer's Guide](/blog/best-survey-software-2026) — From SurveyMonkey to Koji, we compare the top survey tools of 2026 across features, pricing, and use case fit — and explain when traditional\n- [Can I Paste User Interviews into ChatGPT? A Guide to GDPR and LLMs](/blog/can-i-paste-user-interviews-into-chatgpt-a-guide-to-gdpr-and-llms) — Every product manager wants to ask an LLM about their user feedback. But pasting customer transcripts into public models is a GDPR nightmare\n\n<!-- further-reading:blog -->\n","category":"Survey & Study Templates","lastModified":"2026-05-13T00:26:36.807295+00:00","metaTitle":"Concept Testing Survey Guide: Monadic, Sequential & Scoring Frameworks | Koji","metaDescription":"Complete guide to concept testing surveys. Learn monadic vs sequential testing, purchase intent scales, concept scoring frameworks, and how AI interviews uncover hidden objections before you build.","keywords":["concept testing survey","concept test template","monadic testing","sequential monadic testing","purchase intent scale","concept scoring","product concept testing","idea validation survey"],"aiSummary":"Comprehensive guide to concept testing surveys covering monadic, sequential monadic, comparative, and proto-monadic methodologies. Includes purchase intent scales, concept scoring frameworks, stimulus design, and how Koji AI interviews uncover hidden objections and emotional reactions."}],"pagination":{"total":1,"returned":1,"offset":0}}