How to Run Concept Testing Surveys Before You Build the Wrong Thing

The graveyard of failed products is full of ideas that seemed brilliant in the conference room. Google Glass. Amazon Fire Phone. Quibi. Each had enormous resources behind it. Each failed because the concept did not resonate with real users in real contexts.

Concept testing is the discipline of evaluating ideas before you commit resources to building them. It answers the fundamental question: does this concept solve a real problem for real people in a way they find compelling? And it answers it before you spend months and millions on development.

But traditional concept testing has a critical flaw. Surveys measure surface-level reactions, stated preferences, and top-of-mind responses. They miss the deeper emotional responses, hidden objections, and contextual factors that actually determine whether someone will adopt a new product.

Koji solves this by combining structured concept evaluation with conversational depth. The AI interviewer presents concepts with standardized measurement questions, then probes for emotional reactions, usage context, objections, and comparisons that reveal whether a concept has genuine potential or just polite approval.

Why Concept Testing Matters

Concept testing sits between ideation and development. It is the critical validation step that most teams either skip entirely or do superficially.

Without concept testing, you risk:

• Building a product nobody wants (the most expensive mistake)
• Launching a marketing message that does not resonate
• Pricing at a level the market will not support
• Missing a critical objection that kills adoption
• Targeting the wrong audience with the right product

With proper concept testing, you gain:

• Validated demand before engineering commitment
• Optimized messaging and positioning
• Identified adoption barriers and objections
• Comparative data on multiple concepts
• Segment-level insights on who wants what

Concept Testing Methodologies

Monadic Testing

In monadic testing, each respondent evaluates only one concept. This eliminates comparison bias and gives you a pure reaction to each concept.

Advantages:

• No order effects or comparison bias
• Captures the natural first reaction
• Simulates real-world experience where users encounter one product at a time
• Shorter survey per respondent

Disadvantages:

• Requires larger sample sizes (each concept needs its own group)
• Cannot directly compare concepts within the same respondent
• More expensive to recruit for

When to use monadic testing:

• Early-stage concepts where you need unbiased reactions
• Testing messaging or positioning where framing effects are strong
• When you have budget for larger sample sizes
• When concepts are in the same category and comparison would create anchoring

Sequential Monadic Testing

Each respondent evaluates multiple concepts, one at a time, in a randomized order. This combines some benefits of monadic testing with the efficiency of within-subjects design.

Advantages:

• Smaller sample size needed
• Can collect relative preference data
• More cost-efficient

Disadvantages:

• Order effects (first concept seen often rated higher)
• Comparison effects alter ratings of subsequent concepts
• Longer survey per respondent, leading to fatigue

When to use sequential monadic:

• When you need to compare 2-4 concepts directly
• When budget constrains sample size
• When you want both independent ratings AND relative preference

Comparative Testing

Respondents see all concepts simultaneously and compare them directly.

Advantages:

• Most efficient for identifying a winner
• Forces trade-off thinking

Disadvantages:

• Heavy comparison bias
• Does not simulate real-world evaluation
• Only tells you which is best among options, not if any are good enough

Proto-Monadic Testing

A hybrid approach: respondents first evaluate one concept monadically (pure reaction), then see the other concepts for comparison. This gives you both unbiased individual reactions and comparative preference data.

This is ideal for Koji because the AI interviewer can seamlessly transition: "Now I'd like to show you a few other concepts we're considering. Having seen them all, which one appeals to you most and why?"

Concept Scoring Frameworks

The Five-Point Purchase Intent Scale

The most widely used concept evaluation metric:

Purchase Intent (Single Choice): "Based on what you've seen, how likely would you be to [buy/use/subscribe to] this product?"

• Definitely would
• Probably would
• Might or might not
• Probably would not
• Definitely would not

Interpreting Purchase Intent:

• Top Box (Definitely would): Your core audience. Typically 10-20% for strong concepts.
• Top 2 Box (Definitely + Probably): Your addressable market. Strong concepts score 50%+.
• Industry benchmarks vary dramatically by category. Compare to your own historical data when possible.
• Critical insight: Purchase intent consistently over-predicts actual purchase behavior. Apply a discount factor (typically 0.3-0.5x for "definitely would" and 0.1-0.2x for "probably would").

Koji's AI follow-up adds essential context: "You said you would probably buy this. What would need to be true for that to become a definite yes?" This captures the conditions and objections that purchase intent alone misses.

Concept Appeal Score

A multi-dimensional evaluation of concept strength:

Uniqueness (Scale 1-5): "How different is this product from anything else available?"

Relevance (Scale 1-5): "How relevant is this product to your needs?"

Believability (Scale 1-5): "How believable are the claims made about this product?"

Appeal Score = (Uniqueness + Relevance + Believability) / 3

Strong concepts score above 3.5. Concepts below 3.0 need fundamental rethinking. The diagnostic power is in the individual dimensions: a concept can be highly unique but not relevant (a solution looking for a problem) or highly relevant but not believable (claims that seem too good to be true).

Concept Strength Index

A more comprehensive framework used by major CPG and tech companies:

Novelty (Scale 1-7): "This product offers something I can't get anywhere else"

Desirability (Scale 1-7): "I would really like to have this product"

Credibility (Scale 1-7): "I believe this product can deliver what it promises"

Value (Scale 1-7): "This product would be worth the price"

Concept Strength Index = (Novelty x 0.2) + (Desirability x 0.3) + (Credibility x 0.25) + (Value x 0.25)

Weight desirability highest because want is the strongest predictor of adoption.

Designing Concept Stimuli

What to Include in a Concept Description

A concept stimulus should contain:

1. Headline: The core promise in one sentence
2. Insight: The problem or need being addressed
3. Benefit statement: How the product solves the problem
4. Key features: 3-5 supporting features (not exhaustive)
5. Price point (if testing price): Include to anchor realistic evaluation
6. Visual (if available): Mockup, rendering, or prototype screenshot

What to Avoid

• Overselling: Keep the language neutral and factual
• Too much detail: Concepts should be digestible in 30-60 seconds
• Jargon: Use language your target audience uses naturally
• Multiple ideas in one concept: Test one core proposition per concept
• Finished-looking designs: These bias toward visual evaluation rather than concept evaluation

Concept Stimulus Example

TaskFlow AI — Your tasks, prioritized by AI

You have too many tasks and not enough clarity on what matters most.

TaskFlow AI analyzes your calendar, deadlines, dependencies, and energy patterns to automatically prioritize your daily task list. It learns what you procrastinate on, what blocks others, and what has the highest impact.

Key features:

• AI-prioritized daily task list based on your patterns
• Automatic deadline and dependency detection
• Energy-based scheduling (hard tasks when you are most focused)
• One-click daily planning that adapts in real time

Starting at $12/month

Structuring a Koji Concept Test

For Monadic Testing

Step 1 - Screener: Verify the respondent is in your target audience with 2-3 qualification questions.

Step 2 - Context Setting: "I'm going to share a product concept with you. Please read it carefully and share your honest reaction. There are no right or wrong answers."

Step 3 - Concept Presentation: Present the concept stimulus (Koji supports rich text and images in the study guide).

Step 4 - Structured Evaluation:

Overall Reaction (Scale 1-7): "What is your overall reaction to this concept?" (1 = Very negative, 7 = Very positive)

Purchase Intent (Single Choice): "How likely would you be to try this product?"

• Definitely would try
• Probably would try
• Might or might not try
• Probably would not try
• Definitely would not try

Uniqueness (Scale 1-5): "How unique or different is this concept from what is currently available?"

Relevance (Scale 1-5): "How relevant is this concept to your needs?"

Step 5 - AI Qualitative Probing:

This is where Koji transforms concept testing. The AI follows up on every structured response:

• On high ratings: "You gave this a 6 out of 7. What specifically excited you about it?"
• On low ratings: "You rated the relevance as a 2. Is this solving a problem you do not actually have, or is the solution approach wrong?"
• On purchase intent: "You said you probably would not try this. What is your main hesitation?"
• Probing for objections: "Is there anything about this concept that concerns you or seems unrealistic?"
• Usage context: "If this product existed today, when during your week would you use it?"
• Competitive framing: "How does this compare to what you currently use for [task]?"

Step 6 - Open Feedback: "Is there anything else about this concept you'd like to share?"

For Sequential Monadic Testing

Follow the same structure but present 2-4 concepts sequentially (randomize order). After all individual evaluations, add:

Preference (Ranking): "Now that you've seen all the concepts, please rank them from most to least appealing."

Preference Reasoning (Open-ended): "Why did you rank [top choice] first?"

Analyzing Concept Testing Data

Quantitative Analysis

1. Calculate Top Box and Top 2 Box purchase intent percentages
2. Calculate mean scores for each evaluation dimension
3. Compare concepts on a radar chart (novelty, desirability, credibility, value)
4. Identify the strongest concept across multiple metrics, not just purchase intent
5. Segment the data: Which concept wins with which audience?

Qualitative Analysis from Koji

The AI follow-up conversations reveal:

• Hidden objections: "I like the idea but I would never pay monthly for a task manager"
• Missing features: "This would need to integrate with Slack for me to consider it"
• Emotional reactions: "This sounds exactly like what I've been looking for" vs. "Meh, another productivity tool"
• Use case mapping: Specific workflows and scenarios where users envision using the product
• Competitive positioning: How users mentally categorize the concept relative to alternatives

The Concept Optimization Matrix

Create a 2x2 matrix:

• X-axis: Purchase intent (low to high)
• Y-axis: Uniqueness (low to high)

Common Concept Testing Mistakes

Mistake 1: Testing Concepts That Are Too Abstract

Concepts need enough specificity for meaningful evaluation. "An AI tool that helps you work better" is too vague. Include concrete features and a price point.

Mistake 2: Ignoring Non-Buyers

The "definitely would not" group is as informative as the "definitely would" group. Their objections reveal fatal flaws and boundary conditions for your market. Koji's AI specifically probes this group: "What would this product need to change for you to consider it?"

Mistake 3: Optimizing for Purchase Intent Alone

A concept with 60% purchase intent and 20% uniqueness will face brutal competition. A concept with 40% purchase intent and 80% uniqueness might be a category creator. Look at the full profile.

Mistake 4: Testing with the Wrong Audience

Concept testing with a general population when your product targets data engineers will give you meaningless results. Screen rigorously.

Mistake 5: Only Testing Concepts You Like

Include a "throwaway" concept that you expect to lose. If it wins, you have learned something invaluable. If it loses, it validates your instincts with data.

When to Run Concept Testing

• Before product development: Test 3-5 concepts to identify the strongest direction
• Before marketing campaigns: Test messaging and positioning variations
• Before line extensions: Test new flavors, models, or tiers with existing customers
• Before market entry: Test localized concepts in new geographic markets
• Before pivot decisions: Test the new direction against the current one

The Bottom Line

Concept testing is the cheapest insurance against building the wrong product. A well-designed concept test costs a fraction of a single sprint of engineering work, yet it can prevent months of wasted development.

Koji elevates concept testing from a checkbox exercise to genuine discovery. Traditional surveys tell you that 45% of respondents "probably would" try the product. Koji tells you that AND why, along with the specific objections that would stop them, the features they would need to commit, the competitors they would compare you to, and the exact use case where they see the product fitting.

That is the difference between launching with hope and launching with conviction.