How to Run Usability Testing Surveys That Improve Your Product

Usability testing tells you whether people can use your product. Usability surveys tell you how they felt while using it. The most powerful approach combines both: observe behavior, then have a conversation about the experience.

Traditional usability studies require a facilitator, a lab (or Zoom), scheduling, recording, and hours of analysis per participant. They're valuable but expensive. Most teams can only afford 5-8 participants. That's not enough to detect patterns reliably.

Koji enables a new paradigm: unmoderated usability conversations at scale. Give participants a task, then let Koji's AI interview them about their experience. You get the behavioral data (task success, time-on-task) combined with rich qualitative feedback explaining why they struggled or succeeded.

Core Usability Metrics

System Usability Scale (SUS)

The industry standard 10-question post-test questionnaire. Produces a 0-100 score:

• Below 50: Unacceptable usability
• 50-70: Marginal
• 70-80: Good
• 80-90: Excellent
• Above 90: Best-in-class

Task Success Rate

Percentage of participants who complete a given task successfully. Below 78% indicates a significant usability problem.

Time on Task

How long each task takes. Compare against benchmarks or expert completion times.

Single Ease Question (SEQ)

"How easy was this task?" on a 1-7 scale. Quick, per-task metric. Below 5.5 indicates difficulty.

Building Usability Studies with Koji

Study Design

Pre-Task Context (Open-ended): "Before we start, can you tell me about your experience with [product category]? How often do you use similar tools?"

• Sets baseline familiarity
• Probing depth: 1

Per-Task Block (repeat for each task):

Task Description: "Please try to [specific task]. When you're done (or if you get stuck), come back here and we'll talk about it."

SEQ Score (Scale, 1-7): "How easy or difficult was that task?"

• Labels: 1 = "Very difficult", 7 = "Very easy"
• Anchor probing: Scores below 5 trigger deeper exploration

Task Experience (Open-ended): "Walk me through what you did. What was your experience like?"

• Probing depth: 3 for low SEQ scores, 1 for high scores
• AI instruction: "Focus on specific moments of confusion, delight, or frustration. Get step-by-step walkthroughs of where they got stuck."

Expectations (Open-ended): "Was there anything that didn't work the way you expected?"

• Probing depth: 2
• Captures mental model mismatches

Post-Test Overall:

SUS Score (10 Scale questions): Administer the standard 10 SUS questions as Scale type (1-5 each). Koji can ask these conversationally rather than as a form.

Overall Impression (Open-ended): "Overall, what was your impression of [product]?"

• Probing depth: 2

Comparison (Open-ended): "How does this compare to other tools you've used for [similar purpose]?"

• Probing depth: 1

Multi-Task Studies

For studies with 3-5 tasks, Koji maintains conversational flow between tasks. The AI naturally transitions: "Great, thanks for sharing that. Let's try another task..." This feels more natural than a form that resets after each task.

Analysis

What Koji Reports Generate

• SUS score calculation with industry benchmark comparison
• Per-task SEQ distribution identifying which tasks cause the most difficulty
• Usability problem clustering grouping similar issues across participants
• Severity rating based on frequency and impact (how many people hit this issue, and how much it blocked them)
• Improvement recommendations prioritized by severity
• Verbatim quotes describing specific UX breakdowns

Turning Results Into Action

1. Fix critical blockers first: Tasks with less than 60% success rate or SEQ below 4
2. Address confusion patterns: Where multiple users misinterpret the same element
3. Validate fixes: Re-run the same tasks after changes to confirm improvement
4. Track SUS trend: Measure quarterly to ensure overall usability is improving

Usability Testing Best Practices

Sample size

• Qualitative patterns: 5-8 participants per user segment reveal ~85% of usability problems
• Quantitative confidence: 20+ participants for statistically reliable SUS scores
• Koji makes larger samples affordable, which means you can do both: deep qualitative AND reliable quantitative

Participant selection

• Recruit from your actual user base, not generic testers
• Include a mix of experience levels (new users + power users)
• Match demographics to your actual user population
• Avoid UX professionals (they test differently than real users)

Task design

• Write tasks as goals, not instructions: "Find a product under $50" not "Click the filter button"
• Include 3-5 representative tasks covering core workflows
• Order from easy to hard to build participant confidence
• Include at least one task you expect to be difficult (this is where the real learning happens)

Avoid leading

• Don't tell participants where to click or what to look for
• Don't ask "Did you find the navigation easy?" (leading). Ask "Tell me about the navigation" (neutral).
• Koji's AI is trained to probe without leading, asking "What did you expect to happen?" rather than "Was it confusing?"

Why Koji Beats Traditional Usability Testing Tools

Koji doesn't replace screen recording for observing behavior. But it dramatically improves the post-task interview that reveals why users behaved the way they did, and it does so at a scale that makes statistically reliable usability data accessible to every team.