{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-06-05T06:02:49.826Z"},"content":[{"type":"documentation","id":"a7c4799b-0ada-41f8-a2f1-29d6402bbb50","slug":"usability-testing-questions","title":"Usability Testing Questions: What to Ask Before, During, and After a Test","url":"https://www.koji.so/docs/usability-testing-questions","summary":"A phase-by-phase bank of usability testing questions: pre-test screening and warm-up, neutral in-task think-aloud probes, and post-task/post-test reflection plus standardized metrics (SUS, SEQ, NPS). Covers the rules for asking non-leading questions, cites Nielsen's five-user / 85%-of-problems finding and Steve Krug, and explains how AI-moderated testing with Koji runs the same script at scale with consistent probing.","content":"Usability testing questions fall into three phases: **pre-test** questions that screen and warm up the participant, **in-task** questions and probes that capture thinking while they use the product, and **post-test** questions that measure satisfaction and gather reflections. The golden rule is to ask open, non-leading questions and to *let users struggle* - your job is to observe behavior, not to coach. This guide gives you ready-to-use questions for every phase, the think-aloud probes that get users talking, the standardized post-test scales (SUS, SEQ, NPS), and how AI-moderated testing lets you run the same script with dozens of users at once.\n\n## Why the Questions Matter More Than You Think\n\nIn usability testing you are watching what people *do*, but the questions you ask shape what you learn. Ask a leading question and you contaminate the result. As usability expert Steve Krug writes in *Rocket Surgery Made Easy*, \"Testing one user is 100 percent better than testing none\" - the method is forgiving, but only if you stay neutral and let the user reveal problems on their own.\n\nThe payoff is large. Jakob Nielsen, co-founder of the Nielsen Norman Group, famously found that testing with just five users uncovers roughly [85% of the usability problems](https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/) in an interface, because the same core issues recur quickly. The questions below are designed to surface those issues fast.\n\n## Pre-Test Questions (Screening and Warm-Up)\n\nBefore the test begins, confirm the participant fits your target user and set them at ease. Make it clear you are testing the product, not them.\n\n**Screening / background:**\n- How often do you use [product category]? (daily, weekly, monthly, rarely)\n- Walk me through the last time you tried to [accomplish the core task].\n- What tools or apps do you currently use for this?\n- What is your role, and how does [task] fit into your day?\n- Have you used [our product] before? If so, how often?\n\n**Warm-up and framing:**\n- There are no right or wrong answers - we are testing the design, not you. Does that make sense?\n- As you go, please think out loud and tell me what you are looking at and expecting.\n- Before we start, what do you expect a product like this to help you do?\n- If you get stuck, that is exactly the kind of thing we want to see - please do not worry about it.\n\n## In-Task Questions and Think-Aloud Probes\n\nDuring tasks, talk as little as possible. When you do speak, use neutral probes that keep the user narrating without steering them. Never answer \"Where would you click?\" - turn it back to them.\n\n**Setting up a task (use scenarios, not instructions):**\n- Say you wanted to [goal] - show me how you would do that.\n- You just received [situation]. What would you do next?\n- Imagine it is your first day using this. Where would you begin?\n\n**Neutral in-task probes:**\n- What are you thinking right now?\n- What did you expect to happen when you clicked that?\n- Tell me what you are looking for.\n- How does that compare to what you expected?\n- What would you do next?\n- You went quiet - what is going through your mind?\n- Is this what you thought it would do?\n\n**When a user gets stuck (do not rescue too early):**\n- What are you trying to do here?\n- What would you expect to see at this point?\n- If I were not here, what would you do next?\n- On a scale of 1-7, how difficult was that task? (the Single Ease Question)\n\n**Echo and silence techniques:**\n- Repeat their last few words as a question (\"It is confusing?\") to prompt elaboration.\n- Stay silent for a few seconds - most people fill the gap with exactly the insight you need.\n\n## Post-Task and Post-Test Questions\n\nAfter each task, capture the immediate reaction. After the full session, gather overall impressions and standardized metrics.\n\n**Post-task (ask right after each task):**\n- How easy or difficult was that to complete? Why?\n- Was anything surprising or confusing?\n- Did the product behave the way you expected?\n- What, if anything, would have made that easier?\n\n**Post-test reflection:**\n- What was your overall impression?\n- What was the most frustrating part of the experience?\n- What did you like most?\n- If you had a magic wand, what one thing would you change?\n- Would you use this in your real work or life? Why or why not?\n- How would you describe this product to a friend?\n- Is there anything you expected to find that was missing?\n\n**Standardized post-test metrics (so you can benchmark over time):**\n- **System Usability Scale (SUS):** 10 standardized agree/disagree statements producing a 0-100 score - the industry standard for comparing usability across versions.\n- **Single Ease Question (SEQ):** a single 1-7 rating of task difficulty, asked immediately after each task.\n- **Net Promoter Score (NPS):** \"How likely are you to recommend this to a friend or colleague?\" on a 0-10 scale.\n\n## The Rules for Asking Usability Questions\n\n- **Never lead.** \"Was that easy?\" presumes the answer. Ask \"How was that?\"\n- **Do not ask users to design.** \"What color should this button be?\" yields opinions, not usability data. Watch what trips them up instead.\n- **Separate behavior from preference.** What people *do* is more reliable than what they *say* they prefer.\n- **Ask \"why,\" not \"do you like.\"** Likeability is noisy; the reasoning behind a struggle is gold.\n- **Stay quiet during tasks.** Every word you say is a potential bias. Let silence do the work.\n- **Distinguish observed problems from reported ones.** A user saying \"this is fine\" while clicking the wrong thing three times is a usability problem, regardless of what they report.\n\n## Running Usability Questions at Scale With AI\n\nTraditional moderated usability testing is powerful but slow: you schedule one session at a time, take notes, and manually synthesize. Nielsen recommends iterative rounds of five users, which is sound - but each round still consumes hours of a researcher's week.\n\n**Koji**, an AI-native research platform, lets you script these exact questions once and have an AI moderator run them with many participants in parallel. The AI asks your pre-test, in-task, and post-test questions, listens to the responses, and - critically - probes intelligently: when a participant says \"this was confusing,\" Koji follows up with \"What specifically confused you?\" the way a trained moderator would, on every session.\n\n- **Consistent moderation, zero moderator bias.** Every participant gets the same neutral script, so you never accidentally lead one user and not another.\n- **Voice or text think-aloud.** Voice interviews capture tone and hesitation; text interviews scale to large, distributed audiences.\n- **Built-in metrics with structured questions.** Embed your SEQ, SUS items, and NPS directly using Koji's six structured question types - open_ended, scale, single_choice, multiple_choice, ranking, and yes_no - so qualitative reactions and quantitative scores live in one study and report. See the [structured questions guide](/docs/structured-questions-guide).\n- **Automatic synthesis.** Themes, friction points, and representative quotes are aggregated for you, turning a week of note-taking into a same-day report.\n\nYou still get the depth of moderated testing - the open-ended \"why\" behind every struggle - without being the bottleneck. As the research democratizes, you no longer need a dedicated lab or a research team to ask great usability questions of a meaningful sample.\n\n## Putting It Together: A Sample Usability Test Script\n\n1. **Intro (3 min):** \"We are testing the design, not you. Please think out loud.\"\n2. **Pre-test (3 min):** 2-3 screening/background questions.\n3. **Tasks (20-30 min):** 3-5 scenario-based tasks, each followed by the SEQ and a \"how was that?\" probe.\n4. **Post-test (10 min):** reflection questions + SUS + NPS.\n5. **Wrap (2 min):** \"Anything we did not ask that you want to share?\"\n\nRun it with five users per round if you moderate manually, or with 20-50 in parallel if you let Koji moderate - either way, this script will surface the issues that matter.\n\n## Tailoring Questions to the Type of Test\n\nNot every usability test uses the same questions. Adjust the set to the format:\n\n- **Moderated vs unmoderated:** In a moderated session you can ask live probes (\"What are you thinking?\"). In an unmoderated test, bake those probes into the task instructions and add a post-task open-ended box, since you cannot react in real time.\n- **Prototype vs live product:** With an early prototype, frame questions around expectation (\"What would you expect this button to do?\") rather than outcome, because flows may be incomplete. With a live product, focus on actual task completion and friction.\n- **First-time vs returning users:** Ask new users about discoverability and first impressions; ask returning users about efficiency, shortcuts, and what slows them down.\n- **Desktop vs mobile:** On mobile, add questions about reachability, typing effort, and interruptions (\"Would you finish this on your phone, or switch to a computer?\").\n\nMatching the question set to the test type keeps every prompt relevant and stops you from asking about flows the user could not realistically complete.\n\n## Related Resources\n\n- [Structured Questions Guide](/docs/structured-questions-guide) - embed SEQ, SUS, and NPS with 6 question types\n- [Usability Testing Guide](/docs/usability-testing-guide) - the full methodology end to end\n- [Usability Testing Script Template](/docs/usability-testing-script-template) - a ready-to-edit moderator script\n- [Think-Aloud Protocol](/docs/think-aloud-protocol) - the technique behind in-task narration\n- [System Usability Scale Guide](/docs/system-usability-scale-guide) - scoring and benchmarking SUS\n- [Single Ease Question (SEQ) Guide](/docs/single-ease-question-seq-guide) - the fastest per-task usability metric","category":"Research Methods","lastModified":"2026-06-05T03:17:53.002547+00:00","metaTitle":"Usability Testing Questions: 40+ to Ask (Before, During, After)","metaDescription":"Usability testing questions for the pre-test, in-task, and post-test phases - think-aloud probes, screening, SUS, SEQ, and a faster AI-moderated approach with Koji.","keywords":["usability testing questions","usability test questions to ask","think aloud questions","post-test usability questions","moderated usability testing questions","usability testing script questions"],"aiSummary":"A phase-by-phase bank of usability testing questions: pre-test screening and warm-up, neutral in-task think-aloud probes, and post-task/post-test reflection plus standardized metrics (SUS, SEQ, NPS). Covers the rules for asking non-leading questions, cites Nielsen's five-user / 85%-of-problems finding and Steve Krug, and explains how AI-moderated testing with Koji runs the same script at scale with consistent probing.","aiPrerequisites":["usability-testing-guide"],"aiLearningOutcomes":["Ask non-leading questions across pre-test, in-task, and post-test phases","Use think-aloud probes that surface friction without steering users","Apply standardized metrics like SUS, SEQ, and NPS","Scale moderated usability questions with AI moderation"],"aiDifficulty":"beginner","aiEstimatedTime":"10 min read"}],"pagination":{"total":1,"returned":1,"offset":0}}