Usability Testing Questions: What to Ask Before, During, and After a Test

Usability testing questions fall into three phases: pre-test questions that screen and warm up the participant, in-task questions and probes that capture thinking while they use the product, and post-test questions that measure satisfaction and gather reflections. The golden rule is to ask open, non-leading questions and to let users struggle - your job is to observe behavior, not to coach. This guide gives you ready-to-use questions for every phase, the think-aloud probes that get users talking, the standardized post-test scales (SUS, SEQ, NPS), and how AI-moderated testing lets you run the same script with dozens of users at once.

Why the Questions Matter More Than You Think

In usability testing you are watching what people do, but the questions you ask shape what you learn. Ask a leading question and you contaminate the result. As usability expert Steve Krug writes in Rocket Surgery Made Easy, "Testing one user is 100 percent better than testing none" - the method is forgiving, but only if you stay neutral and let the user reveal problems on their own.

The payoff is large. Jakob Nielsen, co-founder of the Nielsen Norman Group, famously found that testing with just five users uncovers roughly 85% of the usability problems in an interface, because the same core issues recur quickly. The questions below are designed to surface those issues fast.

Pre-Test Questions (Screening and Warm-Up)

Before the test begins, confirm the participant fits your target user and set them at ease. Make it clear you are testing the product, not them.

Screening / background:

• How often do you use [product category]? (daily, weekly, monthly, rarely)
• Walk me through the last time you tried to [accomplish the core task].
• What tools or apps do you currently use for this?
• What is your role, and how does [task] fit into your day?
• Have you used [our product] before? If so, how often?

Warm-up and framing:

• There are no right or wrong answers - we are testing the design, not you. Does that make sense?
• As you go, please think out loud and tell me what you are looking at and expecting.
• Before we start, what do you expect a product like this to help you do?
• If you get stuck, that is exactly the kind of thing we want to see - please do not worry about it.

In-Task Questions and Think-Aloud Probes

During tasks, talk as little as possible. When you do speak, use neutral probes that keep the user narrating without steering them. Never answer "Where would you click?" - turn it back to them.

Setting up a task (use scenarios, not instructions):

• Say you wanted to [goal] - show me how you would do that.
• You just received [situation]. What would you do next?
• Imagine it is your first day using this. Where would you begin?

Neutral in-task probes:

• What are you thinking right now?
• What did you expect to happen when you clicked that?
• Tell me what you are looking for.
• How does that compare to what you expected?
• What would you do next?
• You went quiet - what is going through your mind?
• Is this what you thought it would do?

When a user gets stuck (do not rescue too early):

• What are you trying to do here?
• What would you expect to see at this point?
• If I were not here, what would you do next?
• On a scale of 1-7, how difficult was that task? (the Single Ease Question)

Echo and silence techniques:

• Repeat their last few words as a question ("It is confusing?") to prompt elaboration.
• Stay silent for a few seconds - most people fill the gap with exactly the insight you need.

Post-Task and Post-Test Questions

After each task, capture the immediate reaction. After the full session, gather overall impressions and standardized metrics.

Post-task (ask right after each task):

• How easy or difficult was that to complete? Why?
• Was anything surprising or confusing?
• Did the product behave the way you expected?
• What, if anything, would have made that easier?

Post-test reflection:

• What was your overall impression?
• What was the most frustrating part of the experience?
• What did you like most?
• If you had a magic wand, what one thing would you change?
• Would you use this in your real work or life? Why or why not?
• How would you describe this product to a friend?
• Is there anything you expected to find that was missing?

Standardized post-test metrics (so you can benchmark over time):

• System Usability Scale (SUS): 10 standardized agree/disagree statements producing a 0-100 score - the industry standard for comparing usability across versions.
• Single Ease Question (SEQ): a single 1-7 rating of task difficulty, asked immediately after each task.
• Net Promoter Score (NPS): "How likely are you to recommend this to a friend or colleague?" on a 0-10 scale.

The Rules for Asking Usability Questions

• Never lead. "Was that easy?" presumes the answer. Ask "How was that?"
• Do not ask users to design. "What color should this button be?" yields opinions, not usability data. Watch what trips them up instead.
• Separate behavior from preference. What people do is more reliable than what they say they prefer.
• Ask "why," not "do you like." Likeability is noisy; the reasoning behind a struggle is gold.
• Stay quiet during tasks. Every word you say is a potential bias. Let silence do the work.
• Distinguish observed problems from reported ones. A user saying "this is fine" while clicking the wrong thing three times is a usability problem, regardless of what they report.

Running Usability Questions at Scale With AI

Traditional moderated usability testing is powerful but slow: you schedule one session at a time, take notes, and manually synthesize. Nielsen recommends iterative rounds of five users, which is sound - but each round still consumes hours of a researcher's week.

Koji, an AI-native research platform, lets you script these exact questions once and have an AI moderator run them with many participants in parallel. The AI asks your pre-test, in-task, and post-test questions, listens to the responses, and - critically - probes intelligently: when a participant says "this was confusing," Koji follows up with "What specifically confused you?" the way a trained moderator would, on every session.

• Consistent moderation, zero moderator bias. Every participant gets the same neutral script, so you never accidentally lead one user and not another.
• Voice or text think-aloud. Voice interviews capture tone and hesitation; text interviews scale to large, distributed audiences.
• Built-in metrics with structured questions. Embed your SEQ, SUS items, and NPS directly using Koji's six structured question types - open_ended, scale, single_choice, multiple_choice, ranking, and yes_no - so qualitative reactions and quantitative scores live in one study and report. See the structured questions guide.
• Automatic synthesis. Themes, friction points, and representative quotes are aggregated for you, turning a week of note-taking into a same-day report.

You still get the depth of moderated testing - the open-ended "why" behind every struggle - without being the bottleneck. As the research democratizes, you no longer need a dedicated lab or a research team to ask great usability questions of a meaningful sample.

Putting It Together: A Sample Usability Test Script

1. Intro (3 min): "We are testing the design, not you. Please think out loud."
2. Pre-test (3 min): 2-3 screening/background questions.
3. Tasks (20-30 min): 3-5 scenario-based tasks, each followed by the SEQ and a "how was that?" probe.
4. Post-test (10 min): reflection questions + SUS + NPS.
5. Wrap (2 min): "Anything we did not ask that you want to share?"

Run it with five users per round if you moderate manually, or with 20-50 in parallel if you let Koji moderate - either way, this script will surface the issues that matter.

Tailoring Questions to the Type of Test

Not every usability test uses the same questions. Adjust the set to the format:

• Moderated vs unmoderated: In a moderated session you can ask live probes ("What are you thinking?"). In an unmoderated test, bake those probes into the task instructions and add a post-task open-ended box, since you cannot react in real time.
• Prototype vs live product: With an early prototype, frame questions around expectation ("What would you expect this button to do?") rather than outcome, because flows may be incomplete. With a live product, focus on actual task completion and friction.
• First-time vs returning users: Ask new users about discoverability and first impressions; ask returning users about efficiency, shortcuts, and what slows them down.
• Desktop vs mobile: On mobile, add questions about reachability, typing effort, and interruptions ("Would you finish this on your phone, or switch to a computer?").

Matching the question set to the test type keeps every prompt relevant and stops you from asking about flows the user could not realistically complete.

Related Resources

• Structured Questions Guide - embed SEQ, SUS, and NPS with 6 question types
• Usability Testing Guide - the full methodology end to end
• Usability Testing Script Template - a ready-to-edit moderator script
• Think-Aloud Protocol - the technique behind in-task narration
• System Usability Scale Guide - scoring and benchmarking SUS
• Single Ease Question (SEQ) Guide - the fastest per-task usability metric