{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-11-14T11:30:30.633Z"},"content":[{"type":"blog","id":"8416df62-8c6c-43cd-9758-a7fa93e744c0","slug":"usability-test-debrief-ux-researcher-playbook","title":"Usability Test Debrief: A UX Researcher Playbook for Finding Out Why Users Failed the Task","url":"https://www.koji.so/blog/usability-test-debrief-ux-researcher-playbook","summary":"A playbook for UX researchers on running a usability test debrief after every task. Self-reported completion is unreliable (MeasuringU: 93% self-reported vs 33% verified in four unmoderated studies; about 13% of task experiences were failures with high confidence), so the method compares what the participant pressed, what the recording shows and what they said. It covers sample sizes, an ease and confidence question pair, a seven-question debrief guide, and how to run the task and debrief in one Koji study with a Task question, which reports finished and stuck counts, median time on task, recorded moments and a verdict from the recording.","content":"A usability test debrief is the short interview straight after each task, where you find out why someone failed, hesitated or took the long way round. Run with an AI moderator that hands over the task, stays quiet while people work and asks what happened the moment they finish, the debrief stops being the part you rush at the end of a session and becomes part of every task, for every participant. By the end of this playbook you'll have a debrief method you can run this week, a rule for when a \"success\" isn't one, a copy-ready guide and the exact setup in Koji.\n\nSay you're the only researcher on a product team. Design wants results on the new invite flow by Friday's review. Last month's unmoderated test gave you a 60% completion rate and a folder of recordings nobody had time to watch, so the team argued about the cause and shipped a guess. This time you want the number and the reason, from the same people, in the same week.\n\n## Why the usability test debrief matters now\n\nThe first reason is that participants are poor judges of their own success. MeasuringU looked at four unmoderated competitive benchmark studies and found self-reported task completion averaged 93%, while verified completion averaged 33% ([MeasuringU](https://measuringu.com/?p=291)). Self-reported rates ranged from 63% to 100%, verified rates from 4% to 88%. People press \"done\" when they think they've finished, and in an unmoderated test with no follow-up, that button is all you have.\n\n![Column chart of self-reported vs verified task completion in four unmoderated studies: average 93% said, 33% verified, lowest 63% vs 4%.](/visuals/self-reported-vs-verified-task-completion/self-reported-vs-verified-task-completion.png \"People say they finished far more often than they did · Source: MeasuringU, self-reported vs verified completion analysis\")\n\nIt gets worse when people feel sure. In a MeasuringU analysis of 174 tasks, 1,200 users and 14 websites and products, about 13% of task experiences were what Jeff Sauro calls disasters: the participant failed and said they were extremely confident they had succeeded ([MeasuringU, Measuring User Interface Disasters](https://measuringu.com/ui-disasters/)). Across 19,576 task observations in another analysis, 14% of people who failed a task still gave it the top satisfaction score ([MeasuringU](https://measuringu.com/failed-sat/)). A post-task rating on its own will not catch them.\n\nThe second reason is load. In Maze's Future of User Research Report 2026, a survey of nearly 500 researchers, designers and product professionals, 66% said demand for research had grown in the past 12 months, up from about 55% a year earlier, and 69% now use AI in at least some studies ([Maze, 2026](https://maze.co/blog/future-user-research-2026/)). Evaluative work is where that demand piles up, because every sprint ships something testable. When you're stretched, the debrief is what gets cut.\n\nAnd there's now real evidence on AI moderation of usability tests. At CHI 2026, Zhu and colleagues compared an AI audio moderator with a human moderator in think-aloud usability testing in a randomized controlled trial with 60 participants ([PolyU research record](https://research.polyu.edu.hk/en/publications/agentic-audio-moderator-vs-human-moderator-in-think-aloud-usabili/)). The question has moved from \"can it be done\" to \"how do you run it well\".\n\nFor context on what a normal result looks like: across 1,189 tasks from 115 usability tests, MeasuringU found an average completion rate of 78% ([MeasuringU](https://measuringu.com/?p=138)), and the average Single Ease Question score sits around 5.5 out of 7 ([MeasuringU, Evolution of the SEQ](https://measuringu.com/evolution-of-seq)). If your flow lands well below either, you have a problem. The debrief tells you which one.\n\nSkip it, and you report a completion rate people will argue about, fix the symptom someone guessed at, and run the same test again next quarter.\n\n## What good looks like\n\nHere is how researchers who do this well run it. We call it the **Pressed, Saw, Said check**: for every task, compare what the participant pressed, what the recording shows and what they told you.\n\n![A UX researcher and a designer at a desk, watching a recorded usability session on a laptop and taking notes on what went wrong](https://images.unsplash.com/photo-1758598305340-95c68b43ce49?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3w5NDUzMDN8MHwxfHNlYXJjaHwzfHxkZXNpZ25lciUyMHdhdGNoaW5nJTIwbGFwdG9wJTIwc2NyZWVuJTIwbm90ZXN8ZW58MXwwfHx8MTc5NDY0MjYxMHww&ixlib=rb-4.1.0&q=80&w=1080)\n\n_Photo by [Vitaly Gariev](https://unsplash.com/@silverkblack?utm_source=koji&utm_medium=referral) on [Unsplash](https://unsplash.com/?utm_source=koji&utm_medium=referral)_\n\n**Debrief every task, not the session.** A debrief at the end of a 40-minute session gets a blur. Ask about each task straight after it, while they still remember where they looked first.\n\n**Ask about expectations before reactions.** \"What did you expect to happen when you clicked Share?\" reveals the mental model. \"How did that feel?\" mostly gets politeness.\n\n**Never take \"done\" at face value.** The recording decides the success rate. The interview explains the gap. If someone pressed done but never reached the goal, count it as a failure and ask what they thought \"finished\" meant. That answer is often your best finding.\n\n**Separate direct from indirect success.** Someone who got there after three wrong turns didn't have the same experience as someone who went straight there. Report both, because the wrong turns are your fix list.\n\n**Pair one ease rating with one confidence question.** Ease (the SEQ) tells you how hard it felt. Confidence tells you whether they'd trust the result. A failed task with high confidence is the one you would never hear about otherwise.\n\n**Size by what you'll claim.** Five people per segment are enough to find the big problems, in Jakob Nielsen's long-standing guidance ([NN/g](https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/)). If you'll put a completion rate in front of leadership as a number, plan on 20 or more and give a confidence interval with it.\n\n**Common mistakes:**\n\n- Reporting the button presses as the completion rate\n- Writing tasks that name the button (\"Use Share to invite...\")\n- Asking \"Was that easy?\" instead of asking what they expected\n- Debriefing once, at the very end\n- Watching only the failures, so you never learn what the fast people did differently\n- Changing the task wording halfway through a round\n\n## When a success isn't one\n\nBefore you report a completion rate, run every participant through the same three checks. If they pressed I'm stuck, it's a failure, and the debrief tells you where they gave up. If they pressed done but the recording shows they never reached the goal, it's a failure too, and the most useful one to read. If they got there after wrong turns, it's an indirect success: count it as a success, and put the wrong turns on the fix list. If someone didn't share their screen there's no recording to check, so lean on the debrief and their confidence answer.\n\n![Flowchart for counting usability task success: did they press done, does the recording show the goal reached, and were there wrong turns.](/visuals/did-they-really-finish-the-task-flowchart/did-they-really-finish-the-task-flowchart.png \"Run this for every participant before you report a completion rate\")\n\n## How to set up the test and debrief in Koji, step by step\n\nIn Koji the task and the debrief are one study: a **Task** question hands the participant a real web page, and Koji asks what happened as soon as they press I'm done or I'm stuck.\n\n1. **Start the study.** From the dashboard's template gallery, filter to UX Researchers and pick **Evaluative Post-Test Interviews**. Or paste this brief into the \"What do you want to research?\" box: _\"Find out whether people can [do the task] on [page] and why those who fail or hesitate do, so we can fix the root cause before [release].\"_\n2. **Shape the brief.** Set the decision to inform (\"Which issues in the [flow] are critical, and what causes each\"), the target participant and a guardrail: the interviewer must never hint at the right path.\n3. **Add a Task question.** Paste the page address into **Task page**: a live site, a staging build, a competitor's site or a prototype share link. Write the task as a goal in the user's words. If the page needs a login, switch on the option to always open it in a new tab. Task questions carry a Beta tag, and each task adds 3 credits to every completed interview.\n4. **Add the debrief questions after it.** Add the SEQ as a Scale question on the 1 to 7 range, with \"Very difficult\" and \"Very easy\" as end labels, and a single-choice confidence question. Set the follow-up depth per question, and add probing guidance such as \"ask what they expected before asking how it felt\".\n\n![Koji question editor with open-ended, scale and single-choice questions, each with its own follow-up setting.](/email/shots/shot-question-types.jpg \"The ease and confidence questions sit right after the task, each with its own follow-up depth\")\n\n5. **Choose voice or text.** Both work. On desktop, Koji offers to record the shared tab and the participant's voice, and people can decline and still do the task. In voice, Chrome and Edge on desktop can keep Koji in a small floating window while the page is open. A text interview with one task uses 4 credits, a voice interview 6.\n6. **Invite people.** Add a screening question for the segment you need. Email your own users from Participants with reminders (Insights plan and up), give each person their own invite link from a CSV (Interviews and Enterprise), or recruit from Koji's panel on a paid plan.\n7. **Preview it yourself.** Previews are free and not counted. Run the task once as a participant to check the page opens and the card reads clearly.\n8. **Read the results.** The report's task section shows how many finished and got stuck, the median time on task, how many shared their screen, a reel of recorded moments with what was said, and a flow of the pages people visited. On the Interviews table, each person's task details show Koji's verdict from the recording (Direct success, Indirect success or Failed) next to what they pressed, with time, pages visited and wrong turns. That's your Pressed, Saw, Said check, already lined up.\n\nWhich plan? Free gives you 10 credits at signup, enough for two pilot interviews with one task in text. For a real round of 20, budget about 80 credits in text, since only interviews that score 3 or more on quality use credits or reach the report ([how quality scores work](/docs/understanding-quality-scores)). Interviews (79 credits a month, 3 seats) adds personal invite links for matching people to your CRM list.\n\n## Copy this interview guide\n\nPaste this into any tool. Swap the bracketed parts and keep the wording fixed for the whole round.\n\n**Opening:** \"Thanks for helping. I'll give you something to try on a website. There are no wrong answers. If something is confusing, that's the site's fault, not yours, and it's exactly what we need to know.\"\n\n1. _Task._ \"[Goal in their words, e.g. You want a colleague to see this project but not edit it. Set that up.]\" Follow-up: \"Tell me about a moment where you weren't sure what to do next.\"\n2. _Open._ \"Before you started, what did you expect to do first?\" Probe: \"What made you expect that?\"\n3. _Open._ \"Walk me through the moment you slowed down or changed your mind.\" Probe: \"What were you looking for right then?\"\n4. _Scale 1 to 7._ \"Overall, how difficult or easy was that task?\" Probe: \"What would have made it one point easier?\"\n5. _Single choice._ \"How sure are you that it's set up the way you wanted?\" Not sure, fairly sure, completely sure. Probe: \"What would you check to be certain?\"\n6. _Open._ \"If a colleague asked you how to do this tomorrow, what would you tell them?\" Probe: \"Where would you tell them to look?\"\n7. _Open._ \"Was there anything you expected to see that wasn't there?\" Probe: \"Where did you expect it?\"\n\n**Closing:** \"Of everything you tried today, what was the most frustrating part, and is there anything we didn't ask about?\"\n\n![Question bank for a usability test debrief in three sections: right after the task, ease and confidence, how they think it works.](/visuals/usability-test-debrief-questions/usability-test-debrief-questions.png \"Ask these straight after each task, while the moment is fresh · Source: ease item after the Single Ease Question, MeasuringU\")\n\n## What you get\n\nBy the end of the round you have, for each task:\n\n- **A completion rate you can defend:** finished and stuck counts, plus the verdict from the recordings, so \"pressed done but failed\" cases are counted as failures and nobody can wave them away.\n- **The gap list:** every person whose button and recording disagree, with their own explanation. These are the disasters MeasuringU warns about, and the first thing to fix.\n- **Median time on task,** to compare against the next round.\n- **The clips that make the case:** the reel of recorded moments, each with what the participant said, which is what convinces a design review faster than a slide of percentages.\n- **A readout:** download the report as slides or a PDF and lead with one task, its verdicts and three clips.\n\nThe designer gets the gap list the same afternoon. The PM gets one slide: can people do it, how long it takes, and the one fix that matters most.\n\n## The same playbook elsewhere\n\nThe Pressed, Saw, Said check works anywhere people report their own success. Product managers run the same task-plus-debrief study on a staging build before a release. Researchers measuring a redesign can add the task to both rounds of a [before-and-after study](/blog/redesign-before-after-study-ux-researcher-playbook). And if your team is still deciding whether to trust AI moderation at all, the [AI moderator pilot playbook](/blog/ai-moderator-pilot-ux-research-lead-playbook) shows how to test it against your own moderators first. Every researcher playbook is at [/use-cases/researchers](/use-cases/researchers).\n\n## Why Koji for usability test debriefs\n\nHere is how the realistic options compare on what a debrief needs.\n\n|                                   | Moderated sessions           | Unmoderated test with a post-task survey | Koji                                          |\n| --------------------------------- | ---------------------------- | ---------------------------------------- | --------------------------------------------- |\n| Asks why after each task          | Yes                          | Fixed questions only                     | Yes, follow-ups on what they said             |\n| Catches \"pressed done but failed\" | Yes, you watch it            | Only if someone watches every video      | Verdict from the recording next to the button |\n| Time to results                   | A week or more of scheduling | Days, plus hours of video                | Interviews run in parallel                    |\n| Participants at once              | One per moderator            | Many                                     | Many                                          |\n| Same probing for everyone         | Varies by moderator          | Same questions, no probing               | Same protocol, adaptive follow-ups            |\n| Analysis effort                   | Hours of notes               | Hours of video                           | Report with moments, times and quotes         |\n| Native mobile apps                | Yes                          | Depends on the tool                                    | No, web pages only                            |\n\n![Comparison table of three ways to run a usability test debrief, Koji highlighted, on follow-ups, catching false successes, speed and analysis effort.](/visuals/usability-debrief-method-comparison/usability-debrief-method-comparison.png \"Pick by what you can't do without: a live moderator, scale, or both\")\n\nWhy we'd run it in Koji:\n\n1. **The debrief is built into every task.** Koji asks what happened as soon as they press I'm done or I'm stuck, so no participant skips it.\n2. **False successes surface on their own.** The task details put Koji's verdict from the recording next to the button the participant pressed.\n3. **The moderator stays neutral.** Koji introduces the task, the card carries it, and while people work it never says where to click or whether they're on the right track.\n4. **Tasks and questions live in one study.** Mix tasks with open questions, ratings, choice questions and price tests, instead of stitching a test tool to a survey.\n5. **Low-effort interviews stay out.** Interviews scoring under 3 on quality don't reach the report or use credits.\n\n## Questions you might have about AI interviews\n\n**Will an AI lead people to the answer?** The interviewer is set up not to. It introduces the task in one sentence, the card holds the instructions, and while people work it stays quiet. It never explains how to do the task or hints whether they're on track. Run a preview yourself and listen for it.\n\n**Will people share their screen?** It's optional and asked in the browser's own prompt. Those who decline still do the task, and you still get the outcome, the time and the debrief, though no recorded moments or page flow for them.\n\n**Can I trust the verdict and the moments?** Outcome and time are exact: they come from the button press. The verdict, moments and page path come from watching the recording, so treat them as approximate and check the clip behind any finding before you act on it.\n\n## When Koji may not be the right fit\n\nIf you're testing a native mobile app, use a mobile testing tool: Koji's Task questions open web pages, and screen recording needs a desktop browser that can share a tab. If you need click maps, first-click tests or tree tests, use a dedicated platform, because Koji doesn't run those. And for a deeply exploratory session on a messy early prototype, where you want to change the plan mid-conversation, a live moderator is still the better choice.\n\n## Metrics to track\n\n- Verified completion rate per task, and the gap between pressed and verified\n- Share of direct versus indirect successes\n- Median time on task, round over round\n- Mean SEQ per task, against the 5.5 average\n- Share of failed tasks with \"completely sure\" confidence\n- Number of gap-list issues fixed before the next round\n\n## Frequently asked questions\n\n**What is a usability test debrief?** The short interview after a usability task, where you ask what the participant expected, where they hesitated and how sure they are of the result. It explains the behaviour the recording shows.\n\n**Should I debrief after each task or at the end?** After each task. The details of one task blur into the next, and a single end-of-session debrief blends everything together.\n\n**How many participants do I need?** Five per segment find most of the big problems. If you'll report a completion rate as a number, plan on 20 or more and show a confidence interval.\n\n**What do I do when someone says they finished but didn't?** Count it as a failure in the completion rate, then read their explanation. In MeasuringU's data, self-reported completion averaged 93% against 33% verified, so this will happen often.\n\n**Can I run a usability debrief on a Figma prototype?** Yes, if the prototype has a share link that opens in a browser. Paste the link as the task page and write the task as a goal.\n\n## Run your first debriefed round this week\n\nSign up free, open the template gallery, start from **Evaluative Post-Test Interviews** and add a Task question with your page, or paste the brief from step 1. Run two previews, then invite your first five. For more on how researchers use Koji, see [Koji for researchers](/for/researchers), and for the measurement itself, the [Single Ease Question guide](/docs/single-ease-question-seq-guide), [how to write usability tasks](/docs/usability-test-tasks-scenarios) and [usability metrics explained](/docs/usability-metrics-guide).\n","category":"Use Cases","lastModified":"2026-11-14T07:50:10.750593+00:00","metaTitle":"Usability Test Debrief: Why Users Failed the Task","metaDescription":"Run a usability test debrief after every task: the Pressed, Saw, Said check, a copy-ready question guide and the setup in Koji with Task questions.","keywords":["usability test debrief","post-task interview questions","usability testing follow-up questions","task success rate","self-reported task completion","single ease question","AI-moderated usability testing"],"aiSummary":"A playbook for UX researchers on running a usability test debrief after every task. Self-reported completion is unreliable (MeasuringU: 93% self-reported vs 33% verified in four unmoderated studies; about 13% of task experiences were failures with high confidence), so the method compares what the participant pressed, what the recording shows and what they said. It covers sample sizes, an ease and confidence question pair, a seven-question debrief guide, and how to run the task and debrief in one Koji study with a Task question, which reports finished and stuck counts, median time on task, recorded moments and a verdict from the recording.","aiKeywords":["usability test debrief","post-task questions","unmoderated usability testing","AI moderated usability test","task completion rate","Single Ease Question","UX researcher playbook","Koji Task questions"],"aiContentType":"guide","faqItems":[{"answer":"The short interview after a usability task, where you ask what the participant expected, where they hesitated and how sure they are of the result. It explains the behaviour the recording shows.","question":"What is a usability test debrief?"},{"answer":"After each task. The details of one task blur into the next, and a single end-of-session debrief blends everything together.","question":"Should I debrief after each task or at the end?"},{"answer":"Five per segment find most of the big problems, in Jakob Nielsen's guidance. If you will report a completion rate as a number, plan on 20 or more and show a confidence interval.","question":"How many participants do I need for a usability test with a debrief?"},{"answer":"Count it as a failure in the completion rate, then read their explanation. In MeasuringU's analysis of four unmoderated benchmark studies, self-reported completion averaged 93% against 33% verified.","question":"What do I do when a participant says they finished but didn't?"},{"answer":"Yes, if the prototype has a share link that opens in a browser. In Koji, paste the link as the task page and write the task as a goal in the user's words.","question":"Can I run a usability debrief on a Figma prototype?"}],"relatedTopics":["Usability testing","Task success rate","Single Ease Question","AI-moderated research","UX research"]}],"pagination":{"total":1,"returned":1,"offset":0}}