Process
A 45-minute usability severity rating protocol
A six-step protocol for rating the severity of usability problems in about 45 minutes. It has at least three people rate each problem independently, report the mean and the spread, discuss only the items where ratings split by two points or more, and record the evidence for every catastrophe rating. It suits UX researchers and product teams after a usability study. Koji can supply measured frequency and participant ratings to feed into it.
Where this appears
What's in this process
- Compile the raw findings list: One observable line per problem, the participant IDs who hit it and a timestamp or clip.
- Rate independently, in silence: Three raters minimum, including at least one person who did not run the sessions.
- Never rate as a group first: The first confident voice anchors everyone and you lose the disagreement signal.
- Compute the mean and the spread: Report both. Ratings of 4, 4, 4 and 4, 3, 1 are different findings.
- Discuss only the split items: A range of two points or more gets five minutes. One rater usually knows something the others do not.
- Record the reason for every 4: Write down the frequency and impact evidence now, before someone challenges it.
Use this image
Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:
html
<a href="https://www.koji.so/visuals/usability-severity-rating-protocol-six-steps"><img src="https://www.koji.so/visuals/usability-severity-rating-protocol-six-steps/usability-severity-rating-protocol-six-steps.png" alt="Process showing six steps to rate usability problem severity: list findings, rate alone, avoid group rating, report mean and spread, discuss splits, justify 4s." width="1080" height="1080" /></a>
<p>Source: <a href="https://www.koji.so/visuals/usability-severity-rating-protocol-six-steps">A 45-minute usability severity rating protocol</a>, Koji</p>
