All frameworks and cheat sheets

Key figure

The evaluator effect: 47% of problems found by one evaluator

Stat card: 47% of unique usability problems were found by only one of four evaluators who analysed the same unmoderated study.
Change the analyst and you change the report, which is why one rater is not enough. Source: MeasuringU, evaluator effect in unmoderated usability testing
Download PNGRead the full guide

A key figure on the evaluator effect in usability testing. When four evaluators with very different levels of experience analysed the same unmoderated study, they found 38 unique problems, but 47% of them were spotted by only one evaluator and 18% by all four. Use it to make the case for independent ratings from at least three people. Koji supports this by clustering the same problem across many sessions so frequency is counted, not guessed.

Where this appears

What's in this key figure

47% of unique usability problems were found by only one of four evaluators watching the same studyFour evaluators, one unmoderated study, 38 unique problems. Only 18% were found by all four, and pairwise agreement averaged 41%.

Use this image

Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:

html
<a href="https://www.koji.so/visuals/evaluator-effect-problems-found-by-one-evaluator"><img src="https://www.koji.so/visuals/evaluator-effect-problems-found-by-one-evaluator/evaluator-effect-problems-found-by-one-evaluator.png" alt="Stat card: 47% of unique usability problems were found by only one of four evaluators who analysed the same unmoderated study." width="1200" height="669" /></a>
<p>Source: <a href="https://www.koji.so/visuals/evaluator-effect-problems-found-by-one-evaluator">The evaluator effect: 47% of problems found by one evaluator</a>, Koji</p>

Related

Process showing six steps to rate usability problem severity: list findings, rate alone, avoid group rating, report mean and spread, discuss splits, justify 4s.

A 45-minute usability severity rating protocol

Table of the Nielsen 0 to 4 usability severity scale, from 0 not a problem and 1 cosmetic to 4 usability catastrophe, with the action for each rating.

Nielsen's usability severity rating scale (0 to 4)

Stat card: 8 of 359 randomised drug trials in five leading medical journals reported the number needed to treat, and 18 reported absolute risk reduction.

Only 8 of 359 trials reported number needed to treat