Checklist
AI moderator pilot scorecard
Five criteria to write down before fieldwork starts
A five-criterion scorecard for a side-by-side pilot of an AI interviewer and human moderators. It is meant for UX research leads who need a result their team will accept. Score the transcripts blind, with the arm labels removed, and set the pass marks before any interview is run. Koji compiled the pass marks as rules of thumb, so adjust them to your team.
Where this appears
What's in this checklist
- Theme coverage: AI arm finds at least 80% of the human arm's themes
- Probe depth: follow-up rate within 20 points of the human arm
- Surprise test: did either arm raise something you did not expect?
- Traceability: 10 of 10 sampled findings trace to quotes
- Participant experience: compare completion and ask how it felt
Use this image
Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:
html
<a href="https://www.koji.so/visuals/ai-moderator-pilot-scorecard-checklist"><img src="https://www.koji.so/visuals/ai-moderator-pilot-scorecard-checklist/ai-moderator-pilot-scorecard-checklist.png" alt="Checklist of five criteria for scoring an AI moderator pilot: theme coverage, probe depth, surprise test, traceability, participant experience." width="1200" height="630" /></a>
<p>Source: <a href="https://www.koji.so/visuals/ai-moderator-pilot-scorecard-checklist">AI moderator pilot scorecard</a>, Koji</p>
