All frameworks and cheat sheets

Guide

Five pilot testing mistakes with interview guides

Listicle of five pilot testing mistakes: trusting one quiet round, not fixing between rounds, ignoring severity, friendly participants, borrowed growth rates.
Check your pilot plan against these five before you call a guide finished.

Five common mistakes when pilot testing an interview guide, and what each one does to your read of how ready the guide is. They include treating one quiet round as proof, fitting a curve without fixing anything, ignoring severity, piloting with friendly participants and borrowing someone else's growth rate. Use it as a review list before you call a guide done. Koji quality scores per question help show where a guide is still breaking.

Where this appears

What's in this guide

  1. Treating one quiet round as proof: At two defects per round, a round with none is well within noise.
  2. Fitting the curve without fixing: If you do not repair defects between rounds, you are re-measuring the same broken guide.
  3. Ignoring severity: Weight defects by what they cost downstream, or a typo counts as much as a question nobody can answer.
  4. Piloting with friendly participants: It holds the defect rate down and makes the guide look better than it is.
  5. Borrowing another study's growth rate: Compute your own from your first three rounds. It takes two minutes.

Use this image

Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:

html
<a href="https://www.koji.so/visuals/pilot-testing-mistakes-interview-guide"><img src="https://www.koji.so/visuals/pilot-testing-mistakes-interview-guide/pilot-testing-mistakes-interview-guide.png" alt="Listicle of five pilot testing mistakes: trusting one quiet round, not fixing between rounds, ignoring severity, friendly participants, borrowed growth rates." width="1200" height="790" /></a>
<p>Source: <a href="https://www.koji.so/visuals/pilot-testing-mistakes-interview-guide">Five pilot testing mistakes with interview guides</a>, Koji</p>