All frameworks and cheat sheets

Flowchart

When to stop piloting an interview guide

Flowchart for deciding when to stop piloting an interview guide: run and fix rounds, check the growth rate, then compare defect cost with round cost.
Keep piloting while the next round is likely to find defects worth more than the round costs. Source: Koji synthesis, after J. T. Duane, reliability growth postulate

A decision flow for the economic stopping rule for pilot testing. Run pilot rounds and fix the guide between them, compute your growth rate after three rounds, check it sits in the normal band, then keep going while the expected cost of the defects the next round would find is higher than the cost of running it. For researchers and product teams who pilot interview guides before a study. Koji lowers the cost of a pilot round, which moves the stopping point later.

Where this appears

What's in this flowchart

Start: Your interview guide is in pilot

  1. Run a pilot round, then fix. Analyse every defect and rewrite the questions that broke before the next round.
  2. Compute your own growth rate. After three rounds, plot sessions per defect against sessions on log-log axes. The slope is the rate.
  3. Is the growth rate between 0.24 and 0.65?
    • Yes: go to 4.
    • No: Check that your fixes stick. Too low: fixes are not sticking. Too high: the easy defects are gone. Then go to 4.
  4. Project new defects for the next round. Weight each defect by the usable sessions it would cost if it escaped.
  5. Is expected defect cost above the cost of a round?
    • No: go to 6.
    • Yes: go back to 1.
  6. Finish: Stop piloting and launch. There is no natural end. Stop when looking harder costs more than it finds.

Use this image

Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:

html
<a href="https://www.koji.so/visuals/when-to-stop-piloting-interview-guide-flowchart"><img src="https://www.koji.so/visuals/when-to-stop-piloting-interview-guide-flowchart/when-to-stop-piloting-interview-guide-flowchart.png" alt="Flowchart for deciding when to stop piloting an interview guide: run and fix rounds, check the growth rate, then compare defect cost with round cost." width="1200" height="1452" /></a>
<p>Source: <a href="https://www.koji.so/visuals/when-to-stop-piloting-interview-guide-flowchart">When to stop piloting an interview guide</a>, Koji</p>