All frameworks and cheat sheets

Process

What to do instead of a spot-check

Process with six steps to replace research spot-checks, from working out your break-even defect rate to stating the OC curve.
Fix the stage that makes defects, rather than checking harder at the end.

Six steps for research teams that keep letting bad data through. Work out the break-even defect rate from the cost of a check and the cost of an escape, measure the real defect rate once by inspecting a whole batch, then fix or automate instead of sampling. Prevent defects at capture, keep review for learning and say a sampling plan's false-accept rate out loud. Koji scores every interview, so the check is never a sample.

Where this appears

What's in this process

  1. Compute your break-even: Estimate the cost of one check (k1) and of one escape (k2). The ratio is your threshold defect rate.
  2. Measure the defect rate once, properly: Take one batch and inspect all of it, so you measure the process instead of guessing.
  3. Above the threshold, fix or automate: Don't sample. Cut defects at source, or make each check cheap enough to inspect everything.
  4. Mistake-proof at capture: A screener a fraudulent respondent can't pass beats any amount of transcript reading later.
  5. Use review for learning: Peer-review the study design before defects are created, not the output after.
  6. Say the OC curve out loud: "We will check ten" has a known false-accept rate. Write it in the doc.

Use this image

Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:

html
<a href="https://www.koji.so/visuals/alternatives-to-research-spot-checks"><img src="https://www.koji.so/visuals/alternatives-to-research-spot-checks/alternatives-to-research-spot-checks.png" alt="Process with six steps to replace research spot-checks, from working out your break-even defect rate to stating the OC curve." width="1080" height="1080" /></a>
<p>Source: <a href="https://www.koji.so/visuals/alternatives-to-research-spot-checks">What to do instead of a spot-check</a>, Koji</p>