Process
What to do instead of a spot-check

Six steps for research teams that keep letting bad data through. Work out the break-even defect rate from the cost of a check and the cost of an escape, measure the real defect rate once by inspecting a whole batch, then fix or automate instead of sampling. Prevent defects at capture, keep review for learning and say a sampling plan's false-accept rate out loud. Koji scores every interview, so the check is never a sample.
Where this appears
What's in this process
- Compute your break-even: Estimate the cost of one check (k1) and of one escape (k2). The ratio is your threshold defect rate.
- Measure the defect rate once, properly: Take one batch and inspect all of it, so you measure the process instead of guessing.
- Above the threshold, fix or automate: Don't sample. Cut defects at source, or make each check cheap enough to inspect everything.
- Mistake-proof at capture: A screener a fraudulent respondent can't pass beats any amount of transcript reading later.
- Use review for learning: Peer-review the study design before defects are created, not the output after.
- Say the OC curve out loud: "We will check ten" has a known false-accept rate. Write it in the doc.
Use this image
Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:
<a href="https://www.koji.so/visuals/alternatives-to-research-spot-checks"><img src="https://www.koji.so/visuals/alternatives-to-research-spot-checks/alternatives-to-research-spot-checks.png" alt="Process with six steps to replace research spot-checks, from working out your break-even defect rate to stating the OC curve." width="1080" height="1080" /></a>
<p>Source: <a href="https://www.koji.so/visuals/alternatives-to-research-spot-checks">What to do instead of a spot-check</a>, Koji</p>Related

How often a 10-item spot-check passes a bad batch

Share of passed survey responses truly authentic, by fraud rate

Who runs which research when you're a team of one

Continuous discovery vs traditional research

Four reasons research participants don't show up

How to synthesise research across studies in 7 steps