All frameworks and cheat sheets

Column chart

False-positive rate by number of interim looks

Column chart of the false-positive rate as interim looks increase at a nominal 5%: 8.3% with two looks, 24.8% with twenty and 100% with unlimited looks
Twenty looks at a 5% threshold give you about a one in four chance of a false result. Source: Armitage, McPherson and Rowe, JRSS Series A, 1969

Column chart of how often repeated significance tests on accumulating data find an effect that is not there, from Armitage, McPherson and Rowe's 1969 paper. One look keeps the 5% rate. Two equally spaced looks raise it to 8.3%, twenty to 24.8% and unlimited looks to 100%, and unevenly spaced looks are worse at every step. Use it to explain why a live results dashboard needs a monitoring plan.

Where this appears

What's in this column chart

Equally spaced looksWorst case, uneven looks
15%5%
28.3%9.8%
310.7%14.3%
412.6%18.5%
514.2%22.6%
1019.3%40.1%
2024.8%64.2%
Unlimited100%100%

Reference line: Nominal level, 5%.

Each look tests at a nominal 0.05 level on data with no true effect.

Use this image

Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:

html
<a href="https://www.koji.so/visuals/false-positive-rate-by-number-of-interim-looks"><img src="https://www.koji.so/visuals/false-positive-rate-by-number-of-interim-looks/false-positive-rate-by-number-of-interim-looks.png" alt="Column chart of the false-positive rate as interim looks increase at a nominal 5%: 8.3% with two looks, 24.8% with twenty and 100% with unlimited looks" width="1200" height="793" /></a>
<p>Source: <a href="https://www.koji.so/visuals/false-positive-rate-by-number-of-interim-looks">False-positive rate by number of interim looks</a>, Koji</p>