All frameworks and cheat sheets

Bar chart

False results in A/A tests, by how often you look

Bar chart of false results in A/A tests: 57% when checking after every visitor, 26% every 500, 20% every 1,000 and 3% with a sequential test
Same peeking, different statistics: a sequential test brings the false-result rate from 57% to 3%. Source: Optimizely and Stanford statisticians, A/A test simulations

Bar chart from A/A test simulations Optimizely ran with Stanford statisticians, where both variants were identical so any declared winner or loser was false. Checking after every visitor gave a 57% chance of a false result, every 500 visitors 26% and every 1,000 visitors 20%. A sequential test checked after every visitor brought it down to 3%. Worth showing anyone who watches a live experiment dashboard.

Where this appears

What's in this bar chart

False winner or loser
Look after every visitor57%
Look every 500 visitors26%
Look every 1,000 visitors20%
Sequential test, every visitor3%

A/A tests: both variants are identical, so every declared winner or loser is false.

Use this image

Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:

html
<a href="https://www.koji.so/visuals/aa-test-false-result-rate-by-peeking-frequency"><img src="https://www.koji.so/visuals/aa-test-false-result-rate-by-peeking-frequency/aa-test-false-result-rate-by-peeking-frequency.png" alt="Bar chart of false results in A/A tests: 57% when checking after every visitor, 26% every 500, 20% every 1,000 and 3% with a sequential test" width="1200" height="630" /></a>
<p>Source: <a href="https://www.koji.so/visuals/aa-test-false-result-rate-by-peeking-frequency">False results in A/A tests, by how often you look</a>, Koji</p>