All frameworks and cheat sheets

Reference table

Sequential testing corrections and their thresholds

Table comparing five ways to correct for interim looks, Haybittle-Peto, Pocock, O'Brien-Fleming, alpha spending and always-valid inference, by threshold
Pocock and O'Brien-Fleming figures assume five equal looks at a one-sided 0.025 level. Source: Haybittle 1971, Pocock 1977, O'Brien and Fleming 1979, Lan and DeMets 1983, Johari, Pekelis and Walsh

A comparison table of the standard ways to look at a running study without inflating false positives. It gives the interim and final thresholds for Haybittle-Peto, Pocock and O'Brien-Fleming boundaries, and says when Lan-DeMets alpha spending or always-valid inference fits better. Use it to pick a rule before fielding and write it into the study brief.

Where this appears

What's in this reference table

MethodInterim thresholdFinal thresholdBest for
Haybittle-Peto (1971)p at or below 0.001 at every lookAlpha minus 0.001 for each interim lookTeams with no statistician
Pocock (1977)p of 0.0079 at each of five looksThe same 0.0079, which costs powerRarely: Pocock advised against it
O'Brien-Fleming (1979)From 0.0000025 at the first of five looks0.0207 against an uncorrected 0.025Early stopping with a near-normal final test
Lan-DeMets alpha spending (1983)Worked out as you go from a spending functionSet by the spending functionChanging the number or timing of looks
Always-valid inferenceValid whenever you decide to actNo fixed final lookLive dashboards and A/B testing platforms

Use this image

Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:

html
<a href="https://www.koji.so/visuals/sequential-testing-corrections-thresholds-compared"><img src="https://www.koji.so/visuals/sequential-testing-corrections-thresholds-compared/sequential-testing-corrections-thresholds-compared.png" alt="Table comparing five ways to correct for interim looks, Haybittle-Peto, Pocock, O'Brien-Fleming, alpha spending and always-valid inference, by threshold" width="1200" height="798" /></a>
<p>Source: <a href="https://www.koji.so/visuals/sequential-testing-corrections-thresholds-compared">Sequential testing corrections and their thresholds</a>, Koji</p>