Reference table
Sequential testing corrections and their thresholds

A comparison table of the standard ways to look at a running study without inflating false positives. It gives the interim and final thresholds for Haybittle-Peto, Pocock and O'Brien-Fleming boundaries, and says when Lan-DeMets alpha spending or always-valid inference fits better. Use it to pick a rule before fielding and write it into the study brief.
Where this appears
What's in this reference table
| Method | Interim threshold | Final threshold | Best for |
|---|---|---|---|
| Haybittle-Peto (1971) | p at or below 0.001 at every look | Alpha minus 0.001 for each interim look | Teams with no statistician |
| Pocock (1977) | p of 0.0079 at each of five looks | The same 0.0079, which costs power | Rarely: Pocock advised against it |
| O'Brien-Fleming (1979) | From 0.0000025 at the first of five looks | 0.0207 against an uncorrected 0.025 | Early stopping with a near-normal final test |
| Lan-DeMets alpha spending (1983) | Worked out as you go from a spending function | Set by the spending function | Changing the number or timing of looks |
| Always-valid inference | Valid whenever you decide to act | No fixed final look | Live dashboards and A/B testing platforms |
Use this image
Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:
html
<a href="https://www.koji.so/visuals/sequential-testing-corrections-thresholds-compared"><img src="https://www.koji.so/visuals/sequential-testing-corrections-thresholds-compared/sequential-testing-corrections-thresholds-compared.png" alt="Table comparing five ways to correct for interim looks, Haybittle-Peto, Pocock, O'Brien-Fleming, alpha spending and always-valid inference, by threshold" width="1200" height="798" /></a>
<p>Source: <a href="https://www.koji.so/visuals/sequential-testing-corrections-thresholds-compared">Sequential testing corrections and their thresholds</a>, Koji</p>




