Interim Analysis and Sequential Testing: How to Check a Running Study Without Inflating False Positives (2026)

Answer first: looking at a running study is not the problem. Looking at a running study and applying the same significance threshold you would have applied at the end is the problem. Repeated significance tests on accumulating data inflate the false-positive rate in a way that is precisely quantified and precisely fixable. With two equally spaced looks at a two-sided 0.05 threshold, the chance of a spurious significant result is 8.3%. With five looks it is 14.2%. With twenty looks it is 24.8%. Look continuously and it converges on 100% - you are guaranteed to see significance eventually, whether or not anything is there. Clinical trials solved this in the 1970s with group sequential boundaries and alpha spending functions. Online experimentation platforms solved it again in 2015 with always-valid inference. Customer research has not solved it at all, mostly because the field never named the problem.

This guide gives you the numbers, the four standard correction methods with their actual thresholds, and a monitoring plan you can apply to a survey, an experiment, or a round of AI-moderated interviews.

The inflation is bigger than almost anyone guesses

Armitage, McPherson and Rowe published the definitive numbers in Repeated Significance Tests on Accumulating Data (Journal of the Royal Statistical Society Series A, 1969, 132(2):235-244). Their result is stark: if you test at a fixed nominal level at several points while data accumulates, the probability of at least one significant result under the null hypothesis rises far above the nominal level.

Three things about this table deserve emphasis.

It applies to nearly every test you would run. The same inflation appears with t-tests, tests of proportions, log-rank tests and Cox models. It is a property of repeated testing, not of any particular statistic. If you are comparing two groups on a scale question or a yes_no question and checking weekly, you are in this table.

Uneven spacing makes it worse. The worst case column assumes the looks are nearly independent - for example, a look after 10 respondents and another after 10,000, where the overlap is negligible. Ad hoc peeking is closer to the worst case than to the equally spaced case, because nobody peeks on a schedule. They peek when the dashboard email arrives, when a stakeholder asks, and when the early numbers look interesting.

The last row is the one that matters in product work. With unlimited looks the false-positive rate goes to 1. A live dashboard is an unlimited-look device. Nothing about the platform is broken; the statistics simply were never designed for an audience that watches.

For the specific case of online experiments, the empirical numbers are just as blunt. Working with a team of Stanford statisticians, Optimizely simulated A/A tests - experiments where the two variants are identical, so every significant result is by construction false - and reported that if you looked at the results after each new visitor there is a 57% chance of declaring a winner or loser. Looking every 500 visitors gave a 26% chance of a false declaration; every 1,000 visitors, 20%. Under sequential testing, looking after each visitor, the same error number dropped to 3%, and the platform-wide false-positive rate went from above 20% to below 5%.

Why customer research is more exposed than clinical trials

A clinical trial has a protocol, a fixed primary endpoint, and a committee that controls who may see comparative results. A product research study typically has none of those. It has a link, a dashboard, and a Slack channel.

The FDA defines an interim analysis, in its 2019 guidance Adaptive Designs for Clinical Trials of Drugs and Biologics, as "any examination of data obtained from subjects in a trial while that trial is ongoing and is not restricted to cases in which there are formal between-group comparisons." Read that definition against your own practice. Opening the report on day two of fielding is an interim analysis. Reading the first eight transcripts is an interim analysis. The fact that you did not run a significance test does not exempt you, because the decision you make afterwards - to keep fielding, to stop, to add a segment, to rewrite a question - is exactly the decision an interim analysis is supposed to govern.

There is a second exposure that is less well known. The same FDA guidance notes that type I error inflation and biased estimation "can also apply to any endpoint correlated with the primary endpoint," and that most secondary endpoints are correlated with the primary, often very highly. A product dashboard showing twelve metrics that all move together is not twelve independent chances to be wrong; it is worse than one and better than twelve, and nobody has calculated which.

The four standard corrections, with their real thresholds

Every method below buys the same thing: the right to look, in exchange for a stricter threshold at the moments you look. They differ in where they charge you.

Haybittle-Peto: the one a product team can actually run

Proposed by Haybittle in 1971 and refined with the Bonferroni inequality, this is the simplest rule in the family. Use a p-value threshold of 0.001 at every interim look, and a threshold of alpha minus (k-1) times 0.001 at the final look. With three interim analyses and one final analysis at a one-sided 0.025 level, you reject at an interim if p is at or below 0.001, and at the end if p is at or below 0.022.

Its virtues are exactly the ones a non-specialist team needs. It is trivial to implement. It is valid regardless of when the looks happen, because Bonferroni does not require knowing the joint distribution of the test statistic over time - so you can peek irregularly without invalidating it. And the final threshold is close to what it would have been with no monitoring at all, so you barely pay for the looks.

Its known weakness is the reversal of fortune problem: the boundary drops sharply at the end, so you can fail to cross it at the penultimate look, see the result partially reverse, and then cross it at the final analysis. That is logically uncomfortable, but it is a far smaller problem than an uncorrected 24.8%.

Pocock: constant boundary, expensive ending

Pocock (Biometrika, 1977, 64(2):191-199) raises the z-statistic boundary by the same amount at every look. For five equally spaced looks at a one-sided 0.025 level, the boundary is z = 2.413, a nominal p-value of 0.0079 at every look including the last one. The constant threshold is intuitive, but the price at the final analysis is steep enough to cost real power and require a larger sample. Pocock himself recommends against his own procedure for that reason.

O'Brien-Fleming: cheap looks early, near-free ending

O'Brien and Fleming (Biometrics, 1979, 35(3):549-556) set the boundary proportional to 1 divided by the square root of the look number, so early looks are punishingly conservative and later ones are close to normal. For five equally spaced looks at a one-sided 0.025 level the nominal p-value thresholds are approximately:

The final threshold of 0.0207 against an uncorrected 0.025 is the entire selling point. You get four free looks and you pay about two-tenths of a percentage point for them. This is why O'Brien-Fleming became the default in confirmatory trials, and why any team that wants both the option to stop early and a normal-looking final analysis should start here.

Alpha spending: change the schedule mid-study

Lan and DeMets (Biometrika, 1983, 70(3):659-663) reframed the problem. Instead of fixing the number and timing of looks up front, define a function that says how much of your total error budget has been spent by any given point in the study. Boundaries are then derived on the fly. Spending functions exist that approximate both the Pocock and O'Brien-Fleming shapes.

The practical consequence is the one that matters for research operations: the number and the timing of interim looks can be updated while the study is in progress without breaking error control. If fielding runs slow and you want to look at 55% rather than 50%, alpha spending absorbs that. A fixed group sequential design does not.

Always-valid inference: the version built for dashboards

Johari, Pekelis and Walsh, in Always Valid Inference: Bringing Sequential Analysis to A/B Testing (arXiv:1512.04922), defined always-valid p-values and confidence intervals designed for exactly the case where "users endogenously choose sample sizes by continuously monitoring their tests." The guarantee is that the inference is valid whenever the user decides to act, not only at a pre-declared endpoint. Their methodology was implemented in a large-scale commercial A/B testing platform and used to analyze hundreds of thousands of experiments.

If your quantitative work runs through an experimentation platform, check whether it uses sequential or fixed-horizon statistics. If it uses fixed-horizon statistics and shows a live significance indicator, that indicator is inflated in exactly the way the table above describes.

What to do when the numbers do not apply: qualitative studies

You cannot compute an O'Brien-Fleming boundary for a round of 25 in-depth interviews, and you should not pretend to. But the underlying failure mode transfers completely, and so does the fix.

The failure mode in qualitative work is not a spurious p-value. It is premature conviction: after six interviews a pattern appears, the researcher starts probing for it, later participants get asked leading follow-ups, and the study confirms a hypothesis that was formed on a sample of six. The mechanism is identical to optional stopping - a decision conditioned on interim results - and it is discussed in detail in our guide to p-hacking and researcher degrees of freedom.

Four disciplines that do transfer:

1. Separate monitoring from analysis. Monitoring asks operational questions: is recruitment on pace, are respondents finishing, is a segment under-filled, is a question confusing people. Analysis asks substantive questions: what is the answer. You can monitor daily. You should analyse once, at a pre-declared point.
2. Write down the stopping criterion before fielding. For qualitative work this is usually a saturation rule rather than a boundary - see data saturation in qualitative research and how many user interviews do you need for how to set one that is defensible.
3. Do not change the probing strategy on the basis of interim findings. Changing the instrument mid-field is a protocol amendment, and it has its own rules - see changing a study while it is running.
4. Count the looks and report them. A readout that says "we reviewed results at 25%, 50% and 100% of fielding and did not alter the design" is dramatically more credible than one that is silent about it, and silence is what reviewers now assume means daily peeking.

A monitoring plan you can write in ten minutes

Put this in the study brief before fielding starts. It is the single highest-return artifact in this entire guide.

That last row is the one teams skip and the one that does the most work. The FDA guidance on adaptive designs is unusually direct about it: "it is strongly recommended that access to comparative interim results be limited to individuals with relevant expertise who are independent of the personnel involved in conducting or managing the trial and have a need to know."

How Koji changes the economics of this problem

Koji shows results in real time as AI-moderated interviews complete. That is genuinely useful and it is also, unmanaged, a peeking machine. Two features change the calculation.

Parallel fielding compresses the window in which peeking can do damage. Traditional interviewing is sequential: twenty interviews across three weeks means twenty opportunities to update your hypothesis and drift your probing. Koji runs voice and text interviews in parallel, so a 40-participant study can complete in a day or two. A study that finishes before the first interim look is a study with zero peeking exposure - the cheapest possible fix, and it is a scheduling decision rather than a statistical one.

Structured questions make a pre-declared analysis possible. Koji supports six structured question types - open_ended, scale, single_choice, multiple_choice, ranking, and yes_no - so the primary outcome can be a specific scale or single_choice item named in the brief rather than a theme someone identifies later. A monitoring plan is only enforceable if the primary outcome was defined in advance, and a free-text-only study makes that almost impossible. See the structured questions guide for how to specify them.

The AI moderator does not drift toward a favoured hypothesis. A human moderator who has seen the interim results probes harder on the promising theme. Koji applies the same probing instructions to participant 3 and participant 30, which removes the collection-side channel through which interim looks corrupt the data - even when the researcher has been watching the whole time.

Traditional survey tools such as SurveyMonkey, Typeform and Qualtrics will happily show you a live results page with a significance indicator computed as if you had looked exactly once. Platforms like Koji do not remove the statistical problem, but pairing fast parallel fielding with pre-declared structured outcomes removes most of the opportunity for it to bite.

Frequently asked questions

Is it ever safe to just look without any correction?

Yes, in two cases. First, when you are monitoring operations rather than outcomes - completion rate, recruitment pace, segment fill, whether a question is confusing people. None of that touches the primary comparison. Second, when the study is genuinely exploratory and its output will be a hypothesis rather than a decision, provided the readout says so plainly. The inflation only matters when a threshold is being used to license a decision.

How many interim looks should a typical product study have?

One, at roughly half of target sample, is enough for most studies and is what the evidence supports. Each additional look buys a smaller chance of stopping early and costs a stricter threshold. Two interim looks plus a final analysis is a sensible ceiling unless the study runs for months or carries a safety or ethical risk that justifies closer monitoring.

Does this apply to qualitative interview studies?

The arithmetic does not, because there is no repeated significance test. The failure mode does: forming a conclusion from early interviews and then probing for it turns the remaining sample into confirmation rather than evidence. Use a pre-declared saturation rule and keep the probing strategy fixed, and you have captured most of the benefit.

What is the difference between this and p-hacking?

They are two halves of the same problem with opposite prescriptions. The p-hacking literature treats looking at interim data as a researcher degree of freedom to be eliminated - decide n in advance and do not check. Sequential methods treat it as a design feature to be priced - look as often as your plan allows, at thresholds that keep the overall error rate correct. If you need the option to stop early, sequential methods are the honest route; if you do not, fixed-n and no peeking is simpler and cheaper.

Which correction should a team with no statistician use?

Haybittle-Peto. Reject at an interim only if p is at or below 0.001, and drop the final threshold by 0.001 for each interim look you took. It needs no software, no assumption about when the looks occur, and it leaves the final analysis almost unchanged. Every other method in this guide is better in some specific way and harder in every practical way.

Does sequential testing let me stop as soon as I see a winner?

It lets you stop when the result crosses a boundary that was set in advance, which is a much higher bar early in the study than at the end. Under O'Brien-Fleming with five looks, the first-look threshold is a nominal p of about 0.0000025. That is deliberate: results that stop a study at 20% of sample are the ones most likely to be overstated. See futility analysis for what the evidence says about trials that stop early for benefit.

Related Resources

• P-Hacking and Researcher Degrees of Freedom - the fixed-n counterpart to this guide, and where optional stopping is treated as a discipline problem
• Futility Analysis: How to Decide a Running Study Will Never Answer Your Question - the other reason to stop early, and who is allowed to call it
• Intention to Treat vs Per Protocol - deciding which responses count, before you know who dropped out
• Changing a Study While It Is Running - pre-planned adaptations versus reactive protocol amendments
• Statistical Significance in Survey Research - what a p-value does and does not tell you
• Statistical Power and Minimum Detectable Effect - sizing a study so the answer is reachable at all
• Structured Questions Guide - defining a primary outcome you can pre-declare