Who Gets to Watch a Study While It Runs: An Interim Access Policy for Real-Time Research

Real-time research created a governance problem that research operations has not caught up with. When interviews land continuously and themes appear as they arrive, the people who commissioned the study can watch it happen. That is a genuine advantage, and it is also the single fastest way to contaminate a study, because a stakeholder who has seen the first five interviews will spend the remaining twenty looking for confirmation. The answer is not to hide the data. It is to decide, in writing and before fieldwork, who sees interim results, what they are allowed to do with them, and who is kept away from them on purpose.

The short answer

Write an interim access policy into every study that runs longer than a day. It names four things: who may view results before the study closes, what decisions those people may take on the basis of an interim view, who is deliberately excluded, and what triggers an early stop. Ten minutes of work at brief stage, and it prevents the most expensive failure mode in continuous research.

This is a governance question, not a statistical one. If your concern is false positives from repeatedly testing a running dataset, that is a different problem with a different solution, covered in interim analysis and sequential testing and interim analysis and stopping rules. This article is about who is allowed in the room.

Clinical trials solved this in 1967

The concept comes from the Greenberg Report, a 1967 report to the US National Advisory Heart Council that established what became the data monitoring committee. Its founding logic, as the FDA's guidance on establishing and operating clinical trial data monitoring committees summarises it, was that interim monitoring of accumulating data was essential to protect participants, but that individuals closely involved in the design and conduct of a trial may not be able to be fully objective in reviewing that interim data. The remedy was to route interim data to expert advisors who were external to the trial organisers, sponsors and investigators.

Sit with that for a moment. The people who designed the study are considered the least suitable people to look at its early results. Not because they are dishonest, but because they cannot un-know what they have seen.

The FDA guidance is more specific about the mechanism, and each of its concerns has an exact product analogue.

On who should see interim data. Knowledge of unblinded interim comparisons is generally not necessary for those conducting or sponsoring the trial, and such knowledge can bias the outcome by inappropriately influencing the trial's continuing conduct or the plan of analyses. Interim data should generally not be accessible to anyone other than committee members and the statisticians preparing the analyses for them.

On changing the study midway. Recommendations to change inclusion criteria, endpoints or study size are best made by those without knowledge of the accumulating data. When trial organisers are the ones reviewing interim data, the guidance is blunt that their awareness of interim results cannot help but affect their determination of whether such changes should be made, and that changes made in that setting would inevitably impair the credibility of the results.

On the people recruiting participants. Investigators enrolling subjects have a distinct conflict: knowledge of interim results could influence their conduct. An investigator aware of early trends might change their recruitment pattern or how they monitor participants. The guidance therefore recommends that committee members not include investigators in the trial.

On writing it down. Committees operate under a written charter with defined standard operating procedures, which matters for the same reason a protocol matters: it documents that the procedures were pre-specified, reducing the concern that operations were inappropriately influenced by the interim data. Among the topics the charter must address is the specification of who will have access to interim data and who may attend all or part of the committee's meetings.

The translation

The recruitment row deserves emphasis because it is the analogue almost nobody applies. If you can see that the first eight participants are lukewarm about the concept, and you still have twelve slots to fill, your screener will drift. You will approve the enthusiastic-sounding applicant you would otherwise have skipped. Nobody decides to do this. It happens through a hundred small judgement calls, which is exactly why the clinical answer is structural separation rather than a reminder to be careful.

The three harms, in the order they occur

Premature closure. The stakeholder reads the first five sessions, forms a conclusion, and communicates it. Everything after that is received as either confirmation or noise. By the time the real finding arrives at n=25, the organisation has already moved, and reversing a stated position costs political capital that nobody wants to spend. This is confirmation bias with a live feed attached.

Recruitment and instrument drift. Seeing early answers changes what you ask next and who you ask. Some drift is legitimate and valuable in discovery work, where following the interesting thread is the method. It is illegitimate when the study exists to test a claim, because the sample and the instrument were supposed to be fixed for that purpose.

Narrative lock-in. Early sessions are disproportionately memorable and disproportionately unrepresentative, because the first people to accept an invitation differ systematically from the last. The team's mental model forms on that sample and the final report has to fight it.

Writing the policy

Keep it inside the brief. It should fit in a short block.

Interim access policy

Study closes at:      n=25 completed, or 14 March, whichever first
Full interim access:  Research lead (running the study)
Monitor:              Dana K, Design Research (not on the roadmap decision)
                      Reviews at n=10. May recommend stop or continue.
No interim access:    Product lead, engineering, the recruiting owner
Early-stop triggers:  (1) Any safeguarding or distress signal
                      (2) Instrument failure: >3 participants misread Q4
                      (3) Two consecutive quality scores below 2
Interim views may:    Fix a broken instrument, halt the study
Interim views may not: Change the screener, change the sample size,
                       be shared outside the named roles

Four points on getting it right.

Name a monitor, not a committee. One colleague with no stake in the decision is enough at product scale. Their remit is narrow: is the study working, and is anyone being harmed. They are not there to form a view of the answer.

Separate operational from substantive triggers. Halting because a question is broken is an operational judgement that anyone can make without contaminating anything. Halting because the answer looks clear is a substantive judgement, and it is the one that requires the pre-specified rule; see futility analysis.

Write the exclusion list explicitly. "Not everyone" is not a policy. Naming the excluded roles is what makes the policy enforceable, and it is what protects the researcher when a senior person asks for a look. "The brief you approved says no interim access for your role" is a much easier sentence than an improvised refusal.

Set the close condition in advance. A study with no stated end runs until someone likes the answer.

Doing this in Koji

Koji's real-time reporting is a real advantage, and the discipline here is about using it deliberately rather than by default. Themes, quotes and quality scores appear as interviews complete, which is what makes a two-day study possible at all; see real-time research insights for what the live view contains.

Three mechanics make the policy enforceable rather than aspirational:

Workspace roles. Teams have three roles: owner, admin and member. Decide before launch which roles carry study access, and keep the decision-maker's account out of the workspace where the live study sits if the study is genuinely contested. This is not distrust; it is the same reasoning that keeps trial sponsors away from unblinded data.

Published reports as the release mechanism. Koji reports are generated and then published deliberately. Treat publication as the moment the study opens to its audience, and let the pre-close period stay with the research lead and the monitor. One clean release beats a fortnight of ambient half-findings.

Structured questions as the drift check. The six structured question types, open_ended, scale, single_choice, multiple_choice, ranking and yes_no, are fixed for every participant. If your final results are strange, you can establish immediately whether the instrument was stable, because the structured questions were identical throughout even where the AI probed differently. That is a much stronger position than a set of human-moderated sessions where nobody can reconstruct exactly what was asked. Details in the structured questions guide.

Worth noting what AI moderation removes from this problem entirely. In a human-moderated study, the moderator accumulates interim knowledge across every session and carries it into the next one, in tone, in follow-up choice, in what they let pass. There is no way to blind them. Koji's AI interviewer asks the pre-agreed questions and probes on what the participant actually said, not on what the previous four participants said. The interviewer, in effect, is blind to the accumulating result by construction. That is a property no traditional research method has ever had.

What to do if you are already three studies into a contaminated quarter

Do not re-run everything. Take the studies where a decision is still open, check whether the finding rests on sessions collected after the interim view was shared, and re-collect only that portion with a fresh screener. Note the contamination in the write-up rather than quietly correcting it. A finding with a stated limitation is usable; a finding with a hidden one is a liability the next time someone re-examines it.

Frequently asked questions

Is this not overkill for a five-interview study?

For a study that runs in an afternoon, yes. The policy earns its keep when fieldwork spans more than a day or two, when the study exists to settle a contested question, or when someone with authority over the outcome has access to the workspace. A quick discovery round with a curious team does not need it and should not have it.

Does excluding stakeholders damage buy-in?

It does the opposite when it is framed at the start rather than imposed midway. Stakeholders understand that a study designed to withstand challenge has to be protected from the people it might contradict, and they generally prefer a result they can defend. What breaks buy-in is denying a request for access that was never discussed. Agree it in the brief and it becomes a shared standard rather than a personal refusal.

What if the interim data shows something urgent?

That is what the trigger list is for, and it is why the monitor exists. Safeguarding concerns, participant distress and clear instrument failure should escalate immediately. The distinction to hold is between acting on the study's conduct, which is always permitted, and acting on the study's answer, which is not until the pre-specified point.

How is this different from stopping rules and alpha spending?

Stopping rules govern the statistical question: how many times you may look at accumulating data before the risk of a false positive becomes unacceptable. This governs the human question: who is permitted to look at all, and what they may do with what they see. A study can have impeccable stopping rules and still be ruined because the person who owns the roadmap watched it unfold.

Should the monitor read transcripts or only summaries?

Summaries, quality scores and completion rates are usually enough for the operational judgement they are making, and reading everything pulls them toward forming a view of the answer. Give them full access if a safeguarding question arises. The narrower their view, the better they perform the role.

Does this apply to always-on research programmes?

Yes, with an adjustment. In a continuous programme there is no single close, so define the unit of analysis instead: a fixed window or a fixed n that gets analysed and released as a block. Without that, everyone watches everything continuously and every conclusion is formed on whatever arrived most recently. See continuous discovery for how to structure the cadence.

Related Resources

• Structured Questions in AI Interviews - the six question types that keep an instrument stable
• Real-Time Research Insights - what the live view shows as interviews complete
• Interim Analysis and Sequential Testing - the statistical half of the problem
• Futility Analysis - deciding a running study will never answer your question
• Confirmation Bias in User Research - what an unmanaged live feed amplifies
• How to Write a Research Brief - where the access policy belongs