Interference Between Users: Why Your A/B Test Control Group Is Not a Control Group (2026)

Answer first: every A/B test assumes that what you do to a treated user has no effect on an untreated user. In marketplaces, social products, support-backed workflows, and anything sharing a ranking model, that assumption is false - and when it breaks, the bias is not small. An eBay email campaign that a naive test scored as a clear success turned out to have an effect statistically indistinguishable from zero once test-control interference was removed. An Airbnb pricing experiment lost at least 20% of its measured effect the moment the randomisation moved from individuals to clusters. Interference does not add noise you can average away with a bigger sample. It biases the point estimate, and a bigger sample makes you more confident in the wrong number.

This guide explains the assumption you are breaking, the four channels through which treatment leaks into your control group, how to tell which one you have, and the three designs that fix it. It also covers the part most engineering write-ups skip: interference is a question about mechanism, and mechanisms are not in your event log. They are in your users' heads.

The assumption has a name, and you have never checked it

Every standard A/B analysis rests on the Stable Unit Treatment Value Assumption (SUTVA). Informally: each unit's outcome depends only on its own treatment assignment, not on anyone else's. When that fails, the comparison you are making is not the comparison you think you are making.

Blake and Coey of eBay Research Labs put the problem plainly in their 2014 ACM Conference on Economics and Computation paper: "Classical statistical inference of experimental data assumes that the treatment affects the test group but not the control group. This assumption will typically be violated when experimenting in marketplaces because of general equilibrium effects: changing test demand affects the supply available to the control group."

The mechanics in their case are almost embarrassingly simple. eBay randomised users into receiving marketing emails or not. Test and control users then bid in the same auctions. If the email pushes a test user to win an auction, it necessarily pushes some control user to lose one. The treatment did not create new inventory. It moved inventory from one arm to the other.

Their finding: "Ignoring test-control interference leads to estimates of the campaign's effectiveness which are too large by a factor of around two." When they reconstructed the comparison at the auction level rather than the user level, "the effect of the campaign is not significantly different from zero, whereas a naive user comparison would suggest that the campaign was very successful."

That is the shape of the problem. Not a slightly inflated lift. A campaign that did nothing, scored as a win.

Four channels, and most teams only know about one

The marketplace case is famous, which has had the unfortunate effect of convincing non-marketplace teams that interference is somebody else's problem. It is not. Here are the four channels, in rough order of how often they go unnoticed.

The third and fourth are the ones that quietly ruin B2B and AI experiments.

Shared human capacity. You test a proactive onboarding nudge. Treated accounts book more onboarding calls. Your onboarding team has a fixed number of hours a week, so control accounts wait longer for theirs. Your treatment did not just help the treatment group; it degraded the control group. The measured gap is the real effect plus the harm you did to your own baseline. This is the same zero-sum structure as an eBay auction, and almost nobody models it, because the contested resource is a rota rather than an inventory table.

Shared model or ranking. This one is new enough that many teams have not caught up. If you are testing a change to a recommender, a search ranker, or a retrieval-augmented AI feature, and the underlying model continues learning from the pooled behaviour of both arms, then the treatment group's interactions are training the model that serves the control group. The arms converge. Your measured effect shrinks toward zero, and you conclude the feature does not work.

Note what that does to the direction of the bias. Interference is not a synonym for inflation. Blake and Coey derive this formally: the magnitude of the bias grows as supply becomes more inelastic, and the direction "will be positive when demand is elastic, and negative otherwise." A contested-resource experiment tends to overstate. A shared-model experiment tends to understate. You cannot sign the bias without knowing the mechanism, and the mechanism is not a statistical quantity - it is a fact about how your product works.

The evidence that it is worth fixing

Three numbers are worth carrying into a design review.

A factor of two. Blake and Coey's eBay email campaign, where the naive user-level estimate was roughly twice the interference-corrected one, and the corrected effect was not distinguishable from zero.

Up to 90%. In a simulation of an online market for short-term lodging, Fradkin found that user-level search experiments can overstate market-wide outcomes by 90%, a result Blake and Coey cite as motivating their own work.

At least 20%. Holtz, Lobel, Lobel, Liskovich and Aral ran what they call a meta-experiment - an experiment over experiments - on Airbnb, comparing individual-level Bernoulli randomisation against cluster randomisation for the same platform fee change. Their finding, published in Management Science: "at least 20% of the TATE estimate produced by an individual-level randomized evaluation of the platform fee increase we study is attributable to interference bias and eliminated through the use of cluster randomization." They also note, in the same paper, that prior work suggests naive designs can "lead to TATE estimates that are overstated by up to 100%."

One caution on that 20%. An earlier working-paper version of the same study reported a larger figure. The peer-reviewed number is the one to quote, and the honest reading of it is a lower bound: cluster randomisation removes the interference that crosses cluster boundaries, so whatever leaks within a cluster is still in there.

The three designs that fix it

1. Cluster randomisation: change the unit, keep the comparison

Randomise groups rather than individuals, choosing groups so that most interference happens inside a group rather than across groups. On Airbnb, geography is the natural cluster: two listings in Lisbon compete with each other far more than either competes with a listing in Osaka.

The cost is statistical power. You now have as many independent units as you have clusters, not as many as you have users, and outcomes within a cluster are correlated. A study with 200,000 users and 40 cities has, for inference purposes, something much closer to 40 units. Holtz and colleagues also found suggestive evidence that "the efficacy of cluster randomization at reducing interference bias increases with cluster quality" - a badly drawn cluster boundary buys you the power cost without the bias fix.

2. Switchback: randomise time instead of people

If the contested resource is the entire market, no partition of users is clean. The alternative is to stop splitting users at all and switch the whole system between treatment and control over successive time periods.

Bojinov, Simchi-Levi and Zhao, writing in Management Science, describe switchbacks as "among the most prevalent designs used in the technology sector, with applications ranging from ride-hailing platforms to online marketplaces." Their contribution is the optimal design, derived "under a range of different assumptions on the order of the carryover effect - the length of time a treatment persists in impacting the outcome."

Carryover is the whole game. Switch too fast and the system has not settled, so period t still carries the previous period's treatment; your control periods are contaminated by treatment. Switch too slowly and you have very few periods, which means very little power and heavy exposure to time-of-day and day-of-week confounding. The standard mitigation is a washout: discard the data from the first stretch of each period and analyse only the settled remainder. Bojinov and colleagues also supply theory for the case where you guess the carryover order wrong, which is the realistic case.

Switchback is not stepped wedge, and confusing them is a design error. The stepped wedge randomises when each unit crosses over because you cannot randomise who - everyone must eventually get the change, and no one is denied anything. Units cross once, in one direction, and never revert. A switchback randomises when because randomising who would be invalid, and the system flips back and forth many times. Use a stepped wedge when the constraint is political or operational. Use a switchback when the constraint is interference.

3. Two-sided and budget-split designs

For a genuinely two-sided marketplace, randomising one side still lets the effect propagate through the other. Two-sided designs randomise both sides and use the cells to identify how much of the effect travelled across the market. These are more complex to analyse and mostly worth it when the decision is large.

Detecting interference before you design: a four-question audit

You do not need a paper to work out whether you have this problem. Ask four questions about the change you are about to test.

1. Is there anything finite that both arms draw from? Inventory, seats, delivery slots, ad impressions, support hours, sales capacity, a rate-limited API. If yes, you have contested-resource interference, and the naive estimate is probably too high.
2. Can a treated user's behaviour reach an untreated user? Content, invitations, shared documents, leaderboards, referrals, a shared team workspace. If yes, you have network interference.
3. Does any model that serves both arms keep learning during the test? Rankers, recommenders, personalisation, an AI feature with an online feedback loop. If yes, your arms are converging, and the naive estimate is probably too low.
4. If the treatment worked perfectly for everyone, would the total pie grow, or would it just be redistributed? This is the sharpest of the four. A treatment that only redistributes a fixed quantity will always look like a success at the user level and always be worth nothing at the company level. That is exactly the eBay result.

If any answer is yes, the design decision comes before the analysis decision, and no amount of post-hoc statistics rescues a badly chosen unit of randomisation.

The part the engineering literature leaves out

Every technique above requires you to already know the mechanism. Cluster randomisation requires you to know which boundary the interference does not cross. Switchback requires you to know roughly how long carryover lasts. The four-question audit requires you to know what is shared.

None of that is in your event log. Your log records that a control user's conversion fell. It does not record that they lost three auctions to someone who got an email, waited nine days for an onboarding call that used to take two, or saw a feed that had quietly been retrained on someone else's behaviour.

The control group is the only group that can tell you it was affected, and nobody ever asks it. This is the qualitative counterfactual applied to interference, and it is the same logic that makes the quasi-experimental designs work: confounders and interference channels are discovered from people who lived through the period, not extracted from aggregate counts. It is also why synthetic control carries an explicit no-spillover assumption on its donor pool - that assumption has to be defended with knowledge, not tested with data.

Four questions to put to a sample of your control group during or immediately after a test:

• "Did anything about the experience change for you over the last few weeks?" Control users who describe a change are reporting interference directly.
• "Was there anything you tried to do and could not, or had to wait longer for than usual?" This finds contested-resource and shared-capacity interference, including the queue effects that never appear as an event.
• "Did you hear about, or see, anything new from other people using the product?" This finds network channels, including the ones through Slack groups and communities you do not instrument.
• "When it did not work the way you expected, what did you do instead?" Workarounds are the failures that never get logged, and they are systematically more common in a degraded control arm.

Run the same questions on the treatment group and the difference between the two narratives is your interference story, told by the people it happened to.

The modern approach: making the control-group interview affordable

The reason nobody interviews the control group is arithmetic. A test runs for two weeks; scheduling, running and analysing forty interviews across two arms takes longer than the test itself, and by the time the analysis lands the decision has been made. So teams ship on the point estimate and hope SUTVA held.

This is the constraint AI-native research removes. Koji runs AI-moderated interviews - voice or text - with both arms in parallel, on the same discussion guide, at whatever sample size the decision deserves. A study fielded on the Monday a test concludes can return a coded, quantified read by mid-week, while the decision is still open. Where a traditional research cycle is measured in weeks and a legacy survey tool like SurveyMonkey gives you fixed questions that cannot probe an unexpected answer, an AI moderator hears "I could not get a slot" and asks the follow-up a human researcher would: how long did you wait, what did you do instead, had that happened before.

The design detail that matters for interference work is Koji's structured questions. Six types - open_ended, scale, single_choice, multiple_choice, ranking, and yes_no - can be mixed inside a single conversation. That combination is what makes an interference audit quantitative rather than anecdotal:

• yes_no: "Did you experience any delay getting help during this period?" gives you a clean rate to compare across arms.
• scale: "How long did you wait compared with what you expected?" gives a distribution, not a vibe.
• ranking: ask users to rank which parts of the experience got better or worse, and a control-arm degradation shows up as a rank shift.
• multiple_choice: enumerate the shared resources you identified in the audit and let users mark which ones bit them.
• open_ended: the AI probes the mechanism, which is the thing no closed question can find.

Because the structured answers are typed, the report aggregates them across both arms automatically, so "22% of control users reported a wait they did not expect, against 4% of treated users" is a chart rather than a fortnight of manual coding. That number is not a substitute for the corrected estimate. It is the evidence that told you the naive estimate was wrong, and the reason you redesigned the experiment before spending the traffic.

You do not need a causal inference PhD to run this. You need to ask the untreated half of your experiment whether the experiment happened to them.

Frequently asked questions

What is interference in an A/B test?

Interference occurs when one user's treatment assignment affects another user's outcome, violating the Stable Unit Treatment Value Assumption (SUTVA) that standard A/B analysis depends on. It happens when the two arms share something finite (inventory, support capacity), when users can influence each other (social and network features), or when a model serving both arms keeps learning from pooled behaviour. Interference biases the point estimate rather than adding noise, so a larger sample increases your confidence in a wrong number.

How much bias does interference actually cause?

Enough to reverse decisions. Blake and Coey found an eBay email campaign's effectiveness was "too large by a factor of around two" under naive user-level analysis, and the interference-corrected effect was not statistically different from zero. Holtz and colleagues found in a Management Science meta-experiment on Airbnb that at least 20% of the individually randomised treatment effect estimate was attributable to interference bias. A cited simulation of an online lodging market put the overstatement of market-wide outcomes as high as 90%.

Does interference always inflate the measured effect?

No, and assuming so is a common error. Blake and Coey show the direction depends on demand elasticity: the bias "will be positive when demand is elastic, and negative otherwise," with magnitude increasing as supply becomes more inelastic. Contested-resource interference typically inflates, because treatment gains come at control's expense. Shared-model interference typically deflates, because the treatment group's behaviour trains the model serving the control group and the arms converge. You cannot sign the bias without knowing the mechanism.

What is the difference between a switchback and a stepped wedge?

Both randomise timing rather than people, but they solve different problems. A stepped wedge randomises the order in which units cross over to the intervention because you cannot randomise who receives it; units cross once and never revert. A switchback flips the entire system back and forth between treatment and control many times, because splitting users would be invalid in the first place given interference. Choose a stepped wedge for operational or political constraints, and a switchback for interference.

How long should each switchback period be?

Long enough for carryover to dissipate, short enough to get sufficient periods for power. Bojinov, Simchi-Levi and Zhao derive optimal designs as a function of the order of the carryover effect, which they define as the length of time a treatment persists in impacting the outcome. In practice teams discard a washout window at the start of each period and analyse only the settled remainder, and they treat their carryover assumption as a guess that should be checked rather than a known constant.

Can qualitative research detect interference?

Yes, and it is often the only thing that can. The interference channels that matter are mechanisms - a contested queue, a workaround, a peer effect through a community you do not instrument - and mechanisms live in user accounts of what happened, not in aggregate counts. Interviewing the control group about whether anything changed, whether anything took longer than usual, and what they did instead when something did not work will surface channels no dashboard holds. Koji makes this affordable by running AI-moderated interviews on both arms in parallel and returning coded results in days rather than weeks.

Related Resources

• Quasi-Experimental Design: How to Measure Impact When You Cannot Run an A/B Test - the designs to reach for when randomisation is off the table entirely
• The Stepped Wedge: How to Randomise a Rollout You Cannot Randomise - the other design that randomises time, and when it is the right one
• Synthetic Control: How to Build a Comparison Group That Does Not Exist - where the no-spillover assumption on the donor pool comes from
• A/B Testing vs. User Research: When to Use Each - the broader case for pairing the number with the reason
• Structured Questions Guide - the six question types that make an interference audit quantitative
• AI User Research for Marketplaces - running continuous research on both sides of a two-sided platform