Multi-Armed Bandits vs A/B Tests: When Adaptive Allocation Costs You the Answer (2026)

Answer first: a multi-armed bandit and an A/B test optimise different things, and the difference is not efficiency - it is what you are allowed to conclude afterwards. A bandit shifts traffic toward whichever variant is winning so far, which reduces the number of users exposed to the loser. An A/B test holds allocation fixed, which preserves your ability to estimate how much better the winner actually is. You cannot fully have both. In a benchmark clinical trial sized so that fixed randomisation reached 80% power, the most aggressive bandit rules delivered about 18.6% more successful outcomes during the trial - and cut statistical power to roughly a third of the fixed design's. Worse, the arm averages a bandit reports are systematically biased, because the algorithm's decision about what to sample next is entangled with what it has already seen.

Use a bandit when the decision is short-lived and you only need a winner. Use an A/B test when you need an effect size, when the result will be reused, or when somebody will later ask how much. This guide explains why, with the arithmetic.

The two objectives, stated honestly

Every experiment has a cost: some users get the worse variant. Two different fields have optimised against that cost in opposite directions.

Bandits minimise regret. Regret is the cumulative shortfall between what your users got and what they would have got had you assigned everyone the best arm from the start. A bandit algorithm - epsilon-greedy, Upper Confidence Bound, Thompson sampling, or a Gittins-index rule - continuously reallocates traffic toward arms that are performing well. Fewer users see the loser. The exploration/exploitation trade-off is exactly this: explore enough to find the best arm, exploit enough that finding it was worth something.

A/B tests minimise estimation error. A fixed 50/50 split is not an accident of laziness. It is the allocation that maximises statistical power for a two-arm comparison, and it keeps assignment independent of accumulated outcomes, which is what makes the standard estimator unbiased and the standard confidence interval honest.

These goals conflict. Every unit of traffic a bandit pulls away from the losing arm is a unit of information about the losing arm that you no longer have. And the losing arm's performance is half of the effect size.

The trade, in numbers

The clearest quantification comes from clinical trials, where the ethical stakes of assigning patients to a worse arm forced the field to study this carefully. Villar, Bowden and Wason, writing in Statistical Science (2015), summarise their own results in the abstract: "bandit approaches offer significant advantages, in terms of assigning more patients to better treatments, and severe limitations, in terms of their resulting statistical power."

Their benchmark is worth reproducing because it makes the trade concrete. They simulate a two-arm trial with a true success rate of 30% on control and 50% on treatment, sized at 148 participants so that fixed randomisation attains at least 80% power at a one-sided 5% significance level. Then they run the same trial under different allocation rules.

Read the bottom row twice. The Gittins-index design produced an average gain of 11 successfully treated patients, which the authors note "is only 4 fewer patients on average than the theoretical upper bound" achievable if every patient had been assigned the best treatment from the start. It is near-optimal at treating the people in the trial. And it arrives at the end with roughly a third of the power to demonstrate that the better treatment is better.

The mechanism is not mysterious. Under the Whittle-index rule the expected allocation was about 16 participants to control against about 132 to treatment. Sixteen observations is not enough to estimate a 30% rate with any precision, and the comparison is only as good as its weaker side. As the authors put it: "Adaptive rules have their power reduced because they induce correlation among treatment assignments."

The generalisable form: a bandit spends your sample on the arm you already believe in, and your uncertainty about the effect size lives almost entirely in the arm it stopped sampling.

The bias nobody warns you about

Losing power is a known, quantifiable cost. There is a second problem that is easier to miss because the number still looks perfectly reasonable.

Nie, Tian, Taylor and Zou proved in 2018 that adaptively collected data produce biased arm means. Their abstract: "we prove that when the data collection procedure satisfies natural conditions, then sample means of the data have systematic negative biases." The intuition they give is the useful one: "consider an adaptive clinical trial where additional data points are more likely to be tested for treatments that show initial promise. Our surprising result implies that the average observed treatment effects would underestimate the true effects of each treatment."

Why negative? An arm that gets an unlucky early run is abandoned, and its record is frozen at that unlucky level, never given the chance to regress upward. An arm that gets a lucky early run is sampled heavily, and the additional data drags its average back toward the truth. Winners get corrected; losers get stranded. The conditions are satisfied by most commonly used algorithms, including UCB and Thompson sampling.

The literature has since refined this. Later work showed that the net direction is not universally negative: optimistic sampling produces a negative bias, but optimistic stopping and optimistic choosing produce positive ones, so the sign in a given deployment depends on which of those your setup does. The practical conclusion survives either way, and it is the one to write on the wall: the arm averages on a bandit dashboard are not unbiased estimates, and treating them as effect sizes is a category error. They were produced by a procedure that chose what to look at based on what it had already seen.

The sign inversion worth understanding

There is a genuine irony here for anyone who has worked on interference between users. In an interfering marketplace, the fix is to randomise harder over time - a switchback flips the whole system back and forth so that time, not people, carries the randomisation, and the resulting estimate is clean.

A bandit does the opposite with the same axis. It lets accumulated results determine future allocation, which means treatment assignment becomes correlated with time. And anything else that drifts with time now confounds the comparison: your traffic mix, seasonality, a marketing campaign, the day of the week. Because late periods are heavily skewed toward the leading arm, a shift in population composition between early and late periods gets attributed to the arm rather than to the period. This is a Simpson's-paradox structure: each period can favour one arm while the pooled data favour the other.

Same lever, opposite sign. Randomising over time removes bias caused by units interacting. Adapting over time introduces bias caused by the algorithm interacting with its own history. The defences - time-blocked or batched allocation, stratifying the analysis by period, using inverse-propensity weights that account for the changing assignment probability - all amount to putting back the independence between assignment and time that the bandit removed on purpose.

So when is a bandit the right call?

Bandits are not a trap. They are the correct tool for a specific and common shape of decision.

Use a bandit when:

• The decision is short-lived and self-contained. Which of six subject lines for a one-off send, which hero image during a two-day sale, which of four push-notification wordings this morning. There is no future in which somebody needs the effect size.
• The cost of exposure is high and immediate. Real money, real harm, or a limited audience you cannot re-approach.
• You have many arms and low prior information. Bandits scale gracefully to a dozen creatives in a way a properly powered multi-arm A/B test does not.
• The environment is stable over the horizon of the decision.

Use a fixed A/B test when:

• You need a number, not a winner. Anything feeding a business case, a forecast, a pricing model, or a roadmap prioritisation needs a defensible magnitude with an interval around it.
• The result will be reused or generalised. "Personalised onboarding lifts activation by 4 to 7 points" gets cited for two years. "Variant C won" does not travel.
• You care about the loser. Understanding why the losing variant failed is often more valuable than shipping the winner, and a bandit deliberately starves you of data on it.
• The effect may be heterogeneous across segments. A bandit optimises the pooled average and then commits to it - see the next section, which is the sharpest reason of all.
• The metric matures slowly. Bandits reallocate on early signal, and early signal is exactly what the novelty effect corrupts. A variant that is merely new can win the exploration phase and then be locked in by the exploitation phase before its curiosity spike decays.

That last pairing is the most dangerous failure mode in practice: a bandit run on a short-horizon metric will reliably converge on whichever variant produces the largest transient reaction.

The failure mode nobody puts on the dashboard

A bandit converges. That is the point. But consider what convergence means when the effect is not the same for everyone.

Suppose a redesigned flow is better for 80% of your users and clearly worse for the 20% who use a screen reader, or who are on a slow connection, or who are power users with muscle memory for the old layout. The pooled average favours the new flow. A bandit will drive traffic to it and, in the limit, allocate almost nothing to the old one.

The subgroup that was harmed will never again be sampled on the arm that was better for them, and the data that would have revealed the harm is exactly the data the algorithm stopped collecting. A fixed A/B test at least leaves you a properly sized control arm in which the harm is measurable, if you go and look. A converged bandit leaves you an average and no way back. This is why heterogeneous treatment effects and adaptive allocation are the same problem viewed from two ends: optimising a mean is only safe when the mean describes somebody.

None of this is visible on a bandit dashboard, which shows arm means and allocation shares and looks like it is working exactly as designed. Because it is.

The modern approach: get the why the bandit cannot give you

Both designs share a blind spot. An A/B test tells you the magnitude of a difference. A bandit tells you which arm to serve. Neither tells you what happened to the person, and that is the input you need to decide whether the winner is genuinely better or merely newer, and whether the losing arm lost for a reason you should care about.

This is where AI-native research changes the economics. Traditional research cannot keep pace with an experimentation programme: recruiting, scheduling, moderating, transcribing and coding thirty interviews takes longer than the test, so teams ship on the metric and skip the mechanism. Koji runs AI-moderated interviews - voice or text - with users from each arm in parallel, probes their answers the way a human researcher would, and returns thematic analysis automatically. A read that used to take three weeks lands inside the decision window.

Two studies are worth running alongside any adaptive experiment:

Interview the arm the algorithm abandoned. A bandit stops sampling the loser, so the loser is where your knowledge is thinnest and your risk is highest. Ten interviews with users who had the deprecated variant will tell you whether it lost because it was worse or because it was unfamiliar.

Interview the users who stayed on the winning arm and disengaged anyway. They are the signature of a heterogeneous effect hiding inside a favourable average.

Structured questions are what make these studies produce numbers rather than anecdotes. Koji supports six types in a single conversation - open_ended, scale, single_choice, multiple_choice, ranking, and yes_no - so one twelve-minute interview yields both the mechanism and the rate:

• single_choice: which version the participant used, so every downstream cut is arm-aware.
• scale: perceived ease or confidence, giving a distribution you can compare across arms rather than a win/lose bit.
• ranking: have users rank what improved and what got worse; a harmed subgroup shows up as an inverted ranking long before it shows up in the pooled metric.
• yes_no: "Did anything about the new version make a task harder?" - a clean rate, per arm.
• open_ended: the AI follows up on the surprising answer, which is the only place the mechanism ever lives.

Because the structured answers carry types, the report aggregates them without manual coding, so you can say "31% of participants on the winning variant reported a task that got harder, concentrated among weekly-or-more users" while the experiment is still running. Unlike a legacy survey tool such as SurveyMonkey, where the questions are fixed in advance and cannot chase an unexpected answer, an AI moderator hears "I stopped using the shortcut" and asks which shortcut, how often, and what they do now.

You do not need a statistics background to use this well. You need to know that a bandit gives you a decision, an A/B test gives you a number, and neither gives you a reason.

Frequently asked questions

What is the difference between a multi-armed bandit and an A/B test?

An A/B test holds traffic allocation fixed, usually at an even split, and compares arms at the end; this maximises statistical power and keeps the estimate unbiased. A multi-armed bandit continuously reallocates traffic toward whichever arm is performing best so far, minimising regret - the cumulative cost of exposing users to inferior variants. The bandit gets better outcomes during the experiment; the A/B test gets a better estimate of how much better the winner is.

Do bandits really reduce statistical power that much?

Yes. In a simulated two-arm trial sized at 148 participants so fixed randomisation would attain at least 80% power, Villar, Bowden and Wason found that deterministic index rules such as the Gittins and Whittle indices produced power "approximately 3.5 times smaller than with a FR design." The cause is allocation imbalance: the Whittle rule assigned roughly 16 of 148 participants to control, and you cannot estimate a difference precisely when one side has almost no data. Randomised bandits like Thompson sampling and UCB lose much less power than deterministic index rules.

Are the conversion rates a bandit reports accurate?

Not as unbiased estimates. Nie, Tian, Taylor and Zou proved that under conditions satisfied by common algorithms including UCB and Thompson sampling, sample means from adaptively collected data carry systematic negative biases, because arms with unlucky early results are abandoned before they can regress toward the truth while heavily sampled arms get corrected. Later work showed the net sign can be positive or negative depending on whether the adaptivity is in sampling, stopping, or choosing. Either way, treat bandit arm means as decision inputs, not as effect sizes.

Can a bandit be confounded by time?

Yes, and this is underappreciated. Because a bandit's allocation depends on accumulated results, treatment assignment becomes correlated with time, so anything else that drifts over time - traffic mix, seasonality, day-of-week patterns, a marketing push - confounds the comparison. Late periods are dominated by the leading arm, so a change in population composition gets attributed to the arm instead of the period, which can produce a Simpson's paradox. Batched allocation, period-stratified analysis, and inverse-propensity weighting all help restore the independence the bandit removed.

When should I choose a bandit over an A/B test?

Choose a bandit for short-lived, self-contained decisions with many arms where you only need a winner and exposure to the loser is genuinely costly: subject lines, promotional creatives, a one-day merchandising choice. Choose a fixed A/B test whenever the magnitude matters - business cases, pricing, roadmap prioritisation - whenever the result will be cited later, whenever you need to understand why the losing variant lost, or whenever the effect is likely to differ across segments.

What is the biggest risk of running a bandit on a product feature?

Convergence on an average that hides harm. If a change helps most users and hurts a minority, the pooled average favours it, the bandit drives allocation toward it, and the harmed subgroup is never again sampled on the variant that suited them - while the data that would have revealed the harm is precisely what the algorithm stopped collecting. A secondary risk is early-signal capture: bandits reallocate on the first days of data, which is when novelty effects are strongest, so a merely-new variant can win exploration and get locked in before its curiosity spike decays.

Related Resources

• Interference Between Users: Why Your A/B Test Control Group Is Not a Control Group - the same time axis used to remove bias rather than create it
• Heterogeneous Treatment Effects: Nobody Experienced Your Average - what convergence on a mean costs the subgroup it does not describe
• Statistical Power and Minimum Detectable Effect - the power arithmetic a bandit spends
• Interim Analysis and Sequential Testing - how to look at a running study without inflating false positives
• The Novelty Effect: Why New Features Fool Your Metrics - the transient signal an adaptive algorithm is most likely to lock in
• Structured Questions Guide - the six question types that turn arm-level interviews into comparable rates