Reference table
Bandit or A/B test? Six questions to decide
What each answer about your decision points to
A decision table for choosing between a multi-armed bandit and a fixed A/B test. Six questions cover what you need at the end, how long the decision lives, how many arms there are, whether the loser matters, whether effects differ by segment and how fast the metric matures. Use it in experiment planning before traffic is split. Whichever you run, Koji interviews can explain why an arm won or lost.
Where this appears
What's in this reference table
| Question about your decision | Points to a bandit | Points to a fixed A/B test |
|---|---|---|
| What do you need at the end? | A winner to serve | A magnitude with an interval |
| How long does the decision live? | Hours to days | Months to years |
| How many arms? | Many, with weak priors | Two or three, well specified |
| Does anyone need to know why the loser lost? | No | Yes |
| Could the effect differ by segment? | No, or you do not care | Yes |
| How fast does the metric mature? | Immediately | Slowly, or with a novelty decay |
Use this image
Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:
html
<a href="https://www.koji.so/visuals/bandit-or-ab-test-decision-table"><img src="https://www.koji.so/visuals/bandit-or-ab-test-decision-table/bandit-or-ab-test-decision-table.png" alt="Decision table with six questions for choosing a multi-armed bandit or a fixed A/B test, such as what you need at the end and how long the decision lives." width="1200" height="724" /></a>
<p>Source: <a href="https://www.koji.so/visuals/bandit-or-ab-test-decision-table">Bandit or A/B test? Six questions to decide</a>, Koji</p>
