All frameworks and cheat sheets

Framework

Three ways to use a split ballot

Framework of three ways to use a split-ballot experiment: a one-off diagnostic, permanent randomization of arbitrary choices, and a decision test.
The decision test is the one that settles arguments in the room.

Three uses for a split-ballot survey experiment. Run it once to measure how sensitive a key metric is to wording, keep randomizing arbitrary choices such as answer order, or use two stakeholder framings to test whether a disagreement changes the decision. Use it when you plan a survey whose result someone will act on.

Where this appears

What's in this framework

  1. A diagnostic, run once: Run the key question and a fair alternative for one wave, measure the gap, then settle on one version.
  2. Permanent randomization: For arbitrary choices such as scale direction, keep splitting the sample. It turns a bias into variance.
  3. A decision test: Word each version the way a different stakeholder sees the problem. If both lead to the same decision, the debate was never about customers.

Use this image

Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:

html
<a href="https://www.koji.so/visuals/three-ways-to-use-a-split-ballot"><img src="https://www.koji.so/visuals/three-ways-to-use-a-split-ballot/three-ways-to-use-a-split-ballot.png" alt="Framework of three ways to use a split-ballot experiment: a one-off diagnostic, permanent randomization of arbitrary choices, and a decision test." width="1200" height="630" /></a>
<p>Source: <a href="https://www.koji.so/visuals/three-ways-to-use-a-split-ballot">Three ways to use a split ballot</a>, Koji</p>