Process
How to run a pairwise comparison study
Six steps from option list to a ranking you can trust
Six steps for running a pairwise comparison study: write a tight list of comparable options, frame one clear question, decide between a full round and a sample of pairs, randomise order and position, ask why after some choices, and analyse with a consistency check. It suits feature prioritisation, message testing and naming studies. Koji lets you ask each pair as a choice question with an AI follow-up about why.
Where this appears
What's in this process
- Write a tight list of options: Same kind of thing, same level of detail, short descriptions in the customer's words.
- Frame one clear question: Pick one question, such as which would make the bigger difference, and use it for every pair.
- Decide full round or sample: Five options or fewer: everyone sees every pair. More: give each person a random sample.
- Randomise order and position: Shuffle the pairs and which option shows first, so position effects even out.
- Ask why, at least sometimes: A short follow-up after a choice turns a ranking into an explanation.
- Analyse and check consistency: Work out win rates or model scores, then look for circular choices before you trust the order.
Use this image
Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:
html
<a href="https://www.koji.so/visuals/pairwise-comparison-study-process"><img src="https://www.koji.so/visuals/pairwise-comparison-study-process/pairwise-comparison-study-process.png" alt="Process showing six steps to run a pairwise comparison study, from writing a tight option list to checking consistency in the results" width="1080" height="1080" /></a>
<p>Source: <a href="https://www.koji.so/visuals/pairwise-comparison-study-process">How to run a pairwise comparison study</a>, Koji</p>
