{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-10-21T13:54:45.832Z"},"content":[{"type":"documentation","id":"7b04aad9-fac5-4aea-a1af-4c2c7802551f","slug":"pairwise-comparison-guide","title":"Pairwise Comparison: How to Rank Options One Pair at a Time","url":"https://www.koji.so/docs/pairwise-comparison-guide","summary":"Pairwise comparison ranks options by asking people to choose between two at a time. It dates to Thurstone's 1927 law of comparative judgment and the Bradley-Terry model (1952), and Cattelan's 2012 review notes people find comparing pairs easier than ranking a list. A full round needs n x (n - 1) / 2 pairs (45 for 10 options, 190 for 20), so larger studies give each participant a random sample. The guide gives a six-step process, analysis by win rate, Bradley-Terry scores and consistency checks (Saaty's AHP rule accepts a consistency ratio of 0.10 or less), and the wiki survey approach of Salganik and Levy (2015): in New York's PlaNYC survey, 1,436 respondents gave 31,893 responses on 25 seed and 464 contributed ideas, and 8 of the top 10 ideas came from respondents. It compares pairwise with ranking, rating and MaxDiff, and shows how Koji's single choice questions with AI follow-ups and ranking questions with average rank support the method.","content":"Pairwise comparison is a way to rank options by showing people two at a time and asking which they prefer. It works because picking between two things is easier and more honest than ranking a long list or rating each item on a scale. Its weakness is the number of pairs: a full round grows quickly, so for more than about eight options you show each person a sample of pairs and combine the answers statistically.\n\nThis guide covers how the method works, how many pairs you need, how to run and analyse a study, how wiki surveys scale it to hundreds of ideas, and when to use something else.\n\n## What pairwise comparison is\n\nIn a pairwise comparison (also called paired comparison), each question shows two options and asks the participant to choose one: \"Which of these two features would you rather we built first?\" Across many pairs and many people, the choices add up to an ordering of every option, often with a score that shows how far apart they are.\n\nThe method is old. Psychologist L. L. Thurstone set out a statistical model for paired comparisons in his 1927 \"law of comparative judgment\", and Ralph Bradley and Milton Terry published the model most tools still use in 1952. Today you'll find pairwise comparison in places as different as chess ratings, sports rankings, decision-making methods such as the Analytic Hierarchy Process and the human preference data used to train AI models.\n\nFor product and UX research it answers questions like:\n\n- Which of these 12 feature ideas matter most to customers?\n- Which of five value propositions is most compelling?\n- Which pain points should the roadmap tackle first?\n- Which of several names, headlines or designs do people prefer?\n\n## Why compare in pairs\n\nThree things make pairs attractive:\n\n- **Easier judgements.** Choosing between two options is a simpler task than holding ten in your head and ordering them. A review of paired comparison models by Manuela Cattelan (2012) makes the same point: people find it easier to compare two objects than to rank a list.\n- **No scale to interpret.** Ratings suffer from people using scales differently: one participant's 7 is another's 9. A choice between A and B has no such problem.\n- **Forced trade-offs.** When everything can be rated \"very important\", ratings flatten out. A pair makes the participant give something up, which is closer to how real decisions work.\n\nThe cost is volume. Every extra option adds a row of new pairs.\n\n## How many pairs you need\n\nA full round, where every option meets every other option once, needs n x (n - 1) / 2 pairs. The number grows faster than most people expect:\n\n![Table showing how many pairs a full pairwise comparison needs for 4 to 30 options, and the practical approach for each size](/visuals/pairwise-comparison-number-of-pairs/pairwise-comparison-number-of-pairs.png \"Past about eight options, nobody should see every pair\")\n\n| Options | Pairs for a full round | Practical approach |\n|---|---|---|\n| 4 | 6 | Every participant sees every pair |\n| 5 | 10 | Every participant sees every pair |\n| 8 | 28 | Full round is possible but tiring; consider sampling |\n| 10 | 45 | Each participant sees a random sample of pairs |\n| 15 | 105 | Random sample per participant, larger total sample |\n| 20 | 190 | Random sample per participant, or cut the list first |\n| 30 | 435 | Use an adaptive tool or narrow the list with another method |\n\nYou don't need every person to see every pair. If each participant answers a random selection and you have enough participants, every pair gets covered across the sample, and a model such as Bradley-Terry fills in the overall ranking.\n\n## How to run a pairwise comparison study\n\n![Process showing six steps to run a pairwise comparison study, from writing a tight option list to checking consistency in the results](/visuals/pairwise-comparison-study-process/pairwise-comparison-study-process.png \"Most pairwise studies go wrong at step one, with options that aren't comparable\")\n\n### 1. Write a tight list of comparable options\n\nEvery option should be the same kind of thing at roughly the same level of detail. \"Faster search\" against \"A complete redesign of reporting\" isn't a fair fight. Keep descriptions short and in the customer's language.\n\n### 2. Frame one clear question\n\n\"Which would make the bigger difference to your work?\" is a different question from \"Which would you pay more for?\" Pick one and use the same wording for every pair.\n\n### 3. Decide full round or sample\n\nUse the table above. For five or fewer options, show everyone every pair. Beyond that, decide how many pairs each person sees. Keep the load light enough that people stay focused, and check it in a pilot.\n\n### 4. Randomise order and position\n\nShuffle the order of the pairs and which option appears first. Position can nudge a choice, and randomising spreads that effect evenly across options.\n\n### 5. Ask why, at least sometimes\n\nA choice tells you which option won. It doesn't tell you why. Adding a short \"what made you pick that one?\" after some pairs turns a ranking into an explanation.\n\n### 6. Analyse and check consistency\n\nCalculate the results (see below), then look for contradictions before you trust the order.\n\n## Analysing the results\n\nThere are three common levels of analysis.\n\n**Win rate.** For each option, divide the number of times it was chosen by the number of times it appeared. It's simple, easy to explain and fine when every option appeared about equally often.\n\n**Bradley-Terry scores.** The Bradley-Terry model estimates a strength for each option so that the chance of A beating B depends on the gap between their strengths. It handles uneven coverage and gives scores you can compare on one scale. Most statistics packages include it.\n\n**Consistency checks.** If a participant prefers A to B and B to C, they should prefer A to C. Count how often that breaks (a \"circular triad\"). A few breaks are normal. Many suggest the options weren't clear or the participant wasn't paying attention. In the Analytic Hierarchy Process, which uses rated pairwise comparisons to weight criteria, Thomas Saaty's rule of thumb is to accept a set of judgements with a consistency ratio of 0.10 or less and revise anything above.\n\n## Pairwise at scale: wiki surveys\n\nPairwise comparison can also handle long lists that grow while the study runs. Matthew Salganik and Karen Levy at Princeton built a tool called All Our Ideas around this and described it in a 2015 _PLOS ONE_ paper. They argue such \"wiki surveys\" should be \"greedy, collaborative, and adaptive\": take as much information as each person will give, let participants add new options, and choose the next pair based on what's already known.\n\nTheir best-known case was a wiki survey for New York City's sustainability plan, PlaNYC. The Mayor's Office seeded it with 25 ideas, and residents contributed 464 more. In total 1,436 respondents gave 31,893 responses. Eight of the ten highest-scoring ideas were ones respondents had contributed, not the city's own seeds.\n\nThe lesson for product teams: if you only test your own list, you may miss the options participants care about most. Leave room for new ideas.\n\n## Running pairwise comparisons in interviews\n\nA survey gives you a ranking. An interview gives you the ranking and the reasons. When you run pairwise questions inside a conversation, a few habits help:\n\n![A participant weighing two printed options on a table while a researcher asks what would tip the choice](https://images.unsplash.com/photo-1677545215971-3fef13ddc6b5?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3w5NDUzMDN8MHwxfHNlYXJjaHwxfHx0d28lMjBwZW9wbGUlMjBjb21wYXJpbmclMjBjYXJkcyUyMHRhYmxlfGVufDF8MHx8fDE3OTI1NjE4MTJ8MA&ixlib=rb-4.1.0&q=80&w=1080)\n\n_Photo by [2H Media](https://unsplash.com/@2hmedia?utm_source=koji&utm_medium=referral) on [Unsplash](https://unsplash.com/?utm_source=koji&utm_medium=referral)_\n\n- **Keep pairs to the options that matter.** In an interview, three to six pairs is plenty. Use the rest of the time to ask why.\n- **Ask about the hard choices.** When a participant hesitates, that pair is close for them. Ask what would tip it.\n- **Listen for a third option.** \"Honestly, neither. What I need is...\" is often the most useful answer in the study.\n- **Check that options mean the same thing to everyone.** If a participant describes an option differently from how you wrote it, note it. That option may need rewording.\n\n## Pairwise vs ranking, rating and MaxDiff\n\n- **Ranking** asks people to order the whole list at once. It's fast for short lists, but the lower positions get less reliable as the list grows. See [choice and ranking questions](/docs/choice-ranking-questions-guide) and [average rank analysis](/docs/average-rank-ranking-question-analysis).\n- **Rating** asks for a score on each item. It's quick but suffers from everyone rating everything high. See [ranking vs rating questions](/docs/ranking-vs-rating-questions).\n- **MaxDiff** shows sets of four or five and asks for the best and the worst. It gets more information per screen than pairs. See [MaxDiff analysis](/docs/maxdiff-analysis-guide).\n- **Pairwise** is the simplest choice task and the easiest to explain, and it's the natural fit when you want to ask why after each choice.\n\n## Common mistakes\n\n- **Too many options.** Cut the list with a quick screen or rating question first.\n- **Mixing levels.** Small fixes compared with big bets always lose. Group like with like.\n- **No \"why\".** A ranking without reasons is hard to act on.\n- **Ignoring ties and contradictions.** If two options are within noise of each other, say so instead of declaring a winner.\n\n## How Koji helps\n\nKoji runs AI-moderated interviews by voice or text with structured questions built into the conversation, so you can collect pairwise choices and the reasons behind them in one study.\n\n- **Set up each pair as a choice.** Add a single choice question with the two options for each pair you want to test. In text interviews participants tap their answer, and the report charts how often each option was picked.\n- **Ask why automatically.** Each question can have AI follow-ups, up to three per question, so after a choice the interviewer asks what made the participant pick it. Those answers feed the thematic analysis in the report.\n- **Rank the full list when it's short.** For a short list, a ranking question lets participants drag items into order, and the report shows each item's average rank.\n- **Catch the third option.** Because the interview is a conversation, \"neither, what I really need is...\" gets captured and can surface as its own theme.\n\n![Koji question editor showing open ended, 1 to 10 scale and single choice questions with follow-up settings](/email/shots/shot-question-types.jpg \"Each pair is a single choice question with a follow-up asking why\")\n\n## Related Resources\n\n- [Forced-choice survey questions](/docs/forced-choice-survey-questions)\n- [Ranking vs rating questions](/docs/ranking-vs-rating-questions)\n- [MaxDiff analysis guide](/docs/maxdiff-analysis-guide)\n- [Average rank: analysing ranking questions](/docs/average-rank-ranking-question-analysis)\n- [Feature prioritization survey guide](/docs/feature-prioritization-survey-guide)\n- [Structured questions guide](/docs/structured-questions-guide)\n\n## Sources\n\n- Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. _Biometrika_, 39(3/4), 324–345.\n- Cattelan, M. (2012). Models for paired comparison data: A review with emphasis on dependent data. _Statistical Science_, 27(3), 412–433.\n- Saaty, T. L. (1980). _The Analytic Hierarchy Process_. McGraw-Hill.\n- Salganik, M. J., & Levy, K. E. C. (2015). Wiki surveys: Open and quantifiable social data collection. _PLOS ONE_, 10(5), e0123483.\n- Thurstone, L. L. (1927). A law of comparative judgment. _Psychological Review_, 34(4), 273–286.\n","category":"Research Methods","lastModified":"2026-10-21T05:50:12.64535+00:00","metaTitle":"Pairwise Comparison: Method, Pairs Formula and Analysis (2026)","metaDescription":"How pairwise comparison ranks options two at a time, how many pairs you need, a six-step study process, analysis methods and when to use MaxDiff instead.","keywords":["pairwise comparison","paired comparison method","pairwise comparison survey","pairwise ranking","bradley-terry model","wiki survey","feature prioritization research"],"aiSummary":"Pairwise comparison ranks options by asking people to choose between two at a time. It dates to Thurstone's 1927 law of comparative judgment and the Bradley-Terry model (1952), and Cattelan's 2012 review notes people find comparing pairs easier than ranking a list. A full round needs n x (n - 1) / 2 pairs (45 for 10 options, 190 for 20), so larger studies give each participant a random sample. The guide gives a six-step process, analysis by win rate, Bradley-Terry scores and consistency checks (Saaty's AHP rule accepts a consistency ratio of 0.10 or less), and the wiki survey approach of Salganik and Levy (2015): in New York's PlaNYC survey, 1,436 respondents gave 31,893 responses on 25 seed and 464 contributed ideas, and 8 of the top 10 ideas came from respondents. It compares pairwise with ranking, rating and MaxDiff, and shows how Koji's single choice questions with AI follow-ups and ranking questions with average rank support the method.","aiPrerequisites":["Basic familiarity with survey or interview question types"],"aiLearningOutcomes":["Explain what pairwise comparison is and why pairs are easier to judge than lists","Calculate how many pairs a full round needs and when to sample instead","Run a pairwise study in six steps","Analyse results with win rates, Bradley-Terry scores and consistency checks","Choose between pairwise, ranking, rating and MaxDiff"],"aiDifficulty":"intermediate","aiEstimatedTime":"9 min read"}],"pagination":{"total":1,"returned":1,"offset":0}}