{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-09-20T06:59:19.606Z"},"content":[{"type":"documentation","id":"2ed02093-f480-4316-a3d2-a54dbe0d49cd","slug":"average-rank-ranking-question-analysis","title":"Why Adding One Option Can Reverse Your Ranking Results (2026)","url":"https://www.koji.so/docs/average-rank-ranking-question-analysis","summary":"Average rank, the default summary for ranking questions, depends on which other options are listed. In a five-respondent example B led A on average rank 1.6 to 1.8, and adding an unrelated fourth option flipped it to A 1.8, B 2.2 with no respondent changing their view of A or B. First-place share and pairwise preferences stayed the same in both waves. Ranks are also ipsative, so average ranks cannot be compared across lists of different lengths. Report first-place share and a pairwise matrix alongside average rank.","content":"# Why Adding One Option Can Reverse Your Ranking Results (2026)\n\n**Answer first:** Average rank, the default summary for ranking questions in almost every survey tool, depends on which *other* options are in the list. Add or remove one option that nobody cares about and two items can swap places, even though not a single respondent changed their mind about those two items. Two summaries do not have this problem: **first-place share** (what fraction put each item first) and **pairwise preference** (for each pair, what fraction ranked one above the other). Report those alongside average rank, keep the option list fixed across waves, and never compare average ranks between lists of different lengths. Koji's `ranking` question type collects full orderings and can ask each participant why their top choice beat the runner-up, so you get the priority order and the reasoning behind it.\n\n## The worked example\n\nFive customers rank three features, A, B and C. Three of them rank A first, B second, C third. Two of them rank B first, C second, A third.\n\n| Summary | A | B | C |\n| --- | --- | --- | --- |\n| Average rank (lower is better) | 1.8 | **1.6** | 2.6 |\n| First-place share | **60%** | 40% | 0% |\n| Ranked above the other in the A-vs-B pair | **3 of 5** | 2 of 5 | - |\n\nAverage rank says B is the top priority. First-place share and the head-to-head comparison both say A.\n\nNow the team adds a fourth option, D, to the next wave. Nobody changes their view of A, B or C. The first three customers slot D in second (A, D, B, C); the other two put it last (B, C, A, D).\n\n| Summary | A | B | C | D |\n| --- | --- | --- | --- | --- |\n| Average rank | **1.8** | 2.2 | 3.2 | 2.8 |\n| First-place share | **60%** | 40% | 0% | 0% |\n| A-vs-B head-to-head | **3 of 5** | 2 of 5 | - | - |\n\nAverage rank now says A is the top priority. The roadmap review will see *A overtook B*. Nothing about A or B changed. The only thing that changed is that D pushed B down one position for three respondents, while A, ranked first by those same respondents, could not be pushed.\n\nFirst-place share and the head-to-head count gave the same answer in both waves. Average rank flipped.\n\n## Why average rank behaves this way\n\nAverage rank is the same arithmetic as a Borda count, the points-based voting scheme, and it inherits a property that social-choice theory has studied for more than two centuries: the result between two options can depend on a third, *irrelevant* option. Every option you add takes up a position, and it takes that position from whichever items respondents rank below it. Items that respondents love are shielded because they sit above the newcomer. Items in the middle absorb the displacement.\n\nTwo further properties follow from the arithmetic.\n\n**Ranks are ipsative, meaning every respondent's ranks add up to the same total.** With five options, each respondent's ranks sum to 15, so the average of the five average ranks is always exactly 3.0. With eight options it is always 4.5. One item can only improve if another gets worse. An item's average rank of 2.9 in a five-item list and 4.1 in an eight-item list says nothing about whether it gained or lost ground. Only the relative order within one fixed list is interpretable.\n\n**Partial rankings distort it further.** If respondents rank only their top three of eight, an item that appears in few rankings but always near the top can post an excellent average rank from a tiny base. Always check how many respondents actually placed each item before reading its average.\n\n## The three summaries to report instead of one\n\n| Summary | What it answers | Stable when options change? | Weakness |\n| --- | --- | --- | --- |\n| First-place share | Which item do the most people prioritise above everything else? | Yes, unless the new option is itself someone's first choice | Ignores everything below first place |\n| Pairwise preference matrix | For each pair, what share ranked one above the other? | Yes; adding D never changes an A-vs-B comparison | A table of k(k-1)/2 pairs is harder to present |\n| Average rank | Roughly, where does each item sit overall? | No | Depends on the rest of the list; hides polarisation |\n\nA useful pattern for a readout: lead with first-place share, support it with the pairwise matrix for the top few items, and show average rank only as an overall ordering within this study. If an item wins every one of its pairwise comparisons (a Condorcet winner, in social-choice terms), you can call it the top priority with confidence, whatever its average rank.\n\nPolarisation is the other thing average rank hides. An item ranked first by 40 percent and last by 40 percent will post a middling average, much like an item everyone ranks in the middle. Those are opposite findings: one is a segment-defining feature, the other a nice-to-have. First-place share and a look at last-place share separate them immediately.\n\n## Design rules that prevent ranking artefacts\n\n1. **Fix the option list for any tracked ranking.** If you must add an option, report the old list and the new list separately for one wave so the shift is visible.\n2. **Keep lists short.** Ranking more than about seven items is hard work for respondents, and the bottom half of a long list is mostly noise. For long lists, use [MaxDiff](/docs/maxdiff-analysis-guide) or a [top tasks](/docs/top-tasks-analysis-guide) vote instead.\n3. **Ask for a full ranking, or be explicit about partial ones.** If you collect top-three rankings, report first-place share and the number of respondents who placed each item, not average rank.\n4. **Do not use ranking when you need absolute levels.** A ranking tells you the order, never whether the top item is actually wanted. Pair it with a `scale` or `yes_no` question when absolute appeal matters. The trade-offs are covered in [ranking vs rating questions](/docs/ranking-vs-rating-questions).\n5. **Ask why the top choice beat the runner-up.** The gap between first and second is the most decision-relevant fact in a ranking, and the ranking alone cannot tell you whether it is a chasm or a coin flip.\n\n## How Koji handles ranking questions\n\nKoji's `ranking` type is one of six structured question types, alongside `open_ended`, `scale`, `single_choice`, `multiple_choice` and `yes_no`. In text interviews respondents drag options into order; in voice interviews the AI interviewer asks for the order conversationally. Either way the AI can probe after the answer, which is where rule 5 becomes automatic: participants are asked what separates their first choice from their second, and those explanations are themed across the whole study.\n\nThe Koji report shows the average position of each item, with every data point linked back to the conversation it came from. Each respondent's full ordering is preserved in the study data, so first-place share and the pairwise matrix can be computed directly from a data export or through Koji's MCP server. Because an AI moderator runs every interview, collecting full rankings from a larger sample costs the same researcher effort as a small one, which makes the more stable summaries practical rather than a luxury.\n\nRanking data is ordinal within each respondent, which places it in the ordinal row of the [levels of measurement](/docs/levels-of-measurement-survey-data) table: medians and shares are safe, and arithmetic on positions needs care.\n\n## Frequently asked questions\n\n### How do you analyse ranking question data?\n\nReport three things: first-place share (the percentage who ranked each item first), a pairwise preference matrix (for each pair, the share who ranked one above the other), and average rank as an overall ordering within this study only. First-place share and pairwise preferences are stable when options are added or removed; average rank is not.\n\n### Why is average rank misleading?\n\nBecause it depends on which other options are in the list. In a worked example with five respondents, B led A on average rank (1.6 against 1.8) until a fourth option was added, after which A led (1.8 against 2.2), even though no respondent changed their view of A or B. Average rank also hides polarisation: an item loved by some and disliked by others can look like one everyone ranks in the middle.\n\n### Can I compare average ranks across two surveys?\n\nOnly if the option list is identical. Ranks within a respondent always add up to the same total, so the average of all items' average ranks is fixed by the length of the list: 3.0 for five options, 4.5 for eight. An item's average rank moving from 2.9 to 4.1 across lists of different lengths tells you nothing about whether it gained or lost ground.\n\n### What is a pairwise preference matrix?\n\nA table that shows, for every pair of options, what share of respondents ranked one above the other. It is computed directly from full rankings, it never changes when an unrelated option is added, and an option that beats every other option head-to-head can be called the top priority with confidence.\n\n### How many items should a ranking question have?\n\nIdeally seven or fewer. Beyond that, respondents struggle to order the middle and bottom of the list and those positions become noisy. For longer lists, a MaxDiff exercise or a top tasks vote collects priorities more reliably.\n\n### How does Koji collect and report ranking questions?\n\nRespondents drag options into order in text interviews or give the order conversationally in voice interviews, and the AI interviewer can probe on why their top choice beat the runner-up. The report shows each item's average position with links to the underlying conversations, and full per-respondent orderings are kept in the study data for first-place share and pairwise analysis.\n\n## Related Resources\n\n- [Structured Questions Guide](/docs/structured-questions-guide) - the six question types and how to configure each one\n- [Ranking vs Rating Questions](/docs/ranking-vs-rating-questions) - when to force a trade-off and when to measure absolute levels\n- [Choice and Ranking Questions Guide](/docs/choice-ranking-questions-guide) - configuring ranking questions in Koji\n- [MaxDiff Analysis Guide](/docs/maxdiff-analysis-guide) - prioritising long lists reliably\n- [Levels of Measurement](/docs/levels-of-measurement-survey-data) - which statistics each question type allows\n- [When Relabelling the Scale Reverses Which Group Scores Higher](/docs/ordinal-scale-group-comparison-reversal) - the ordinal-data trap in group comparisons\n","category":"Analysis & Synthesis","lastModified":"2026-09-19T03:23:14.446293+00:00","metaTitle":"How to Analyze Ranking Question Data Without Average Rank Traps","metaDescription":"Average rank depends on which other options are listed. A worked example, plus the first-place and pairwise summaries that stay stable.","keywords":["how to analyze ranking question data","average rank survey","ranking question analysis","first place share","pairwise preference matrix","borda count survey","rank order data analysis"],"aiSummary":"Average rank, the default summary for ranking questions, depends on which other options are listed. In a five-respondent example B led A on average rank 1.6 to 1.8, and adding an unrelated fourth option flipped it to A 1.8, B 2.2 with no respondent changing their view of A or B. First-place share and pairwise preferences stayed the same in both waves. Ranks are also ipsative, so average ranks cannot be compared across lists of different lengths. Report first-place share and a pairwise matrix alongside average rank.","aiPrerequisites":["Familiarity with ranking questions","Basic understanding of averages"],"aiLearningOutcomes":["Explain why average rank depends on the other options listed","Compute first-place share and a pairwise preference matrix","Avoid comparing average ranks across different option lists","Design ranking questions that produce stable results"],"aiDifficulty":"intermediate","aiEstimatedTime":"11 min"}],"pagination":{"total":1,"returned":1,"offset":0}}