Why Your Top Ten Themes Never Change (And What to Track Instead) (2026)

Bottom line up front: Your top ten themes look the same every month because theme frequencies follow a steeply skewed rank-frequency distribution, and the top of such a distribution is stable almost by construction. A month of new interviews cannot reorder a head that is built from thousands of prior mentions. This is not your team failing to find new insight; it is arithmetic. Stop re-reporting the top ten as if it were news, and start reporting four things that do move: rank changes, new entrants, the share of mentions sitting in the tail, and the concentration of the head.

The shape you are fighting

Word frequencies in natural language are notoriously uneven. A handful of items account for an enormous share of all occurrences, and an enormous number of items occur once or twice. The pattern approximately follows Zipf's law, under which an item's frequency falls off roughly in inverse proportion to its rank: the second-ranked item appears about half as often as the first, the tenth about a tenth as often, and so on.

Steven Piantadosi reviewed this literature in Psychonomic Bulletin and Review in 2014, noting that the frequency distribution of words has been a key object of study in statistical linguistics for some seventy years and that it approximately follows this simple mathematical form. Two of his conclusions matter for anyone reading a theme report.

The first is that human language has a highly complex and reliable structure in the frequency distribution over and above the classic law, and that prior data visualisation methods had obscured this fact. The second is that no prior account straightforwardly explains all of the basic facts about the distribution, so progress requires seeking evidence beyond the law itself.

Coded feedback themes are not words, and they do not obey Zipf's law with any precision. But they inherit the same qualitative shape, for an obvious reason: a few problems affect nearly everyone and get mentioned constantly, while a long tail of specific problems affects a few people each. That shape is enough to produce the effect you are living with.

Why the top ten cannot move

Suppose your corpus holds 8,000 coded mentions across 240 themes, and the head looks like this:

Now add a strong month: 400 new mentions, and suppose an unusually hot new issue takes 60 of them - a big month for a single new theme.

That new theme enters at 60 mentions. It ranks somewhere around 30th. To crack the top five it would need to sustain that rate for more than a year. A genuinely urgent, fast-rising problem is invisible in a top-ten list for months. Meanwhile pricing clarity stays at rank one, because 940 plus its share of the new month is still 940-ish, and nothing can catch it.

So the top-ten table is not measuring the present. It is measuring the accumulated past, dominated by whichever themes have been in the corpus longest and are most universal. Re-reading it monthly is like checking whether the pyramids are still there.

Note that this is a different problem from the one covered in why your quarterly metric shows a trend that is not there. That article is about how your sampling interval can manufacture a trend out of a cycle. This one is about how a skewed distribution suppresses real movement at the top of a ranked list. One invents signal; the other hides it.

The four things that actually move

Replace the top-ten bar chart with these.

1. Rank velocity. For each theme, the change in rank over the last period, and the change in its share of mentions. Sort by movement, not by size. A theme going from rank 40 to rank 18 is the most informative row in your report, and a top-ten list cannot contain it. Set a floor (say, at least 8 mentions) so that a theme moving from 1 mention to 3 does not top the chart.

2. New entrants. Themes appearing this period that did not exist before, with their distinct-speaker count attached. Most will be noise. The ones raised by several unrelated accounts in their first month are the single highest-value signal in feedback analysis, because they are new problems before they are big problems.

3. Tail mass. The share of all mentions falling outside your top twenty themes. If that share is growing, your product's problem space is diversifying - often a sign of a broadening user base, sometimes a sign that your taxonomy has stopped fitting. If it is shrinking, either you genuinely fixed the long tail or your coders have started forcing specific comments into generic buckets. Both readings are worth knowing, and neither is visible in a head-only report.

4. Head concentration. The share of mentions held by the top three themes. A rising figure means one or two issues are swamping everything else, which is useful: it tells you the corpus has a dominant story and that smaller themes are being crowded out of attention, not out of existence.

Plot it properly, because the chart is doing the hiding

Piantadosi's observation that earlier visualisation choices had obscured real structure in frequency distributions applies directly to feedback dashboards, where the default chart is actively unhelpful.

A top-ten horizontal bar chart is the worst available view of a skewed distribution. It shows you the part that does not change and truncates the part that does.

Better choices:

• Rank-frequency on log-log axes. Rank on the x axis, mentions on the y, both logarithmic. The whole distribution fits on one screen, and departures from a straight line are where the interesting structure lives. A kink partway down usually marks the boundary between your universal problems and your segment-specific ones.
• A movement chart. Rank this period against rank last period, one point per theme, with a diagonal line. Everything on the diagonal is static; the distance from the diagonal is the story.
• Tail mass over time. A single line. Unglamorous and more informative than any head-of-distribution view.

Avoid binning frequencies into buckets before plotting, which is precisely the kind of smoothing that erases the structure you are looking for.

A caution about what sits at rank one

Before you draw conclusions from the head of your distribution, check whether you put it there. If your interview guide asks every participant about pricing, then pricing will be your most-mentioned theme in every report you ever run, and its rank is a fact about your script rather than about your customers.

The diagnostic: compare the theme's rank among unprompted mentions - things raised before the relevant question was asked, or in response to open-ended questions on other topics - against its rank overall. A theme that is rank one overall and rank nineteen unprompted is an artefact of your instrument. This is also why the tail deserves attention: the tail is where things nobody asked about live, which makes it the only part of the distribution that is genuinely participant-driven.

Relatedly, the themes at the very bottom of the distribution - those mentioned exactly once - carry specific information about how much you have not yet heard. That is a separate calculation, covered in singleton themes and unseen coverage; this article is about the shape of what you did hear and how to read its movement.

How Koji handles this

• Consistent theme extraction over time. Rank velocity is only meaningful if a theme means the same thing in March as it did in January. Koji applies the same theme-extraction logic across the whole corpus, so a theme's movement reflects what customers said rather than a coder's evolving mental model. Human coding drift is the main reason most teams cannot trust their own rank changes.
• Stable question IDs make period comparison valid. Koji's study questions carry stable identifiers preserving traceability from the interview plan, through the AI interviewer, into analysis and report aggregation. That means you can compare this month's themes against last month's for the same question rather than against a blended average of differently-worded studies.
• Structured questions separate prompted from unprompted. Koji's six structured question types - open_ended, scale, single_choice, multiple_choice, ranking, and yes_no - fix exactly what was asked of every participant. Because the instrument is pinned, you can tell which mentions were prompted by a specific question and which arrived unbidden in an open_ended response, which is what makes the rank-one diagnostic above possible. The structured questions guide covers each type.
• AI follow-up questions grow the tail on purpose. A fixed survey can only collect the themes its author anticipated, which truncates the tail artificially. Koji's interviewer probes whatever a participant raises, including topics absent from the study plan, so new entrants can actually enter. A survey tool cannot produce a new entrant it has no question for.
• Real-time reports make velocity usable. Rank velocity and new entrants are only valuable while they are still early. Koji's reports update as interviews complete, so a theme climbing fast is visible during the month rather than in a quarterly readout after the decision was made.
• Quality scores to keep the tail honest. Koji scores each interview 1-5 across relevance, depth and coverage. Low-quality interviews inflate the tail with fragments; filtering them before reading tail mass keeps that metric interpretable.

Common mistakes

• Reporting the top ten every period. It is stable by construction and carries almost no new information.
• Concluding nothing changed because the head did not move. The head is the last place change appears.
• Ranking movement with no volume floor. A theme going from 1 mention to 3 will dominate any percentage-change sort.
• Reading rank one as your biggest problem. It may be your most-asked question.
• Letting the taxonomy change silently between periods. Merging two themes rewrites history and fabricates movement.
• Using a top-ten bar chart as the primary view. It truncates the only part of the distribution that moves.

Frequently asked questions

Why do my top customer feedback themes never change?

Because theme frequencies form a steeply skewed distribution in which the leading themes have accumulated far more mentions than any single period can add. If your top theme has 940 mentions and a strong month adds 400 mentions spread across all themes, no new theme can approach the top, and the leader keeps its rank automatically. The stability is a property of the distribution's shape rather than evidence that nothing is changing in your product.

What is Zipf's law and does it apply to feedback themes?

Zipf's law describes the tendency for an item's frequency to fall roughly in inverse proportion to its frequency rank, so the tenth most common item appears about a tenth as often as the most common. Piantadosi's 2014 review notes that word frequencies have been studied under this law for around seventy years and that real distributions carry substantial additional structure beyond it. Coded feedback themes do not follow the law precisely, but they share its qualitative shape, which is enough to produce the rank stability that makes top-ten reporting uninformative.

What should I report instead of the top ten themes?

Four measures that actually move: rank velocity, meaning each theme's change in rank and share with a minimum volume floor applied; new entrants, meaning themes appearing for the first time, with their distinct-speaker counts; tail mass, meaning the share of mentions outside your top twenty; and head concentration, meaning the share held by the top three. Together these show movement, emergence and whether your problem space is broadening or narrowing.

How do I spot an emerging issue early?

Watch new entrants and rank velocity rather than absolute size, and weight by how many unrelated accounts raised the theme in its first period. A new theme mentioned by five different accounts in one month is far more significant than an established theme gaining fifty mentions, because the former is a new problem and the latter is a known problem continuing. An urgent new issue can take a year to reach a top-ten list, so size-based reporting will always find it late.

Why is my most-mentioned theme the thing I ask everyone about?

Because asking a question guarantees responses to it. If every interview includes a pricing question, pricing will lead your theme ranking regardless of what customers care about most, and the ranking then describes your interview guide rather than your customers. The check is to compare a theme's rank among unprompted mentions, raised without a question inviting them, against its overall rank. A large gap means the theme's position is an artefact of the instrument.

How does Koji help track theme movement rather than theme size?

Koji extracts themes with consistent logic across the whole corpus and preserves each theme's link to its participant and originating question through stable question IDs, so period-over-period rank comparisons are valid rather than confounded by coding drift. Koji's AI follow-up questions let genuinely new themes enter the corpus even when no study question anticipated them, which is what makes the new-entrant signal possible, and real-time reports surface a fast-climbing theme while it is still early enough to act on.

Related Resources

• Structured Questions Guide - the six question types that pin the instrument so prompted and unprompted mentions can be told apart
• Singleton Themes and Unseen Coverage - what the bottom of the distribution tells you about what you never heard
• Theme Dispersion vs Mention Frequency - why each theme's count needs a distinct-speaker column beside it
• Why Your Quarterly Metric Shows a Trend That Is Not There - the opposite failure, where sampling interval invents movement
• Theme Co-Occurrence Analysis - finding the pairs that a ranked list cannot show
• How to Prioritize Customer Feedback - turning a theme report into a roadmap decision