When the Number Is Right and the Answer Is Wrong: Counting Themes by Mention or by Participant (2026)

Take one set of interview transcripts, one codebook, and one analyst. Count how often each theme was mentioned, and rank the themes. Then count how many distinct participants raised each theme, and rank them again. You now have two rankings of the same data. In the worked example below, the two rankings have a Spearman correlation of -0.10 - statistically indistinguishable from no relationship at all - and the theme that ranks first under one counting rule ranks fourth under the other.

Neither number is wrong. No data was corrupted, no participant lied, no formula contains a bug, and no step of the analysis was performed incorrectly. The report is a correct answer. It is a correct answer to a question nobody on the team consciously asked.

This is the failure mode that defeats every quality control aimed at finding defects, because there is no defect. It is worth naming precisely: the process produced accurate output under a configuration you did not choose.

The worked example

Twelve participants. Five themes. 51 total mentions coded across all transcripts.

Rank them both ways:

Every single theme changes rank. Pricing confusion goes from first to fourth. Mobile bugs go from third to last. Reporting gaps climb from fourth to second.

The headline sentence in the readout flips completely:

• Counted by mention, pricing confusion beats onboarding friction by 63.6% (18 versus 11).
• Counted by participant, onboarding friction beats pricing confusion by 125% (9 versus 4).

Same transcripts. Same coding. Same analyst. Opposite conclusions, both correctly computed.

Both numbers are genuinely correct

It is tempting to declare one of these the right answer. Resist that, because the argument collapses as soon as you specify the decision.

How much total customer conversation is this topic absorbing? is a real question, and mentions answer it correctly. It is the right input for sizing a support-content effort or a documentation rewrite - work whose cost scales with volume of discussion.

How widespread is this problem across our customers? is also a real question, and participants answer it correctly. It is the right input for a roadmap prioritization decision, where what matters is how many people are affected.

The two questions have different correct answers because they are different questions. The failure is not in the arithmetic. It is that the report says top theme without saying which question top is answering, and the reader supplies the wrong one.

Four participants out of twelve raised pricing confusion - one third of the sample. They raised it loudly, 18 times between them, averaging four and a half mentions each. Nine of twelve raised onboarding friction, three quarters of the sample, almost all of them once. A roadmap built from the mention ranking prioritizes the loud third over the quiet three quarters, and every number in it is accurate.

The concentration ratio tells you which report you are holding

There is a one-line diagnostic. Divide mentions by distinct participants:

• Ratio near 1.0 means each person raised it about once. The theme ranks the same under either rule, and the distinction does not matter.
• Ratio 3.0 or higher means a small group is generating most of the volume. This theme's position in a mention-ranked list is a fact about a few people's talkativeness, not about prevalence.

In the table above, Integrations sits at 1.00 and is rank-stable. Pricing confusion sits at 4.50 and moves three places. The concentration ratio predicts exactly which themes are configuration-sensitive, which means you can flag them automatically rather than discovering the problem when someone questions the deck.

The fragility is worth seeing directly. If one talkative participant contributed 9 of the 18 pricing mentions, dropping that single person from the study takes pricing from 18 mentions to 9 - from clear first place to tied for third. One participant, one rank order, no error anywhere.

Why this one hides from everything

Bainbridge's 1983 analysis of automated process control describes the mechanism better than anything written since about analysis pipelines. Her observation is that a well-behaved automatic system can mask the very condition you need to see:

Unfortunately automatic control can 'camouflage' system failure by controlling against the variable changes, so that trends do not become apparent until they are beyond control.

And on displays specifically: "it is possible for an interface which is ideal for normal conditions to camouflage the development of abnormal ones."

That is this failure exactly. A report that renders cleanly, cites its sources, passes every validation, and shows a confident ranked bar chart is ideal for normal conditions. Its very cleanliness is what prevents anyone from asking which counting rule produced it.

Consider what each standard control does here:

The last row deserves emphasis because it is the reflex response. More data does not converge these two rankings. It makes both of them more reliable, and the gap between them more durable. This is a failure that scales.

The same shape, four more places

Counting unit is the clearest instance, but the pattern - a legitimate configuration choice that silently determines the answer - recurs across the pipeline:

1. The denominator. Percentages computed over everyone invited versus everyone who completed. Both are defensible; they differ by exactly the completion rate, and nothing on the chart says which was used.
2. The quality filter. Dropping low-scoring interviews produces a cleaner, more coherent set of themes, and systematically removes the confused participants whose confusion was the finding.
3. The time window. A rolling 90 days and a fixed quarter cover almost the same data and can rank trends differently at the boundary.
4. Segment inclusion. Whether an other bucket is counted in the base changes every share in the table.

Each has the same signature: two correct answers, one report, no way to tell from the artifact which you are reading.

How Koji helps

The defense is not better checking. It is making the configuration part of the artifact and keeping the raw material addressable, which is where an AI-native platform like Koji has a structural advantage over a spreadsheet of tagged verbatims.

In Koji, every theme member carries the conversation it came from. Theme clusters retain their citations and their member list with conversation IDs attached, which means the distinct-participant count is not something you have to have thought to record in advance - it is recoverable from the artifact at any time. That is the difference between a configuration you can audit and one you would have to re-run the study to discover. A spreadsheet where someone tallied theme counts by hand has thrown that information away permanently.

The population denominator is an explicit parameter in Koji's report aggregation rather than an inferred default, so the base of a percentage is a stated setting instead of a property of whatever rows happened to be present.

Structured questions remove the ambiguity for the quantitative half of a study entirely. Across the six types - open_ended, scale, single_choice, multiple_choice, ranking, and yes_no - a typed answer has exactly one value per participant, so how many people said this has a single unambiguous meaning. The counting-unit problem is specific to free-text themes, which is a strong argument for asking the prevalence questions you care about as typed questions and using open-ended coding for discovering what to ask.

And because Koji's quality scores are explicit on a 1-5 scale rather than applied as a hidden cutoff, the filter that would silently remove your confused participants is a visible decision.

The practical difference shows up in how long the second count takes. Reconstructing per-participant counts from a legacy tagging spreadsheet means going back to the transcripts, if you still have them. In Koji it is a property of the report you already have, because the AI-moderated interviews and the automatic thematic analysis that produced the themes kept the link to the conversation throughout. That is the whole argument for doing analysis on a platform that retains provenance rather than in a tool that flattens it: the configuration you did not choose is still auditable afterwards.

What to report instead

Four changes, none expensive:

1. Report both counts, always. Two columns, mentions and participants. The cost is one column and it eliminates the whole failure.
2. State the counting rule in the chart title, not in a methods appendix. Themes by share of participants (n=12) is a complete claim. Top themes is not.
3. Publish the concentration ratio for any theme above 2.0, or flag it inline. It is the cheapest possible warning label.
4. Pick the rule from the decision, before you look. Prevalence questions get participant counts; volume questions get mention counts. Choosing after seeing both rankings is how you get a number that supports the conclusion you already had.

Common mistakes

Assuming your tool counts the way you think. Most theme charts do not say. The default is usually mentions, because mentions are what the coding step produces, and the reader almost always hears participants.

Treating a bigger sample as a fix. Both rankings converge on themselves, not on each other.

Comparing studies counted differently. A quarter-over-quarter theme comparison is meaningless if the counting rule changed between them, and nothing in either report will tell you it did.

Letting one participant carry a theme. Without a concentration check, a single talkative person can install a roadmap priority.

Filtering for quality and then reading prevalence. The participants who were confused are the ones most likely to score low and be dropped.

Frequently asked questions

Should I count themes by mention or by participant?

By participant for any question about prevalence - how common a problem is, how many customers are affected, what to prioritize on a roadmap. By mention for questions about volume - how much support burden or conversation a topic generates. Choose from the decision the data will inform, and decide before you see the rankings.

How different can the two rankings actually be?

Completely different. In the worked example above, twelve participants and five themes produce two rankings with a Spearman correlation of -0.10, and every theme changes position. The top theme by mention count lands fourth by participant count. There is no mathematical constraint keeping the two orders close.

What is the concentration ratio?

Mentions divided by distinct participants for a given theme. A ratio near 1.0 means one mention per person and a rank that is stable across counting rules. A ratio of 3.0 or more means a small group is producing most of the volume, and that theme's rank depends heavily on which rule you use. It is the fastest way to flag which findings are configuration-sensitive.

Why do normal quality checks miss this?

Because they look for defects and there is no defect. The data is valid, the arithmetic is right, the citations are real, and the numbers recompute identically. Validation, peer review, and significance testing all confirm a correct computation. None of them asks which question the computation answered.

Does a larger sample fix it?

No, and this is the counterintuitive part. Larger samples make both rankings more stable and preserve the difference between them. The gap is not sampling noise; it is two different quantities being measured. Adding data sharpens both measurements of two genuinely different things.

How do I stop this happening in my reports?

Report both counts side by side, put the counting rule in the chart title rather than a footnote, flag any theme with a concentration ratio above 2.0, and choose the rule from the decision before looking at the results. Keeping per-participant traceability on every theme is what makes the second count available at all.

Related Resources

• Structured Questions in AI Interviews - typed answers give prevalence a single unambiguous meaning
• Singleton Themes: Why One-Off Comments Are the Only Estimate You Have of What You Missed - the other end of the same distribution
• Levels of Measurement: Which Statistics Each Survey Question Type Allows - which operations a given question type permits
• The Complete Guide to Thematic Analysis - where the codebook and the counting rule are set
• How to Analyze Interview Results - the report this configuration silently shapes
• Negative Controls in User Research - testing a process on an input whose answer you know