The Multiple Comparisons Problem: Why Slicing Data Into Segments Manufactures Findings (2026)

Answer first: the multiple comparisons problem is what happens when you run many statistical tests and judge each one against the same 5 percent threshold. Test 20 segments that are in truth identical and the probability of finding at least one significant difference is 64 percent, not 5 percent. Test 50 and it is 92 percent. That result is not a discovery. It is the arithmetic of your dashboard. The fix is not to stop slicing your data - it is to declare in advance how many tests belong to the decision, then control either the family-wise error rate (Bonferroni or Holm, when a single false positive is expensive) or the false discovery rate (Benjamini-Hochberg, when you are screening for leads to investigate).

Every research team has produced this artifact: a quarterly readout with a slide titled something like "Enterprise users in EMEA on annual plans are significantly less satisfied." The gap is real in the data. The chi-square test really does return p = 0.03. And the finding evaporates next quarter, replaced by a different segment with a different grievance.

Nothing went wrong in the analysis. The problem is that the analysis was one of forty, and only one of the forty made the slide.

Why one test at 5 percent becomes a coin flip at twenty

A significance threshold of 0.05 is a promise about a single test: if there is genuinely no difference, you will wrongly claim one about 5 percent of the time. That promise says nothing about what happens when you make the same claim repeatedly.

If your tests are independent, the probability that at least one of them produces a false positive is 1 minus the probability that none of them does:

Read the bottom row slowly. A team that runs a hundred comparisons against a dataset with no real differences in it at all will find something significant with near certainty. This is not a statement about weak data or sloppy analysts. It is true of perfectly collected data analysed by careful people.

The rate at which false positives arrive is also worth internalising: at the 5 percent threshold, roughly one in every twenty null comparisons will cross the line. Forty banner cuts produce about two spurious findings on average. Those two will not be labelled. They will look exactly like the real ones, because a p-value carries no memory of how many siblings it had.

The test count you actually ran is not the test count you reported

Most teams underestimate their own multiplicity by an order of magnitude, because they count the tests they presented rather than the tests they performed.

The real count is multiplicative. A standard quarterly tracker might carry:

• 5 banner variables (region, plan tier, tenure band, platform, company size)
• 9 headline metrics (satisfaction, likelihood to renew, ease of use, support quality, value perception, and four feature-specific ratings)
• 2 waves compared against each other

That is 5 x 9 = 45 segment-by-metric comparisons per wave before anyone looks at change over time, and the wave comparison roughly doubles it. Then, inside a banner with four levels, an analyst rarely tests "is there any difference across regions" - they look at six pairwise contrasts. The honest count runs into the hundreds.

Three further multipliers are almost always invisible in the writeup:

1. Redefinitions. Tenure banded as 0-6/7-12/13+ months is a different test from 0-3/4-12/13+. If you tried both, you ran both.
2. Outcome variants. Top-box share, top-two-box share, and mean score on the same scale item are three tests of one construct. Teams often try all three and report whichever separated the segments.
3. Filters. "Excluding respondents who finished in under 90 seconds" is a fork. So is "among users who have logged in at least once this month."

The discipline that fixes this is boring and effective: before analysis begins, write down the family of comparisons the decision depends on, and count it. If the number embarrasses you, that is the finding. Our guide to cross-tabulation analysis covers how crosstab banners multiply cells, and the companion piece on statistical power and minimum detectable effect explains why each of those thin cells is also underpowered - the two failures compound, because thin cells produce noisy estimates and noisy estimates produce more extreme p-values in both directions.

Family-wise error rate versus false discovery rate

There are two coherent things you might want to control, and choosing between them is a business decision rather than a statistical one.

Family-wise error rate (FWER) is the probability of making even one false claim across the whole family of tests. Controlling it at 5 percent means you are 95 percent confident that every rejection you made is real. This is the right target when a single wrong claim is expensive: a pricing change, a public marketing claim, a regulatory submission, a roadmap commitment.

False discovery rate (FDR) is the expected proportion of your claimed discoveries that are false. Yoav Benjamini and Yosef Hochberg introduced it in "Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing" (Journal of the Royal Statistical Society, Series B, 1995, 57(1):289-300), one of the most cited statistics papers ever written. Controlling FDR at 10 percent means you accept that roughly one in ten of the leads you chase will be a dead end. This is the right target when you are screening - generating a shortlist of segments worth a follow-up study.

The distinction matters because FWER control gets brutally conservative as the number of tests grows. Benjamini and Hochberg designed FDR precisely because family-wise procedures lose almost all their power when the family is large. If you apply Bonferroni to 200 comparisons you will find nothing, ever, and you will conclude that your research is uninformative when in fact your error control was mismatched to your purpose.

The three corrections, worked on the same ten p-values

Suppose a segment sweep produced ten p-values. Sorted ascending: 0.002, 0.009, 0.012, 0.031, 0.038, 0.049, 0.061, 0.140, 0.220, 0.480.

Uncorrected analysis reports six findings. Bonferroni and Holm report one. Benjamini-Hochberg at a 5 percent false discovery rate reports three.

The mechanics:

• Bonferroni divides the threshold by the number of tests. Simple, always valid, and needlessly harsh - it controls the chance of any error at all, which is a stricter guarantee than most product decisions require.
• Holm is a step-down version that is uniformly more powerful than Bonferroni and controls exactly the same thing. There is essentially no situation where Bonferroni is preferable to Holm on statistical grounds; Bonferroni survives because it can be done in your head. If you are correcting for FWER, use Holm.
• Benjamini-Hochberg sorts the p-values, compares the i-th smallest against (i/m) x q, finds the largest rank that passes, and rejects everything up to it. Note that it rejected rank 3 at 0.012 even though rank 3 would fail a Bonferroni test - the procedure borrows strength from the fact that several small p-values appeared together, which is exactly the pattern you would expect if some effects are real.

That last property is the practical argument for FDR in product research. A tracker where six of forty segments show movement is telling you something different from a tracker where one does, and Benjamini-Hochberg is sensitive to that difference while Bonferroni is not.

A correction cannot rescue a bad prior

Here is the part that error-rate control does not solve, and it is the more important half.

John Ioannidis made the argument famous in "Why Most Published Research Findings Are False" (PLoS Medicine, 2005): the believability of a positive result depends on the pre-study odds that the hypothesis was true in the first place, not only on the p-value. His formulation of positive predictive value shows that a finding is less likely to be true when studies are small, when effect sizes are small, when a greater number of relationships are tested with less preselection, and when there is greater flexibility in design and analysis. A segment sweep scores badly on every one of those.

Work the arithmetic on a realistic sweep. You test 100 segment hypotheses. Suppose 10 of them describe real differences and you have 80 percent power to detect them:

• True positives: 10 x 0.80 = 8
• False positives: 90 nulls x 0.05 = 4.5
• Significant results total: 12.5
• Share that are real: 8 / 12.5 = 64 percent

More than a third of your significant segment findings are wrong, and that is the optimistic case. Segment cells are small, so power is usually nowhere near 80 percent. At 20 percent power - a fair estimate for a 200-person cell chasing a 5-point difference - you get 2 true positives against 4.5 false ones, and only 31 percent of your significant findings are real. You would do better flipping a coin about which of your discoveries to believe.

Two implications follow, and they are more useful than any correction formula:

1. Preselection is worth more than correction. Ten pre-specified comparisons grounded in a mechanism you can articulate will beat a hundred exploratory ones even after you correct both. You are not just reducing the multiplier; you are raising the prior.
2. Power is part of error control. Under-powered testing does not merely miss real effects. It degrades the quality of the effects you do find, because it shrinks the numerator while leaving the false-positive denominator untouched. This is the same mechanism that makes selected-on-an-extreme comparisons unreliable, described in our guide to regression to the mean.

What corrections do not fix at all

Multiplicity control is a defence against one specific failure: threshold-crossing by chance. It offers no protection against any of these, all of which are common in commercial research:

• Analytic flexibility. Deciding which comparisons to run after seeing the data is a separate and larger problem, covered in our guide to p-hacking and researcher degrees of freedom. No correction can adjust for tests you did not count because you did not consciously run them.
• Measurement that cannot move. If your satisfaction scale is saturated at the top, no segment will show change and no correction is relevant. See ceiling and floor effects.
• A biased sample. A perfectly corrected p-value on a sample that over-represents your happiest customers is a precise answer to the wrong question. Our guide to survey sample size covers the sampling side.
• Repeat respondents. In a tracker, the same people answering wave after wave produces drift that has nothing to do with your product, described in panel conditioning.
• Correlated tests. Satisfaction, likelihood to renew, and value perception are not independent. The independence-based table above is a useful upper bound on how bad multiplicity gets, but real families of correlated metrics need procedures that account for the dependence.

Ron Kohavi, Diane Tang and Ya Xu, in Trustworthy Online Controlled Experiments, recommend a posture that generalises well past A/B testing: they invoke Twyman's law, "Any figure that looks interesting or different is usually wrong," and advise running validity checks specifically on breakthrough positive results. In segment analysis this translates cleanly - the most surprising cell on your slide is the one most likely to be an artifact, because surprise and extremeness are the same thing measured twice.

Multiplicity is a property of a threshold, not of understanding

There is one escape from the arithmetic, and it is not statistical.

The multiple comparisons problem exists because you are applying a decision rule to a number. Every number in every cell is a draw from a distribution, and drawing enough times guarantees extreme draws. But a mechanism is not a draw. When nine of eleven enterprise customers independently describe the same broken invoice-approval flow, unprompted, in their own words, that is not a p-value that got lucky. It is a causal account, and it does not need a Bonferroni correction because it was never a threshold crossing in the first place.

This is why the strongest segment findings are the ones that arrive with an explanation attached. A quantitative gap tells you where to look and carries a false-positive rate. A described mechanism tells you why and carries a coverage count instead. Report qualitative evidence as coverage - "nine of eleven" - never as a percentage, and never with a significance test bolted on.

The practical rule: treat every significant segment as a hypothesis, not a finding, until someone in that segment has explained it.

How Koji changes the arithmetic

Traditional survey tooling makes multiplicity worse by design. It gives you a crosstab engine, unlimited banners, and no mechanism for recording which comparisons you intended. The tool is optimised for generating tests and silent about counting them.

Koji is built for the opposite workflow.

The comparison set is declared in the brief, before fielding. Koji research briefs capture the decision the study exists to inform. That makes the family of comparisons an artifact of the study design rather than something reconstructed afterwards from a spreadsheet. When the analysis arrives, you already know the denominator.

Structured questions make the test count countable. Koji supports six structured question types - open_ended, scale, single_choice, multiple_choice, ranking, and yes_no - and the five closed types produce typed, pre-coded data. Because the response options are fixed at design time, the set of legitimate comparisons is fixed too. There is no opportunity to quietly redefine a band or recode a variable to find a gap. Our structured questions guide covers how to combine them with open-ended follow-up.

Every gap can be interrogated instead of tested again. This is the important one. When a legacy workflow finds a suspicious segment difference, the only affordable move is to slice the same dataset differently - which adds tests and makes the problem worse. With AI-moderated interviews, the affordable move is to ask the segment. Koji conducts moderated interviews at the cost and speed of a survey, so a follow-up study on the surviving hypothesis takes days rather than the six-to-eight weeks a traditional round would need. Re-testing an old dataset multiplies your error rate; collecting new evidence on one pre-specified question does not.

Automatic thematic analysis reports coverage, not significance. Koji clusters open-ended responses into themes with counts and supporting verbatims, which is the format that mechanism-level evidence should take. Teams using AI-assisted research tools consistently report time-to-insight measured in days rather than weeks, and the compression matters here specifically because it makes confirmation cheap enough to be routine. See our guide to thematic analysis for the method.

Voice interviews reach the segment that a crosstab only describes. A cell in a banner is 200 rows. Fifteen AI-moderated voice interviews with people from that cell will usually tell you within a week whether the gap is real, and if it is, why.

A pre-analysis checklist

Run this before you look at a single crosstab:

1. Write the decision. What action changes depending on the result?
2. List the comparisons that decision depends on. Be specific: metric, banner, contrast.
3. Count them. This is your family size, m.
4. Choose your error target. Confirmatory and expensive to get wrong: control FWER with Holm. Exploratory and screening for follow-up: control FDR with Benjamini-Hochberg at q = 0.10.
5. Check power for the smallest cell you intend to report. If the minimum detectable effect exceeds the difference you care about, do not run the test - resize the study.
6. Label everything else exploratory. Exploratory findings are permitted, useful, and must be reported with the word "exploratory" attached and a plan to confirm.
7. Route every survivor to a qualitative follow-up before it enters a roadmap document.

A team that does this will report fewer findings and will be right far more often. That trade is almost always worth making, and it is the same trade the research peer review gate is designed to enforce institutionally.

The bottom line

The multiple comparisons problem is not an obscure statistical technicality. It is the default failure mode of every dashboard, tracker and segmentation study that lets an analyst look at many things and report the interesting ones. The arithmetic is unforgiving: twenty tests, 64 percent chance of a phantom.

Correction procedures help, and you should use them - Holm when a false claim is expensive, Benjamini-Hochberg when you are generating leads. But the larger gain comes from upstream: fewer, better-motivated comparisons, adequately powered cells, and a cheap way to go back and ask people why. Modern AI-native research makes that last step affordable for the first time, which changes the economics of being careful.

Start free with 10 credits and run a pre-specified study with a declared comparison set, then confirm the survivors with AI-moderated interviews instead of another pass through the same crosstab.

Frequently asked questions

Do I need to correct for multiple comparisons in every study?

No. Correction applies to a family of tests that jointly inform one decision. A single pre-specified primary comparison needs no adjustment. The obligation arises when you are choosing what to report from among many candidates, which is precisely the situation in segment analysis, tracker reads, and multi-metric experiment readouts. If you are unsure whether you have a family, ask what you would have reported had a different cell been the significant one - if the answer is "that one instead," you have a family.

Should I use Bonferroni or Benjamini-Hochberg?

Use Holm (a strictly better version of Bonferroni) when a single false claim is costly: a pricing decision, an external marketing claim, a regulatory filing, a major roadmap bet. Use Benjamini-Hochberg when the output is a shortlist for further investigation and a dead end costs you one follow-up study. Most commercial segment analysis is screening, so Benjamini-Hochberg at q = 0.10 is the more appropriate default than the Bonferroni most teams reach for.

Does a correction make my analysis "safe"?

Not on its own. Correction controls one failure mode - threshold crossing by chance among tests you counted. It cannot help with tests you did not count, with analytic choices made after seeing the data, with an unrepresentative sample, or with a measure too insensitive to move. The base-rate arithmetic above shows that even a correctly executed sweep on an underpowered study can yield significant findings that are mostly false.

How many segments is it safe to look at?

There is no fixed number, because the answer depends on cell size and on how well-motivated the segments are. A more useful rule: look at as many as you like, but pre-specify which ones can produce a reported finding rather than a hypothesis. Exploration is not the problem; unlabelled exploration presented as confirmation is.

Is this the same thing as p-hacking?

They are related but distinct. Multiple comparisons is the arithmetic of running many tests, and it applies even when every test was pre-specified and honestly reported. P-hacking is the behavioural problem of making analytic choices contingent on the results, which inflates false positives even when only one test is ever reported. You need defences against both, and correction procedures only address the first.

Can qualitative research have a multiple comparisons problem?

Not in the same form, because qualitative analysis does not apply a significance threshold to a number. But the analogous risk is real: scanning many transcripts for anything that supports a favoured conclusion is selection by another name. The discipline that prevents it is coverage reporting - state how many participants raised a theme out of how many total, including the ones who contradicted it, as covered in our guide to thematic analysis.

Related Resources

• Statistical Significance in Survey Research - what a p-value does and does not tell you
• Statistical Power and Minimum Detectable Effect - sizing segments so the tests are worth running
• P-Hacking and Researcher Degrees of Freedom - the analytic-flexibility half of the problem
• Ceiling and Floor Effects - when the instrument, not the test, is the constraint
• Cross-Tabulation Analysis - reading crosstabs without fishing in them
• Structured Questions Guide - the six question types and when to use each
• How to Analyze Survey Data - the end-to-end analysis workflow