Every Input Was Accurate and the Difference Was Not: Catastrophic Cancellation in Research Metrics (2026)

Answer first: subtraction is the one arithmetic operation that can destroy every significant figure you paid for. When you subtract two numbers that are nearly equal, the absolute error survives intact while the result shrinks — so the relative error explodes. In a typical importance-minus-performance gap analysis, inputs measured to better than 2% produce a gap uncertain by 87%: a 40-fold amplification, from two averages that were each perfectly respectable. Numerical analysts call this catastrophic cancellation. Research teams call it Tuesday, and then reorganise a roadmap around it.

This is the second article in a three-part sequence. The first sets out the law of propagation of uncertainty and shows the friendly case, where averaging four sub-scores makes the composite more precise than any component. This one is the same law, same formula, opposite outcome. The only thing that changed is a minus sign.

The rule that turns on you

For a sum or a difference, absolute uncertainties combine in quadrature. That is a statement about the numerator of relative error. The denominator — the result itself — is not protected at all, and a difference of two similar quantities is small by construction.

So:

• Absolute uncertainty of the difference: roughly the same size as the inputs' (a little larger, by the square root of 2, if they are comparable and independent).
• Value of the difference: much smaller than either input.
• Relative uncertainty: absolute over value, and the denominator just collapsed.

David Goldberg, in What Every Computer Scientist Should Know About Floating-Point Arithmetic (ACM Computing Surveys, 1991) — still the canonical treatment — puts the mechanism precisely: when two rounded quantities are subtracted, "cancellation can cause many of the accurate digits to disappear, leaving behind mainly digits contaminated by rounding error."

His worked case is the discriminant in the quadratic formula, with b = 3.34, a = 1.22, c = 2.28. The exact value of b squared minus 4ac is 0.0292. But b squared rounds to 11.2 and 4ac rounds to 11.1, so the computed answer is 0.1 — wrong by a factor of more than three, from inputs that were each accurate to three significant figures.

Then the sentence that is the whole thesis of this article:

"The subtraction did not introduce any error, but rather exposed the error introduced in the earlier multiplications."

Nothing went wrong at collection. Nothing went wrong in the subtraction. The error was always there, hidden under two large numbers, and subtracting them took the cover off.

1.9% in, 87% out

Here is the same failure in the shape a product team actually meets it. You have run an importance-and-satisfaction study. For one attribute:

Both inputs are measured to better than two percent. The gap is uncertain by eighty-seven percent. The amplification factor is 46.9 — the formula multiplied your relative error by nearly fifty, and it did so silently, because the spreadsheet cell just says 0.13.

The 95% confidence interval on that gap runs from -0.09 to +0.35. It contains zero. It contains negative values. On this evidence you cannot say that importance exceeds performance for this attribute at all, let alone rank it against eleven others on a slide.

The same thing happens with percentages. Two independent samples of 600, one reporting 68.3% and the other 65.9%:

The 95% interval on the difference runs from -2.9 to +7.7 percentage points, and the test statistic is 0.885 — nowhere near significance. Group A might be ahead by eight points. Group B might be ahead by three. The amplification here is 40.6.

Net Promoter Score is a difference, and the correlation makes it worse

NPS is promoter share minus detractor share. It is a subtraction, so everything above applies — but there is a second effect that most teams get backwards.

Promoters and detractors come from the same respondents. In a multinomial sample the two shares are negatively correlated: every extra promoter is one fewer possible detractor. Intuition says correlated inputs should help. For a difference, negative correlation hurts, because subtracting a negatively correlated quantity is effectively adding.

Work it through for 50% promoters, 30% passives, 20% detractors at n = 400, which is an NPS of 30:

Ignoring the covariance understates the interval by 22%. And even done correctly, NPS on 400 responses carries a wider interval than a plain proportion on the same 400 responses — because it is a difference, and differences cost precision. (A Monte Carlo of 40,000 simulated samples returns a standard error of 3.89 pp against the analytic 3.91, so this is not a modelling artefact.)

Now take the delta between two waves — NPS 30 in Q1, NPS 33 in Q2, 400 responses each:

• Change: +3.0 points
• Standard error of the change: 5.50 pp
• 95% interval: -7.8 to +13.8
• Relative uncertainty of the change: 183%

Each wave's score is known to about 12% relative. The change between them is known to 183% — a fifteen-fold degradation, purely from the subtraction. The quarterly business review that opens with "NPS is up three points" is discussing a number whose sign is not established.

Overlapping confidence intervals are not the test

There is a widespread shortcut here that is worth naming, because it fails in both directions: eyeballing whether two error bars overlap.

The U.S. Census Bureau tells its own users not to do it, in its ACS handbook: "Data users should not rely on overlapping confidence intervals as a test for statistical significance because this method will not always provide an accurate result."

The correct procedure is the propagation rule, and the Bureau spells it out as seven steps: compute each standard error, square them, sum the squares, take the square root, divide the difference by that, and compare against 1.645 for 90% confidence, 1.960 for 95%, or 2.576 for 99%. That is a difference test on the difference — the only quantity you actually care about, and the one nobody computed an interval for.

Where this sits next to gap analysis and IPA

This article is deliberately narrow: it is about the arithmetic of a subtraction, not about the method that produces one.

• Customer Needs Gap Analysis owns the method — the three gap types, how to run the importance and satisfaction rounds, and the sample logic. It advises 40-100 respondents for the quantification phase, which is sound guidance for trusting the two averages.
• Importance-Performance Analysis owns the priority matrix — the four quadrants, stated versus derived importance, and how to read the plot.

What neither addresses, and what this article supplies, is that the sample size which makes the two averages trustworthy does not make their difference trustworthy. At 225 respondents per mean, the gap in the table above still carries 87% relative uncertainty. The components and the derived quantity have different precision requirements, and only the derived quantity is on the slide.

Two adjacent pieces sit at different altitudes again: measurement system analysis asks how many digits one instrument can resolve, which is upstream of everything here; common cause versus special cause asks whether a move over time is signal, which is a question about one metric's own history rather than about the formula that built it.

What to do instead

Report the components, not just the difference. "Importance 4.31, performance 4.18" is defensible. "Gap 0.13" is not, unless the interval comes with it. This is the single highest-value change and it costs nothing.

Put the interval on the derived number. Not on the inputs. The inputs are fine. Nobody is making a decision about the inputs.

Rank by something better conditioned. If you need a priority order across attributes, rank by importance among the low-performance set, or by the proportion of respondents rating importance high and performance low — a single proportion, well-conditioned, with an honest interval. A rank order built on differences of 0.13, 0.11 and 0.09 is a rank order of noise.

Rearrange the formula where you can. Goldberg's own remedy: "A formula that exhibits catastrophic cancellation can sometimes be rearranged to eliminate the problem." His example replaces x squared minus y squared with the product of (x minus y) and (x plus y), turning a catastrophic cancellation into a harmless one. The research analogue is to measure the difference directly — ask each respondent the paired question and average the within-person differences — rather than computing it from two separately estimated group means.

That last move is the strongest one available, because a within-person difference has no cancellation problem at all: the subtraction happens before the averaging, at the level of a single respondent, where both quantities are exact.

How Koji helps

Three of the four remedies above need something legacy survey tooling makes awkward, and one of them needs something it cannot do.

Paired, within-person differences. Turning importance-minus-performance into a directly measured quantity means asking one respondent about both, in the same session, with the pairing preserved. Koji's structured questions cover all six types — open_ended, scale, single_choice, multiple_choice, ranking and yes_no — inside a single AI-moderated interview, so the paired scale ratings and the reasoning behind them come from the same person in the same sitting. A SurveyMonkey importance battery and a separate satisfaction battery fielded a week apart to an overlapping-but-unknown sample cannot be paired, which forces you into the badly conditioned between-group estimate.

A ranking question instead of a difference. The ranking type sidesteps cancellation entirely: it elicits the priority order you were trying to reconstruct by subtracting, without ever forming a difference of two noisy means. When the decision is "what do we work on first," this is usually the better instrument, and it is one field change.

Sample sizes that make the derived number, not just the components, decision-grade. The arithmetic above is unforgiving: to hold a gap to a sensible relative precision you need far more respondents than you need for the averages. Traditional moderated research prices that out. Koji runs interviews in parallel and returns findings in real time, so you can watch the interval on the gap narrow and stop when it clears zero rather than when the recruiting budget does.

Voice or text, same paired structure. Koji's voice interviews collect the same structured scale and ranking responses as text interviews, so a paired design does not force respondents into a modality they will abandon halfway. Legacy panels that field an importance battery by email and a satisfaction battery by phone are, in effect, guaranteeing the unpaired estimator.

Themes with intervals attached. Koji's automatic thematic analysis produces theme frequencies, and a difference in theme frequency between two segments is subject to exactly the cancellation described here. Because Koji computes those frequencies from the full transcript set rather than from hand-coded samples, the counts behind them are complete, and the interval on the difference is honest rather than notional. None of this requires a statistics background — it requires the paired data, which is a study-configuration choice Koji makes in one field.

Frequently asked questions

What is catastrophic cancellation?

It is the loss of significant figures that happens when you subtract two nearly equal numbers that each carry some error. The absolute error is preserved by the subtraction, but the result is small, so the error becomes large relative to the answer. The term comes from numerical analysis; the underlying mechanism applies to any measured quantity, including survey estimates.

Why is my gap score so much less reliable than the two scores it came from?

Because the uncertainty of a difference is roughly as large as the uncertainty of its inputs, while the difference itself is much smaller than either input. Two averages of 4.31 and 4.18, each with a standard error of 0.08, give a gap of 0.13 with a standard error of 0.113 — an 87% relative uncertainty from inputs measured to under 2%.

Is NPS statistically weaker than a plain satisfaction score?

On the same sample, yes, for precision purposes. NPS is a difference of two dependent proportions, and the negative correlation between promoter and detractor shares inflates its variance. At n = 400 with a 50/30/20 split, NPS carries a 95% interval of plus/minus 7.7 points, against plus/minus 4.9 for a single proportion on the same respondents. NPS may still be the right business metric; it is simply not a precise one.

Can I just check whether the two confidence intervals overlap?

No. The Census Bureau warns its own data users against this specifically, because it does not always give the right answer. Non-overlapping intervals do imply a significant difference, but overlapping intervals frequently accompany a genuinely significant difference. Compute the standard error of the difference and test that.

How many more respondents do I need to fix a gap score?

Usually far more than is practical, because precision improves with the square root of sample size while the amplification factor is untouched by sample size. That trade-off is worked out in full, with the exact multiplier, in You Cannot Sample Your Way Out of a Badly Conditioned Metric. The short answer is to change the measurement rather than the sample.

Does this mean I should never report differences?

No — it means you should report them with an interval, and prefer designs that measure the difference directly. Paired within-person differences, ranking questions, and single-proportion formulations of the same business question are all well-conditioned alternatives that answer the decision without forming a fragile subtraction.

Related Resources

• Structured Questions in AI Interviews — paired scale items and ranking questions, the two instruments that avoid cancellation entirely.
• Error Propagation in Research Metrics — the general rule, and the case where averaging helps.
• You Cannot Sample Your Way Out of a Badly Conditioned Metric — why more data does not rescue this.
• Customer Needs Gap Analysis — the method that produces the subtraction.
• Importance-Performance Analysis — the priority matrix and its quadrants.
• Margin of Error in Surveys — the single-proportion baseline every comparison here is measured against.