You Cannot Sample Your Way Out of a Badly Conditioned Metric (2026)

Answer first: every derived metric has an amplification factor — a single number saying how much it multiplies the error in its inputs. That factor is a property of the formula, not of the data, so it is fixed the moment someone writes the metric definition, months before the first respondent is recruited. Sample size shrinks input error. It does not touch the amplification. For a typical importance-minus-performance gap the amplification is 65, and holding that gap to plus/minus 10% would take 291 times your current sample. Chain two differences together and the requirement reaches 12,210 times. There is no realistic budget that reaches those numbers, which means the only available fix is a different metric.

This is the third article in a sequence. The first gives the law of propagation of uncertainty. The second shows subtraction destroying every significant figure in the inputs. This one asks the question those two raise and neither answers: given that a metric is badly behaved, what is the lever?

The number nobody computes

Numerical analysis has a name for the amplification: the relative condition number of the function. For a difference of two quantities A and B it has a form you can compute in your head:

Amplification = (|A| + |B|) / |A - B|

That is the worst-case multiplier on relative input error. If your inputs are independently and randomly wrong, the realised amplification is a bit lower — the errors partly cancel — but the same ratio drives it, and the ceiling is what you should design against.

Run it across the metrics a product organisation actually reports:

An amplification of 1 is an average. An amplification of 179 is a difference-in-differences. Everything a team argues about lives in the right-hand half of that table, and none of those numbers depend on how many people you interviewed.

Worked: what it costs to fix by sampling alone

Take the gap: importance 4.31, performance 4.18, each mean carrying a standard error of 0.08 from 225 respondents with an item standard deviation of 1.2. The gap is 0.13 with a standard error of 0.113 — 87% relative uncertainty, and a 95% interval that spans zero.

Suppose you insist on this metric and decide to buy precision. You want the gap known to plus/minus 10% of its value. The arithmetic:

Two hundred and ninety-one times the sample. Not because your instrument is bad — the instrument is fine, and 225 respondents give each mean a very respectable 1.9% relative error. The 291 comes entirely from the shape of the formula, and it scales with the square of the amplification, which is why the numbers get absurd so fast.

Chain a second subtraction on top — the difference-in-differences that every before-and-after study with a control group computes — and the amplification compounds to 179. To hold that to the relative precision of a single raw mean you would need 12,210 times the sample. At 225 respondents per cell today, that is 2.7 million interviews per cell.

This is what "you cannot sample your way out" means. It is not a rule of thumb. It is a multiplication.

Why sample size is the wrong lever

The Census Bureau states the useful half of the relationship in its ACS handbook: "In general, the larger the sample size, the smaller the SE of the estimates produced from the sample data."

True, and it is why the Bureau publishes multi-year files for small areas. But note exactly what it governs: the standard error of an estimate. Sampling acts on the inputs. The amplification acts on whatever the inputs deliver, and it is deaf to n.

Write it as two independent factors:

Output relative error = amplification × input relative error

Sampling drives the second factor down with the square root of n. The first factor is a constant chosen at metric-definition time. Multiply a constant of 179 by anything you can afford and the product is still large. Quadrupling your sample halves the input error and leaves you with 179 times half of what you had.

This is a different failure from the ones a research team is trained to look for. It is not sampling error, not bias, not a bad instrument, not a leading question. Every respondent could be perfectly representative and perfectly honest, every measurement perfectly executed — and the reported number would still be untrustworthy, because the arithmetic that produced it amplifies whatever irreducible error remains.

The defect is in the definition

It is worth naming the class of problem, because it is one a dashboard cannot show you and an audit of your fieldwork will never find.

There is a family of research numbers that cannot be rescued by better data collection. Some of them are uncomputable because the data you hold is the wrong data: you cannot get a rate right when the denominator is a different population than the numerator, and you cannot judge a classifier's precision without knowing the base rate in the population it ran on. Some are uncomputable because the instrument moved: a measurement system that cannot resolve the difference you are describing will not resolve it at any sample size.

This one is different, and in a way that is easy to miss because everything looks healthy. The data is good. Nobody's measurement failed. The defect is in the formula — chosen before collection began, usually by someone reasoning about business meaning rather than about arithmetic, and never revisited. A metric definition is a design decision with a numerical consequence, and it is the only research artefact that determines the precision of a result before a single respondent exists.

The practical implication is that metric definitions deserve the same review that survey instruments get. Nobody would field a questionnaire without someone checking the wording. Most teams ship a metric formula without anyone checking its amplification, and the formula is the more consequential of the two.

The fix is a different formula

Goldberg's remedy in What Every Computer Scientist Should Know About Floating-Point Arithmetic is the right instinct: "A formula that exhibits catastrophic cancellation can sometimes be rearranged to eliminate the problem." His own example rewrites x squared minus y squared as the product of (x minus y) and (x plus y), which computes the same quantity through a well-conditioned route.

The research translations:

Every row swaps a subtraction of aggregates for something measured closer to the respondent. That is the general principle: push the subtraction as far down toward the individual as it will go, and do the averaging afterwards. A difference of averages is fragile. An average of differences is not.

An amplification audit, in an afternoon

1. List every derived metric your organisation reports. Anything that is not a raw count, mean or proportion.
2. Write down the formula. If nobody can, that is the finding.
3. Compute the amplification for each. For a difference it is (|A| + |B|) / |A - B| with typical values plugged in.
4. Sort descending. Anything above about 10 is a metric whose reported precision is fictional.
5. For the top three, find the well-conditioned replacement using the table above.
6. Publish the amplification next to the metric definition, permanently. It is a one-time calculation and it settles the recurring argument about whether a two-point move is real.

Most teams find that their two or three most politically charged numbers sit at the top of that sorted list. That is not a coincidence: badly conditioned metrics generate large, meaningless swings, and large meaningless swings generate meetings.

How Koji helps

The replacements in the table above share a requirement: they need respondent-level data, paired within a person, on both quantities at once. That requirement is the reason most teams keep the badly conditioned version.

Both halves of the pair, from one person, in one sitting. Koji's structured questions run inside an AI-moderated interview across all six types — open_ended, scale, single_choice, multiple_choice, ranking and yes_no — so importance and performance, or before and after, come from the same respondent in the same conversation. That is what makes a paired within-person difference possible at all. A SurveyMonkey battery fielded to one panel and a satisfaction battery fielded to another cannot be paired, which forces the difference-of-aggregates form and its amplification of 65.

Ranking as a first-class instrument. When the decision is a priority order, the ranking question elicits it directly instead of reconstructing it from twelve fragile subtractions. One field change removes the amplification entirely.

Respondent-level export for the regression route. The difference-in-differences fix — model the interaction on individual records rather than subtracting four summary numbers — needs the records. Koji keeps analysis at the respondent level rather than handing back only aggregates, so the well-conditioned estimator is available rather than theoretical.

Panels you can actually re-contact. Wave-over-wave change measured on returning respondents is dramatically better conditioned than change measured on two fresh samples. Re-contacting a panel is an operational problem more than a statistical one, and running interviews in parallel with results arriving in real time is what makes a repeat wave a two-day job instead of a two-month one.

Thematic differences priced the same way. A change in theme frequency between two waves is a difference of proportions and amplifies exactly like a scale gap. Koji's automatic thematic analysis returns those frequencies over the complete transcript set, so the amplification can be computed rather than guessed, and the real-time report shows the components next to the delta.

A definition review that is cheap enough to happen. Because a Koji study is configured rather than fielded, changing a metric's underlying instrument — pair it, rank it, ask the proportion directly — costs an afternoon rather than a quarter. The amplification audit above is only useful if acting on its findings is realistic, and that is a tooling question as much as a statistical one. With a legacy survey platform, changing a metric's instrument means re-fielding; with Koji it means editing the brief and republishing the study.

Frequently asked questions

What is a condition number, in plain language?

It is how much a calculation multiplies the error in its inputs. A condition number of 1 means the output is as reliable as the input. A condition number of 65 means a 1% error in the inputs can become a 65% error in the answer. It depends only on the formula and the typical values involved, never on how the data was collected.

Why does a bigger sample not fix this?

Because output error equals amplification times input error, and sample size only touches the second term. Sampling reduces input error with the square root of n, so you need four times the sample to halve it. The amplification is a constant set by the formula. Multiplying a large constant by a slightly smaller number still gives a large number.

How do I calculate the amplification for my metric?

For a difference of two quantities, divide the sum of their absolute values by the absolute value of their difference. For a ratio or product, the amplification on relative error is approximately 1, which is why ratios are usually better behaved than differences. For anything more complex, compute each input's partial derivative and compare the resulting terms — the standard sensitivity-coefficient approach set out in JCGM 100:2008.

Is a difference-in-differences design always a bad idea?

No — the design is sound and often the only way to control for a confound. What is a bad idea is computing it from four published averages. Fit the model on respondent-level data with an interaction term, where the estimator uses all the variance in the sample rather than four summary numbers, and the precision is far better than the naive subtraction implies.

What amplification is too high?

As a working threshold: below 5 is fine, 5 to 10 warrants an interval on every mention, and above 10 means the reported precision is fictional and the metric should be replaced. Compute it once per metric and write it into the definition, because it does not change between studies.

Does this apply to qualitative research too?

Yes, wherever you subtract. A difference in theme frequency between two segments, a change in sentiment share between two waves, a before-and-after count of a pain point — all are differences of proportions and all amplify. The same fix applies: measure the change within a person where you can, and report the components alongside the difference where you cannot.

Related Resources

• Structured Questions in AI Interviews — paired scale items, ranking questions, and the respondent-level data the well-conditioned estimators need.
• Error Propagation in Research Metrics — where the sensitivity coefficients come from.
• Every Input Was Accurate and the Difference Was Not — the failure this article prices.
• Measurement System Analysis — the instrument-level limit that sits upstream of the formula.
• Mix Shift — another number that misleads while every input is correct.
• Statistical Power and Minimum Detectable Effect — sizing for an effect, once the metric is worth sizing for.