{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-08-24T23:08:17.625Z"},"content":[{"type":"documentation","id":"df4a8168-041d-4b4d-8ce8-b5febfaab33a","slug":"metric-condition-number-error-amplification","title":"You Cannot Sample Your Way Out of a Badly Conditioned Metric (2026)","url":"https://www.koji.so/docs/metric-condition-number-error-amplification","summary":"A derived metric's relative condition number states how much it multiplies input error, and it depends only on the formula and typical values. Sample size reduces input error but never the amplification, so an importance-performance gap with an amplification of 65 would need 291 times the sample to reach 10 percent relative precision. The fix is a differently shaped measurement.","content":"**Answer first:** every derived metric has an amplification factor — a single number saying how much it multiplies the error in its inputs. That factor is a property of the *formula*, not of the data, so it is fixed the moment someone writes the metric definition, months before the first respondent is recruited. Sample size shrinks input error. It does not touch the amplification. For a typical importance-minus-performance gap the amplification is 65, and holding that gap to plus/minus 10% would take **291 times** your current sample. Chain two differences together and the requirement reaches **12,210 times**. There is no realistic budget that reaches those numbers, which means the only available fix is a different metric.\n\nThis is the third article in a sequence. The [first](/docs/error-propagation-derived-research-metrics) gives the law of propagation of uncertainty. The [second](/docs/catastrophic-cancellation-metric-differences) shows subtraction destroying every significant figure in the inputs. This one asks the question those two raise and neither answers: given that a metric is badly behaved, what is the lever?\n\n## The number nobody computes\n\nNumerical analysis has a name for the amplification: the relative condition number of the function. For a difference of two quantities A and B it has a form you can compute in your head:\n\n**Amplification = (|A| + |B|) / |A - B|**\n\nThat is the worst-case multiplier on relative input error. If your inputs are independently and randomly wrong, the realised amplification is a bit lower — the errors partly cancel — but the same ratio drives it, and the ceiling is what you should design against.\n\nRun it across the metrics a product organisation actually reports:\n\n| Metric | Built from | Amplification | What it means |\n|---|---|---|---|\n| Equal-weight composite index | Mean of 4 sub-scores | **1.0** | Perfectly conditioned; output is more precise than any input |\n| Segment difference in a percentage | 68.3% - 65.9% | **55.9** | Inputs at 2.8% relative give a difference at 113% |\n| Importance-performance gap | 4.31 - 4.18 | **65.3** | Inputs at 1.9% relative give a gap at 87% |\n| Wave-over-wave NPS change | 33 - 30 | **21.0** | Waves at 12% relative give a change at 183% |\n| Difference-in-differences | (4.42-4.18) - (4.31-4.15) | **179.2** | Two subtractions, multiplied together |\n\nAn amplification of 1 is an average. An amplification of 179 is a difference-in-differences. Everything a team argues about lives in the right-hand half of that table, and none of those numbers depend on how many people you interviewed.\n\n## Worked: what it costs to fix by sampling alone\n\nTake the gap: importance 4.31, performance 4.18, each mean carrying a standard error of 0.08 from 225 respondents with an item standard deviation of 1.2. The gap is 0.13 with a standard error of 0.113 — 87% relative uncertainty, and a 95% interval that spans zero.\n\nSuppose you insist on this metric and decide to buy precision. You want the gap known to plus/minus 10% of its value. The arithmetic:\n\n| Requirement | Value |\n|---|---|\n| Needed standard error on the gap | 0.0066 |\n| Needed standard error on each mean | 0.0047 |\n| Current sample per mean | 225 |\n| **Required sample per mean** | **65,466** |\n| **Multiplier** | **291×** |\n\nTwo hundred and ninety-one times the sample. Not because your instrument is bad — the instrument is fine, and 225 respondents give each mean a very respectable 1.9% relative error. The 291 comes entirely from the shape of the formula, and it scales with the square of the amplification, which is why the numbers get absurd so fast.\n\nChain a second subtraction on top — the difference-in-differences that every before-and-after study with a control group computes — and the amplification compounds to 179. To hold *that* to the relative precision of a single raw mean you would need **12,210 times** the sample. At 225 respondents per cell today, that is 2.7 million interviews per cell.\n\nThis is what \"you cannot sample your way out\" means. It is not a rule of thumb. It is a multiplication.\n\n## Why sample size is the wrong lever\n\nThe Census Bureau states the useful half of the relationship in its ACS handbook: \"In general, the larger the sample size, the smaller the SE of the estimates produced from the sample data.\"\n\nTrue, and it is why the Bureau publishes multi-year files for small areas. But note exactly what it governs: the standard error of *an estimate*. Sampling acts on the inputs. The amplification acts on whatever the inputs deliver, and it is deaf to n.\n\nWrite it as two independent factors:\n\n**Output relative error = amplification × input relative error**\n\nSampling drives the second factor down with the square root of n. The first factor is a constant chosen at metric-definition time. Multiply a constant of 179 by anything you can afford and the product is still large. Quadrupling your sample halves the input error and leaves you with 179 times half of what you had.\n\nThis is a different failure from the ones a research team is trained to look for. It is not sampling error, not bias, not a bad instrument, not a leading question. Every respondent could be perfectly representative and perfectly honest, every measurement perfectly executed — and the reported number would still be untrustworthy, because the arithmetic that produced it amplifies whatever irreducible error remains.\n\n## The defect is in the definition\n\nIt is worth naming the class of problem, because it is one a dashboard cannot show you and an audit of your fieldwork will never find.\n\nThere is a family of research numbers that cannot be rescued by better data collection. Some of them are uncomputable because the data you hold is the wrong data: you cannot get a rate right when [the denominator is a different population than the numerator](/docs/mix-shift-rate-composition-decomposition), and you cannot judge a classifier's precision without knowing the base rate in the population it ran on. Some are uncomputable because the instrument moved: a [measurement system](/docs/measurement-system-analysis-research-metrics) that cannot resolve the difference you are describing will not resolve it at any sample size.\n\nThis one is different, and in a way that is easy to miss because everything looks healthy. **The data is good. Nobody's measurement failed. The defect is in the formula** — chosen before collection began, usually by someone reasoning about business meaning rather than about arithmetic, and never revisited. A metric definition is a design decision with a numerical consequence, and it is the only research artefact that determines the precision of a result before a single respondent exists.\n\nThe practical implication is that metric definitions deserve the same review that survey instruments get. Nobody would field a questionnaire without someone checking the wording. Most teams ship a metric formula without anyone checking its amplification, and the formula is the more consequential of the two.\n\n## The fix is a different formula\n\nGoldberg's remedy in *What Every Computer Scientist Should Know About Floating-Point Arithmetic* is the right instinct: \"A formula that exhibits catastrophic cancellation can sometimes be rearranged to eliminate the problem.\" His own example rewrites x squared minus y squared as the product of (x minus y) and (x plus y), which computes the same quantity through a well-conditioned route.\n\nThe research translations:\n\n| Badly conditioned | Well-conditioned replacement | Why it works |\n|---|---|---|\n| Group mean A minus group mean B | Paired within-person difference, then average | The subtraction happens per respondent, where both values are exact; the averaging then *reduces* error |\n| Importance minus performance, ranked | Share of respondents rating importance high and performance low | A single proportion; amplification 1 |\n| Wave-over-wave change in a score | Change measured on a returning panel | Respondent idiosyncrasy cancels inside each person |\n| Difference-in-differences on means | Regression on respondent-level data with the interaction term | Uses all the variance instead of four summary numbers |\n| A gap ranked across 12 attributes | A ranking question asked directly | No difference is ever formed |\n| Composite built from differences | Composite built from levels | Averaging is the best-conditioned operation available |\n\nEvery row swaps a subtraction of aggregates for something measured closer to the respondent. That is the general principle: **push the subtraction as far down toward the individual as it will go, and do the averaging afterwards.** A difference of averages is fragile. An average of differences is not.\n\n## An amplification audit, in an afternoon\n\n1. **List every derived metric your organisation reports.** Anything that is not a raw count, mean or proportion.\n2. **Write down the formula.** If nobody can, that is the finding.\n3. **Compute the amplification** for each. For a difference it is (|A| + |B|) / |A - B| with typical values plugged in.\n4. **Sort descending.** Anything above about 10 is a metric whose reported precision is fictional.\n5. **For the top three, find the well-conditioned replacement** using the table above.\n6. **Publish the amplification next to the metric definition,** permanently. It is a one-time calculation and it settles the recurring argument about whether a two-point move is real.\n\nMost teams find that their two or three most politically charged numbers sit at the top of that sorted list. That is not a coincidence: badly conditioned metrics generate large, meaningless swings, and large meaningless swings generate meetings.\n\n## How Koji helps\n\nThe replacements in the table above share a requirement: they need respondent-level data, paired within a person, on both quantities at once. That requirement is the reason most teams keep the badly conditioned version.\n\n**Both halves of the pair, from one person, in one sitting.** Koji's [structured questions](/docs/structured-questions-guide) run inside an AI-moderated interview across all six types — open_ended, scale, single_choice, multiple_choice, ranking and yes_no — so importance and performance, or before and after, come from the same respondent in the same conversation. That is what makes a paired within-person difference possible at all. A SurveyMonkey battery fielded to one panel and a satisfaction battery fielded to another cannot be paired, which forces the difference-of-aggregates form and its amplification of 65.\n\n**Ranking as a first-class instrument.** When the decision is a priority order, the ranking question elicits it directly instead of reconstructing it from twelve fragile subtractions. One field change removes the amplification entirely.\n\n**Respondent-level export for the regression route.** The difference-in-differences fix — model the interaction on individual records rather than subtracting four summary numbers — needs the records. Koji keeps analysis at the respondent level rather than handing back only aggregates, so the well-conditioned estimator is available rather than theoretical.\n\n**Panels you can actually re-contact.** Wave-over-wave change measured on returning respondents is dramatically better conditioned than change measured on two fresh samples. Re-contacting a panel is an operational problem more than a statistical one, and running interviews in parallel with [results arriving in real time](/docs/real-time-research-insights) is what makes a repeat wave a two-day job instead of a two-month one.\n\n**Thematic differences priced the same way.** A change in theme frequency between two waves is a difference of proportions and amplifies exactly like a scale gap. Koji's automatic thematic analysis returns those frequencies over the complete transcript set, so the amplification can be computed rather than guessed, and the real-time report shows the components next to the delta.\n\n**A definition review that is cheap enough to happen.** Because a Koji study is configured rather than fielded, changing a metric's underlying instrument — pair it, rank it, ask the proportion directly — costs an afternoon rather than a quarter. The amplification audit above is only useful if acting on its findings is realistic, and that is a tooling question as much as a statistical one. With a legacy survey platform, changing a metric's instrument means re-fielding; with Koji it means editing the brief and republishing the study.\n\n## Frequently asked questions\n\n### What is a condition number, in plain language?\n\nIt is how much a calculation multiplies the error in its inputs. A condition number of 1 means the output is as reliable as the input. A condition number of 65 means a 1% error in the inputs can become a 65% error in the answer. It depends only on the formula and the typical values involved, never on how the data was collected.\n\n### Why does a bigger sample not fix this?\n\nBecause output error equals amplification times input error, and sample size only touches the second term. Sampling reduces input error with the square root of n, so you need four times the sample to halve it. The amplification is a constant set by the formula. Multiplying a large constant by a slightly smaller number still gives a large number.\n\n### How do I calculate the amplification for my metric?\n\nFor a difference of two quantities, divide the sum of their absolute values by the absolute value of their difference. For a ratio or product, the amplification on relative error is approximately 1, which is why ratios are usually better behaved than differences. For anything more complex, compute each input's partial derivative and compare the resulting terms — the standard sensitivity-coefficient approach set out in JCGM 100:2008.\n\n### Is a difference-in-differences design always a bad idea?\n\nNo — the design is sound and often the only way to control for a confound. What is a bad idea is computing it from four published averages. Fit the model on respondent-level data with an interaction term, where the estimator uses all the variance in the sample rather than four summary numbers, and the precision is far better than the naive subtraction implies.\n\n### What amplification is too high?\n\nAs a working threshold: below 5 is fine, 5 to 10 warrants an interval on every mention, and above 10 means the reported precision is fictional and the metric should be replaced. Compute it once per metric and write it into the definition, because it does not change between studies.\n\n### Does this apply to qualitative research too?\n\nYes, wherever you subtract. A difference in theme frequency between two segments, a change in sentiment share between two waves, a before-and-after count of a pain point — all are differences of proportions and all amplify. The same fix applies: measure the change within a person where you can, and report the components alongside the difference where you cannot.\n\n## Related Resources\n\n- [Structured Questions in AI Interviews](/docs/structured-questions-guide) — paired scale items, ranking questions, and the respondent-level data the well-conditioned estimators need.\n- [Error Propagation in Research Metrics](/docs/error-propagation-derived-research-metrics) — where the sensitivity coefficients come from.\n- [Every Input Was Accurate and the Difference Was Not](/docs/catastrophic-cancellation-metric-differences) — the failure this article prices.\n- [Measurement System Analysis](/docs/measurement-system-analysis-research-metrics) — the instrument-level limit that sits upstream of the formula.\n- [Mix Shift](/docs/mix-shift-rate-composition-decomposition) — another number that misleads while every input is correct.\n- [Statistical Power and Minimum Detectable Effect](/docs/statistical-power-minimum-detectable-effect) — sizing for an effect, once the metric is worth sizing for.\n","category":"Analysis & Synthesis","lastModified":"2026-08-24T03:32:12.239714+00:00","metaTitle":"Metric Condition Number: Why More Data Cannot Fix a Badly Defined Metric (2026)","metaDescription":"A metric's amplification factor is set by its formula, not its data. Compute it, see what sampling would cost, and swap in the well-conditioned replacement.","keywords":["condition number","metric definition","error amplification","difference in differences","research metric design","sample size limits","well conditioned metrics"],"aiSummary":"A derived metric's relative condition number states how much it multiplies input error, and it depends only on the formula and typical values. Sample size reduces input error but never the amplification, so an importance-performance gap with an amplification of 65 would need 291 times the sample to reach 10 percent relative precision. The fix is a differently shaped measurement.","aiPrerequisites":["Understanding of error propagation in derived metrics","A metric definition you are able to change"],"aiLearningOutcomes":["Compute the amplification factor for any difference-based metric","Price what sampling would cost to rescue a badly conditioned metric","Run an amplification audit across your reported metrics","Replace a difference of aggregates with a paired or respondent-level estimator"],"aiDifficulty":"advanced","aiEstimatedTime":"13 min"}],"pagination":{"total":1,"returned":1,"offset":0}}