{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-08-09T13:16:16.263Z"},"content":[{"type":"documentation","id":"7118dc5d-e1f7-40f2-88db-4318a5174211","slug":"equivalence-testing-no-difference","title":"How to Prove There Is No Difference: Equivalence Testing for Product Research (2026)","url":"https://www.koji.so/docs/equivalence-testing-no-difference","summary":"Standard significance testing cannot conclude that two things are the same; it can only fail to detect a difference. Equivalence testing, usually implemented as TOST (two one-sided tests), asks whether the observed difference falls entirely inside a pre-declared smallest effect size of interest. Freiman et al. (1978) found 67 of 71 trials reported as negative had over a 10 percent risk of missing a true 25 percent improvement. Running the difference test and the equivalence test together produces four outcomes rather than two, including the commonly mislabelled inconclusive case. FDA average bioequivalence applies the same logic with an 80 to 125 percent bound on a 90 percent confidence interval.","content":"A standard significance test can never conclude that two things are the same. It can only fail to show that they differ, which is a completely different statement - and the difference is the subject of one of the most-cited one-page papers in medical statistics, Altman and Bland, \"Absence of evidence is not evidence of absence\" (*BMJ*, 1995, 311(7003):485). To claim that two versions, two segments, two pricing tiers or two vendors are equivalent, you need an equivalence test: you declare in advance the smallest difference that would matter, then test whether the observed difference is small enough to rule that out. Product teams make no-difference decisions constantly - sunsetting a feature, migrating a platform, matching a competitor price - and almost never run the test that licenses them.\n\n**Key takeaways**\n\n- A non-significant result has three possible meanings: there is no effect, there is an effect you could not detect, or you never defined what \"no effect\" would look like. Only equivalence testing separates them.\n- Freiman and colleagues re-examined 71 trials reported as negative and found 67 of them had more than a 10 percent risk of missing a genuine 25 percent improvement (*New England Journal of Medicine*, 1978, 299(13):690-694).\n- The standard method is TOST - two one-sided tests - which asks whether the true difference is smaller than a pre-declared bound in both directions (Lakens, *Social Psychological and Personality Science*, 2017, 8(4):355-362).\n- Regulators already work this way. FDA average bioequivalence requires the 90 percent confidence interval for the ratio of log-transformed exposure to fall entirely within 80 to 125 percent - an explicit, numeric definition of \"close enough\".\n- The hardest part is not the statistics. It is writing down the smallest effect size of interest before you field, which is a business decision, not a statistical one.\n\n## The claim a p-value cannot make\n\nNull hypothesis significance testing is built around a single asymmetry. You assume there is no difference, then look for evidence strong enough to reject that assumption. When the evidence is not strong enough, you have failed to reject - which is not the same as having established the assumption was true. A jury that acquits has not proved innocence.\n\nProduct readouts routinely translate \"we failed to reject\" into \"there is no difference\", and then a decision gets made on it: keep the old flow, sunset the feature, treat the two segments as one. Our guide to [statistical significance in survey research](/docs/statistical-significance-survey-research) covers what a p-value does say; this guide covers the claim it structurally cannot make.\n\n**The perverse consequence is that the standard framework rewards a weak study with a clean-sounding answer.** Run 40 people instead of 400 and you are far more likely to come back with \"no significant difference\", which reads in a deck as a decisive finding. The less you spend, the more confidently null your result appears. Equivalence testing inverts that incentive: under TOST, an underpowered study returns \"inconclusive\", which is both the truth and considerably harder to present as a result.\n\n## How badly this goes wrong: the evidence\n\nThe definitive demonstration is old and has never been improved on. Freiman, Chalmers, Smith and Kuebler took 71 randomised controlled trials that had been reported as negative and asked a simple question: were they big enough to have detected a clinically meaningful improvement if one existed?\n\n| Question asked of 71 \"negative\" trials | Trials |\n|---|---|\n| Had greater than 10 percent risk of missing a true 25 percent improvement | 67 |\n| Had the same risk of missing a true 50 percent improvement | 50 |\n| 90 percent confidence interval still allowed a 25 percent improvement | 57 |\n| 90 percent confidence interval still allowed a 50 percent improvement | 34 |\n\nSource: Freiman JA, Chalmers TC, Smith H Jr, Kuebler RR, *New England Journal of Medicine*, 1978, 299(13):690-694.\n\nIn 34 of these 71 trials the data were entirely compatible with the treatment being fifty percent better than control - and every one of them had been written up as showing no difference. The authors concluded that many therapies labelled as no different from control had not, in their words, received a fair test.\n\nAltman and Bland revisited the problem seventeen years later and reported that a follow-up sample of trials from 1988 showed no improvement over the 1978 cohort. Their recommendation was the one that still holds: stop reading the p-value on its own and look at the confidence interval, because the interval tells you what the study has actually ruled out.\n\nThat last sentence is the whole method in miniature. **An equivalence test is a formalised way of asking whether your confidence interval is narrow enough to exclude everything you would have cared about.**\n\n## The method: two one-sided tests\n\nTOST was formalised by Schuirmann in the context of bioequivalence and popularised for behavioural research by Daniel Lakens. The procedure has three steps.\n\n**Step 1: declare the smallest effect size of interest (SESOI).** This is a pair of bounds, an upper and a lower, describing the region around zero you are willing to call \"no meaningful difference\". A 0.4 point movement on a five-point satisfaction scale. A two percentage point difference in task completion. A five percent difference in willingness to pay.\n\n**Step 2: run two one-sided tests.** The first asks whether the true difference is significantly greater than the lower bound. The second asks whether it is significantly less than the upper bound. Both must be significant. Equivalently, and more usefully for a readout, you check whether the 90 percent confidence interval around the observed difference falls entirely inside your bounds.\n\n**Step 3: read the result alongside the ordinary significance test.** This is the step teams skip, and it is where the interesting cases live. Running both tests produces four outcomes, not two.\n\n| | Equivalence test significant | Equivalence test not significant |\n|---|---|---|\n| **Difference test significant** | Real but too small to matter - the effect exists and you can stop caring about it | A meaningful difference - act on it |\n| **Difference test not significant** | Statistically equivalent - you may claim no meaningful difference | Inconclusive - the study could not tell, and the honest report says so |\n\nAdapted from Lakens (2017), *Social Psychological and Personality Science*, 8(4):355-362.\n\nThe bottom-right cell is where most product research actually lands, and under conventional practice it gets reported as the bottom-left cell. That single mislabelling is responsible for more bad no-difference decisions than any other error in commercial research.\n\nThe top-left cell is the one nobody expects. A difference can be real, statistically detectable, and simultaneously proven too small to be worth acting on. Large samples find real effects of no consequence all the time, and without an equivalence bound there is no principled way to say so out loud.\n\n## Regulators have done this for forty years\n\nIf declaring a bound in advance feels arbitrary, note that an entire industry runs on it. FDA average bioequivalence for generic drugs requires that the 90 percent confidence interval for the ratio of the test to reference product, on log-transformed exposure measures such as AUC and Cmax, falls entirely within 80 to 125 percent. Not \"was not significantly different\". A number, chosen in advance, that both edges of the interval must sit inside.\n\nThe asymmetric-looking bounds are an artefact of the logarithmic scale: 80 percent and 125 percent are reciprocals, so the window is symmetric in log space. What matters for our purposes is that the pharmaceutical industry decided decades ago that \"prove these are the same\" required a stated tolerance, and then wrote the tolerance down. There is no reason a product organisation deciding whether a rebuilt checkout is as good as the old one should hold itself to a looser standard than a generic ibuprofen.\n\n## The no-difference decisions you are already making\n\nThe reason this is not an academic exercise is that a surprising share of high-cost product decisions are equivalence claims wearing null-result clothing.\n\n- **Sunsetting.** \"Removing this feature will not hurt satisfaction.\" That is a claim of equivalence between the product with and without it.\n- **Platform migration and redesign.** \"The new version performs as well as the old one.\" Equivalence, on whatever metric you care about, with an implicit bound nobody stated.\n- **Pricing parity.** \"Moving to the competitor price point will not change perceived value.\" Equivalence.\n- **Vendor or model swaps.** \"The cheaper model gives answers of the same quality.\" Equivalence, and increasingly the most common version of this decision in AI-enabled products.\n- **Segment collapsing.** \"Enterprise and mid-market want the same thing, so we can run one roadmap.\" Equivalence between two groups, usually asserted because a segment comparison came back non-significant on a thin cell. Our guide to [the multiple comparisons problem](/docs/multiple-comparisons-problem) covers why those thin cells are unreliable in both directions.\n- **Accessibility and localisation parity.** \"The experience is equally good for screen reader users.\" Equivalence, and one where getting it wrong is not only expensive.\n\nEach of these gets decided on a non-significant result roughly as often as it gets decided on an equivalence test, which is to say almost always on the former.\n\n## Setting the bound without kidding yourself\n\nThe SESOI is where teams stall, because it feels like it requires a precision nobody has. It does not. It requires a decision, and there are three defensible ways to reach one.\n\n**Anchor it to the decision.** The best bound is the smallest difference that would change what you do. If a two-point NPS gap would not alter the roadmap, two points is inside your equivalence region. This is the version to reach for first, because it is the only one that is genuinely a business question and because it forces a conversation worth having.\n\n**Anchor it to a known benchmark.** If a previous study established that a meaningful improvement in this metric looks like six percentage points, half that is a reasonable bound. Our guide to [statistical power and minimum detectable effect](/docs/statistical-power-minimum-detectable-effect) covers how to establish these reference points, and the MDE from a prior study is often the cleanest available anchor.\n\n**Anchor it to what you can afford to detect.** Lakens suggests that when no theoretical bound exists, researchers set the bounds to the smallest effect they have sufficient power to detect given their resources. This is the weakest of the three and should be labelled as such in the readout, because it makes the conclusion a statement about your budget as much as about your product. It is still far better than no bound.\n\nWrite the bound into the brief before fielding. A SESOI chosen after seeing the data is not a bound, it is a rationalisation, and it belongs in the same category as the analytic flexibility described in our guide to [p-hacking and researcher degrees of freedom](/docs/p-hacking-researcher-degrees-of-freedom).\n\n## Sample size: the uncomfortable part\n\nEquivalence testing needs more data than difference testing, not less, and it is worth saying so plainly rather than discovering it mid-analysis. Proving a difference requires the interval to exclude zero. Proving equivalence requires the interval to sit entirely inside a narrow band around zero - a strictly harder geometric condition, and it gets harder as the band narrows.\n\nThe practical consequence is a trade-off that should be made in the open: a tighter, more useful equivalence claim costs more sample. A team that declares a 0.1-point bound on a five-point scale and fields 80 people has not run a rigorous study, it has guaranteed an inconclusive one.\n\nTwo things make this tractable. The first is being honest that some equivalence questions are not worth the sample they require, and should be resolved by judgment rather than a study that will not settle them. The second is reducing what a large sample costs, which is where the last twenty years of tooling has actually moved.\n\n## How Koji makes equivalence claims affordable\n\n**Volume stops being the constraint.** Equivalence testing is a sample-size problem before it is a statistics problem, and the reason product teams do not run these tests is not ignorance - it is that recruiting and moderating 400 conversations was never realistic on a two-week decision cycle. AI-moderated interviews and voice studies change the arithmetic: where a manual study of 25 interviews runs to tens of thousands of dollars, an AI-moderated conversation costs on the order of $20, and teams using AI-assisted research report 60 to 80 percent faster time-to-insight. The sample size that makes an equivalence claim meaningful becomes an ordinary study rather than a quarterly programme.\n\n**The six structured question types give you bounded, comparable measures.** An equivalence bound only means something on a stable instrument. Koji studies are built from six question types - open_ended, scale, single_choice, multiple_choice, ranking and yes_no - described in our [structured questions guide](/docs/structured-questions-guide). Scale items give you the interval arithmetic TOST needs; single_choice and yes_no give you clean proportions; ranking sidesteps the saturation problem covered in [ceiling and floor effects](/docs/ceiling-floor-effects-research), which is worth knowing because a ceiling manufactures false equivalence - two groups pinned at the maximum will always look identical.\n\n**The brief is where the bound lives.** Declaring the SESOI in the study brief before fielding is the whole discipline, and it is easiest to sustain when the brief is a real artifact rather than a document someone was supposed to write. Legacy survey tools like SurveyMonkey or Google Forms have no concept of a pre-declared analysis bound; the question set is the entire specification, so nothing stops the interpretation being invented afterwards.\n\n**Both sides of the comparison, same instrument, same time.** Equivalence between two variants is only interpretable if the two arms were measured identically. Running both arms as one Koji study with a shared question set removes the instrument drift that quietly invalidates most internal parity claims.\n\n**Qualitative equivalence has its own answer.** Not every no-difference question is numeric. \"Do enterprise and mid-market users describe the problem the same way?\" is an equivalence question about meaning, and the answer comes from comparing theme distributions across the two groups in a common vocabulary - which is what automatic thematic analysis over a shared instrument produces. See [thematic analysis](/docs/thematic-analysis-guide) for the underlying method.\n\n## What to write in the readout\n\nReplace the sentence \"there was no significant difference\" with one of these four, and the quality of the resulting decision changes immediately.\n\n- \"The difference was smaller than our pre-declared bound of X. We can treat these as equivalent for this decision.\"\n- \"The difference was statistically detectable but smaller than X, so it is real and not worth acting on.\"\n- \"The difference exceeded X and was statistically detectable. Act on it.\"\n- \"The confidence interval spanned X. This study could not distinguish a meaningful difference from no difference, and the decision should not rest on it.\"\n\nThe fourth is the one that requires organisational courage, and it is also the one that would have prevented most of the failures in the Freiman sample. Reporting inconclusiveness accurately is cheap. Discovering it two quarters later, after the migration, is not.\n\n## Frequently asked questions\n\n### What is the difference between equivalence testing and non-inferiority testing?\nEquivalence testing uses two bounds and asks whether the difference is small in both directions. Non-inferiority uses one bound and asks only whether the new option is not meaningfully worse than the old one - it permits the new option to be arbitrarily better. Non-inferiority is usually the right frame for a migration or a cost-reduction swap, where being better is welcome and only being worse matters. Equivalence is the right frame when a difference in either direction would change the decision.\n\n### Can I just look at the confidence interval instead of running TOST?\nLargely, yes, and Altman and Bland recommended exactly that. If the 90 percent confidence interval around your observed difference falls entirely inside your equivalence bounds, you have the same conclusion TOST would give. Running the formal test adds a p-value for the equivalence claim and forces you to state the bound explicitly, which is its main practical value.\n\n### Why a 90 percent confidence interval rather than 95 percent?\nBecause TOST is two one-sided tests, each run at 5 percent, and the interval that corresponds to that pair of one-sided tests at 5 percent is the 90 percent interval rather than the 95 percent one. Using a 95 percent interval makes the test more conservative than the stated alpha, which is a defensible choice but should be described as such.\n\n### How do I choose the smallest effect size of interest for a qualitative study?\nThe same way, but on a proportion rather than a mean. Decide in advance what share of participants would have to raise a theme for it to be a real difference between groups - for example, that a theme appearing in 20 percent of one group and 25 percent of another is not a meaningful gap. The bound is on the theme prevalence difference, and it needs declaring before coding starts.\n\n### Does an equivalence test fix an underpowered study after the fact?\nNo. It correctly reports that the study was underpowered instead of letting it masquerade as a null finding. That is the value: it converts a misleading conclusion into an accurate one. Fixing the underpowering requires more data, which is a design decision made before fielding - see [statistical power and minimum detectable effect](/docs/statistical-power-minimum-detectable-effect).\n\n### If most of our nulls turn out to be inconclusive, what should we do with them?\nRecord them and pool them. An individually inconclusive null is not worthless; three inconclusive nulls pointing the same direction on the same question are real evidence. This only works if they were all recorded, which is the subject of [publication bias in product research](/docs/publication-bias-product-research), and if they are combined deliberately rather than remembered selectively, which is the subject of [evidence synthesis](/docs/evidence-synthesis-research-findings).\n\n## Related Resources\n\n- [Statistical Power and Minimum Detectable Effect](/docs/statistical-power-minimum-detectable-effect) - what your study can detect before you field it\n- [Statistical Significance in Survey Research](/docs/statistical-significance-survey-research) - what a p-value does and does not say\n- [Publication Bias in Product Research](/docs/publication-bias-product-research) - why null results disappear from your evidence base\n- [Evidence Synthesis](/docs/evidence-synthesis-research-findings) - pooling inconclusive studies into a usable conclusion\n- [Structured Questions Guide](/docs/structured-questions-guide) - the six question types and when to use each\n- [Ceiling and Floor Effects](/docs/ceiling-floor-effects-research) - how a saturated scale manufactures false equivalence\n- [The Multiple Comparisons Problem](/docs/multiple-comparisons-problem) - why thin segment cells mislead in both directions\n- [Survey Sample Size](/docs/survey-sample-size-guide) - how many responses a claim of this kind requires","category":"Research Methods","lastModified":"2026-08-09T03:20:57.935079+00:00","metaTitle":"Equivalence Testing: How to Prove There Is No Difference","metaDescription":"A non-significant result usually means your study could not tell. Learn TOST equivalence testing, how to set a smallest effect size of interest, and how to stop making sunset and migration decisions on false nulls.","keywords":["equivalence testing","TOST","two one-sided tests","absence of evidence is not evidence of absence","how to prove no difference","smallest effect size of interest","SESOI","non-inferiority testing","false negative","null result interpretation"],"aiSummary":"Standard significance testing cannot conclude that two things are the same; it can only fail to detect a difference. Equivalence testing, usually implemented as TOST (two one-sided tests), asks whether the observed difference falls entirely inside a pre-declared smallest effect size of interest. Freiman et al. (1978) found 67 of 71 trials reported as negative had over a 10 percent risk of missing a true 25 percent improvement. Running the difference test and the equivalence test together produces four outcomes rather than two, including the commonly mislabelled inconclusive case. FDA average bioequivalence applies the same logic with an 80 to 125 percent bound on a 90 percent confidence interval.","aiPrerequisites":["Familiarity with p-values, confidence intervals and significance testing","Experience interpreting A/B or survey comparison readouts"],"aiLearningOutcomes":["Explain why a non-significant result cannot establish equivalence","Set a smallest effect size of interest anchored to a decision, a benchmark or available power","Run and interpret a TOST equivalence test using the 90 percent confidence interval","Classify a result into the four cells of the significance-by-equivalence matrix","Recognise which common product decisions are equivalence claims in disguise","Report an inconclusive study accurately instead of as a null finding"],"aiDifficulty":"intermediate","aiEstimatedTime":"15 min"}],"pagination":{"total":1,"returned":1,"offset":0}}