{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-11-21T23:11:48.587Z"},"content":[{"type":"documentation","id":"48cd027a-87cb-4790-823f-f5d4d118c842","slug":"classifier-precision-base-rate-research","title":"The Base Rate Nobody Measured: Why Every Flag in Your Research Stack Has an Unknown Precision (2026)","url":"https://www.koji.so/docs/classifier-precision-base-rate-research","summary":"Precision is a joint property of a classifier and the population it runs over, so a flag with fixed recall and specificity halves in precision when the base rate falls from 6% to 2% while its accuracy rises from 94.1% to 94.7%. Saito and Rehmsmeier showed ROC curves and AUC are unchanged by that shift while precision moves from 75.8% to 23.8%. The fix is two random audits, one of the flagged items and one of the unflagged pile, reported with the prevalence attached.","content":"Every automated flag in your research stack — the AI theme tag, the sentiment label, the churn-risk score, the \"this is a feature request\" classifier, the quality gate, the fraud detector — is a diagnostic test. Its usefulness is governed by the prevalence of the thing it flags, and almost nobody measures that prevalence. Which means the number you care about, precision, is not merely unknown: it is uncomputable from anything on your dashboard. Worse, the metric your dashboard does show — accuracy — moves in the *opposite* direction when prevalence falls. Below, a tagger holds its recall and specificity constant while the rare class gets rarer, and its accuracy improves from 94.1% to 94.7% while its precision collapses from 50.5% to 24.6%.\n\nThis is the general form of the screening problem covered in [screener accuracy](/docs/screener-accuracy-positive-predictive-value), and it applies to every flag you did not personally validate.\n\n## The one thing a flag cannot tell you\n\nA classifier has two properties you can measure without knowing anything about the world: **recall** (of the items that really belong to the class, what share does it flag?) and **specificity** (of the items that do not, what share does it correctly leave alone?). Both are estimated by testing the flag against labelled examples.\n\nWhat you actually want to know runs the other way: **given that this item is flagged, what is the probability it belongs to the class?** That is precision, and it depends on a third quantity that is not a property of the classifier at all — the base rate of the class in the stream you are running it over.\n\nWork it through on 10,000 coded interview responses where 6% are genuinely feature requests, using a tagger with 80% recall and 95% specificity:\n\n| | Really a feature request (600) | Not (9,400) | Total |\n|---|---|---|---|\n| **Tagged** | 480 | 470 | 950 |\n| **Not tagged** | 120 | 8,930 | 9,050 |\n\nPrecision is 480 / 950 = **50.5%**. Half of the items in your \"feature requests\" bucket are not feature requests, from a tagger that catches four out of five real ones and correctly ignores 95% of everything else.\n\nNow change nothing about the tagger and run it on a stream where feature requests are 2% instead of 6% — a different product area, a different quarter, a support channel instead of an interview corpus:\n\n| | Really a feature request (200) | Not (9,800) | Total |\n|---|---|---|---|\n| **Tagged** | 160 | 490 | 650 |\n| **Not tagged** | 40 | 9,310 | 9,350 |\n\nPrecision is 160 / 650 = **24.6%**. The tag became twice as wrong, and nothing about the model changed. The only thing that changed was the population you pointed it at.\n\n![Column chart of one tagger at a 6% and a 2% base rate: precision falls from 50.5% to 24.6% while accuracy rises from 94.1% to 94.7%.](/visuals/classifier-precision-vs-accuracy-by-base-rate/classifier-precision-vs-accuracy-by-base-rate.png \"Accuracy improved while the tag became twice as wrong, with nothing about the model changed · Source: Koji worked example, 10,000 coded responses, 80% recall, 95% specificity\")\n\n## Accuracy is the metric that hides this\n\nHere is the part that makes the failure durable rather than merely common. Compute accuracy for both tables above:\n\n| Base rate of the class | Precision | Accuracy |\n|---|---|---|\n| 6% | 50.5% | 94.1% |\n| 2% | 24.6% | 94.7% |\n\n**Accuracy went up while precision halved.** This is not a paradox or a trick; it is arithmetic. As the positive class gets rarer, the huge and easy negative class dominates the accuracy calculation, so a classifier looks better precisely as its output becomes less trustworthy. Push it to the limit and the point is obvious: on a stream where 4% of items belong to the class, a classifier that flags absolutely nothing scores 96% accuracy.\n\nThe same blind spot afflicts the two other metrics teams reach for. Saito and Rehmsmeier demonstrated it directly for ROC curves and AUC, in a paper whose abstract states that \"the visual interpretability of ROC plots in the context of imbalanced datasets can be deceptive with respect to conclusions about the reliability of classification performance, owing to an intuitive but wrong interpretation of specificity\" (Saito T, Rehmsmeier M. *PLOS ONE* 2015;10(3):e0118432).\n\nTheir simulation makes it concrete. They generated a balanced sample of 1,000 positives and 1,000 negatives, and an imbalanced sample of 1,000 positives and 10,000 negatives. At one operating point the classifier produced 500 true positives and 160 false positives in the balanced case, and 500 true positives and 1,600 false positives in the imbalanced case. The ROC curves are identical — same true positive rate of 0.5, same false positive rate of 0.16 — and, as the authors report, all AUC scores are unchanged. Precision, meanwhile, went from 75.8% to 23.8%.\n\n![Table comparing a balanced and an imbalanced sample at one ROC point: the same true and false positive rates, but precision falls from 75.8% to 23.8%.](/visuals/same-roc-point-different-precision-saito-rehmsmeier/same-roc-point-different-precision-saito-rehmsmeier.png \"The ROC curves are identical. Only precision shows what changed · Source: Saito and Rehmsmeier, PLOS ONE, 2015\")\n\nThe authors' reason for preferring precision-recall plots is the whole point of this article: they \"evaluate the fraction of true positives among positive predictions.\" That fraction is the only one your team will act on, and it is the only one that moves when the base rate moves.\n\n## Why nobody notices\n\nThree ordinary practices conspire to keep the base rate invisible.\n\n**You only ever look at the flagged items.** A researcher opens the \"churn risk\" list, the \"confused about pricing\" theme, the flagged-for-review queue. Those are the true positives and false positives together. The false negatives are, by construction, in the pile nobody opens. You can spot a wrong item in a list; you cannot spot a missing one.\n\n**The vendor's numbers were measured on a different population.** Every published accuracy figure — for a sentiment model, a tagger, a fraud score — was measured on some evaluation set with some class distribution. That distribution is not yours. A model reported at 92% precision on a benchmark where the class runs at 30% will not deliver 92% on your corpus where it runs at 3%, and the gap is not a defect in the model.\n\n**The ground truth is itself wrong.** Even careful, expensive labelling carries error, so measured precision is an estimate against a noisy standard. The canonical study of this found that when confident-learning algorithms flagged candidate label errors in ten widely used benchmarks and humans checked them, only 51% of the algorithmically-flagged candidates turned out to be genuinely mislabelled (Northcutt C, Athalye A, Mueller J, NeurIPS 2021, arXiv:2103.14749) — a 51% precision, for a flag, in the most carefully studied labelling context that exists. The consequences for building evaluation sets are covered in [evaluation datasets for AI products](/docs/ai-evaluation-dataset-golden-set).\n\n## How to actually get the number\n\nYou need two measurements, and only one of them is the obvious one.\n\n![A researcher going through printed pages with a highlighter, checking a random sample of tagged items by hand](https://images.unsplash.com/photo-1559167299-f5ab522753e4?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3w5NDUzMDN8MHwxfHNlYXJjaHw1fHxwZXJzb24lMjBoaWdobGlnaHRpbmclMjBwcmludGVkJTIwcGFnZXMlMjBkZXNrfGVufDF8MHx8fDE3OTQ0MTIyMTJ8MA&ixlib=rb-4.1.0&q=80&w=1080)\n\n_Photo by [Konstantinos Feggoulis](https://unsplash.com/@kostasfeg?utm_source=koji&utm_medium=referral) on [Unsplash](https://unsplash.com/?utm_source=koji&utm_medium=referral)_\n\n**Measurement 1: audit a random sample of the flagged items.** Pull 50 items at random from what the flag caught, have a human classify each one against a written definition, and the share that are genuine is your measured precision, with a confidence interval you can compute. Most teams do a version of this informally. It is necessary and it is not sufficient, because it tells you nothing about the base rate or about what you missed.\n\n**Measurement 2: audit a random sample of the *unflagged* items.** This is the step almost everyone skips, and it is the one that unlocks everything else. Pull 100 items at random from the pile the flag ignored and classify them the same way. The share of genuine cases in that sample estimates your false negative rate; combined with measurement 1, it gives you recall and the base rate itself.\n\nWith a 6% base rate and a flag that catches four in five, a 100-item audit of the unflagged pile should turn up roughly one missed case. That is a small number, which is exactly why the audit has to be a deliberate, scheduled, random draw rather than something you do when you happen to feel suspicious — a suspicion-triggered review of the unflagged pile is not a random sample, and it will systematically over-find in the areas you already worried about.\n\n**Then attach the prevalence to the number.** Report precision as \"51% precision at a 6% base rate, measured on 2026 Q2 interview transcripts\", not \"51% precision\". The second version will be quoted six months later against a corpus with a different composition, and it will be wrong in a way nobody can trace.\n\n## The precision footer\n\nThe practical output of all this is a short block that travels with any report built on automated tags. Six lines:\n\n1. **What the flag claims to identify**, in one sentence with a written inclusion rule.\n2. **Base rate**, estimated from a random audit, with the sample size and date.\n3. **Measured precision**, from the flagged audit, with the sample size.\n4. **Measured recall**, from the unflagged audit, with the sample size.\n5. **The population it was measured on**, named specifically enough that a reader can tell whether their question is about the same one.\n6. **When it was last re-measured.** Base rates drift as your product, your channels and your customer mix change, and a precision figure ages faster than anyone expects.\n\n![Checklist of six lines to add to any report built on automated tags, from what the flag identifies to when it was last re-measured.](/visuals/precision-footer-automated-tags-checklist/precision-footer-automated-tags-checklist.png \"Six lines that let a reader judge whether a tag count means anything\")\n\nA report that carries this footer can be argued with. A report that says \"the AI found 340 pricing complaints\" cannot, because there is no way for a reader to know whether that is 340 of a real 400 or 340 of a real 900 with 200 of the 340 misfiled.\n\n## Where the threshold comes from\n\nEvery flag has a threshold, explicit or implicit, and it encodes a trade you probably never made deliberately. Moving it toward recall finds more real cases and lowers precision; moving it toward precision does the reverse. There is no neutral setting.\n\nThe right way to choose is to price the two errors in the decision the flag feeds:\n\n- **When a flag triggers expensive human work** — routing a session for manual review, escalating an account to a CSM — a false positive costs real hours, so precision dominates and a high threshold is correct.\n- **When a flag feeds a count that a decision rests on** — \"37% of interviews mentioned onboarding friction\" — both errors distort the number and, critically, they do not cancel. Under-detection understates the theme, over-detection inflates it, and the net is whichever is larger at your base rate.\n- **When a flag is a safety net** — catching harmful model outputs, spotting a compliance issue — a missed case costs far more than a wasted review, so recall dominates and you accept a low precision on purpose. State that you did, so nobody reads the flagged list as an inventory of real problems.\n\nAny threshold defended as \"the default\" is a decision that got made by whoever wrote the library.\n\n## How this differs from spot-checking\n\n[The all-or-none rule for research QA](/docs/research-quality-inspection-sampling) answers a different question: given a known defect rate and known costs, should you inspect every item or none of them? That is the economics of inspection. This article is about a prior question — whether you know the defect rate at all, and whether the metric you are reading can tell you. You need this measurement before that decision has any inputs.\n\nIt is also distinct from [the base rate for a product bet](/docs/product-idea-base-rate-outside-view), which is about reference classes for forecasting how often a feature succeeds. Same phrase, different object: there the base rate is a prior on an outcome, here it is the prevalence of a class in a stream.\n\n## The modern approach: flags you can audit\n\nMost tooling makes measurement 2 impossible. If your feedback tool shows you a themed dashboard but not the underlying corpus, you cannot draw a random sample of what it ignored, so you cannot ever estimate recall or the base rate — you are structurally confined to admiring the flagged pile.\n\nKoji is built so both audits are cheap:\n\n- **Every tag traces to a transcript.** Themes and tags in Koji resolve to the verbatim exchanges that produced them, so auditing a flagged item takes seconds rather than a data request. See [understanding themes and patterns](/docs/understanding-themes-patterns) and [AI auto-tagging for customer interviews](/docs/ai-auto-tagging-customer-interviews) for how the tags are produced.\n- **The unflagged pile is right there.** Because the full corpus of interviews stays in the platform rather than being reduced to a dashboard, drawing a random sample of untagged responses for the second audit is a normal operation, not an export project.\n- **Structured questions remove entire classes of inference.** The six question types — open_ended, scale, single_choice, multiple_choice, ranking and yes_no — mean that anything you can ask directly does not need to be inferred by a classifier at all. A yes_no or single_choice answer has a base rate you can count exactly rather than estimate, and a scale item gives you a distribution instead of a flag. The strongest way to improve a classifier's precision is to stop needing it for that variable. The [structured questions guide](/docs/structured-questions-guide) covers when each type replaces an inference.\n- **Quality scores are stated, not hidden.** Koji's 1-5 interview quality score is visible per session, so the flag that gates your data is one you can audit against your own reading of the transcripts rather than a black box with a reported accuracy.\n\n![Koji report summary with a side panel of supporting participant quotes behind one finding](/email/shots/shot-evidence-panel.jpg \"Each finding opens the interview quotes behind it, so a flagged item can be checked in seconds\")\n\nThe pattern generalises beyond tooling. The demand you cannot see is a recurring failure in research — nobody experienced the average, no respondent knows the answer, [nobody counted the one-offs](/docs/singleton-themes-unseen-coverage). This one is the same shape: nobody measured the base rate, so every downstream number inherits an error with no upper bound and no alarm attached.\n\n## Frequently asked questions\n\n### What is the difference between precision and accuracy in plain terms?\n\nPrecision answers \"of the things the flag caught, how many are real?\" Accuracy answers \"of everything, how often did the flag get it right?\" When the class you are flagging is rare, almost everything is a negative, so a classifier scores well on accuracy simply by leaving the big easy pile alone. That makes accuracy nearly useless for rare classes and makes it dangerous, because it moves in the wrong direction: as the class gets rarer, accuracy rises while precision falls. If someone quotes an accuracy figure for a flag over a rare class, they have not told you anything you can act on.\n\n### How large should my audit samples be?\n\nFifty flagged items and 100 unflagged items is a serviceable starting point for a monthly cadence. Fifty flagged items pins precision to roughly plus or minus 14 points at 50% precision, which is enough to tell \"about half\" from \"about nine in ten\" — the distinction that changes what you do. The unflagged sample needs to be larger because misses are rare, and if it turns up zero missed cases that is still informative: it bounds your miss rate rather than proving it is zero.\n\n### The vendor published precision and recall for their model. Is that enough?\n\nNo, because those numbers were measured on their evaluation set, not on your corpus, and precision is a joint property of the classifier and the population. A model reported at 90% precision on a benchmark where the class runs at 30% can easily deliver 40% on your data where it runs at 3%, with no defect in the model. Recall and specificity travel across populations reasonably well; precision does not travel at all. Treat vendor precision as an upper bound measured under favourable conditions and measure your own.\n\n### How often do I need to re-measure?\n\nWhenever the population changes, which is more often than it feels. A new acquisition channel, a pricing change, a seasonal spike, a new market or a model version update all shift either the base rate or the classifier's behaviour. A quarterly re-audit is a reasonable default for a stable product, and any of those events should trigger one immediately. The re-measurement is also the only way to detect drift, since a drifting classifier looks perfectly healthy on accuracy right up until someone checks the flagged pile.\n\n### Does this apply to human coders as well as AI?\n\nEntirely. A human coder applying a codebook is a classifier with a recall and a specificity, and their precision depends on the same base rate. The difference is that human coders can be asked why they made a call, and their errors tend to be systematic and correctable through a written codebook rather than opaque. Both benefit from the same two audits. Agreement metrics between coders are a related but different measurement — they tell you about consistency, not correctness, and two coders can agree perfectly while both being wrong about a rare class.\n\n### If precision is unknowable without an audit, should I stop using automated flags?\n\nNo — the alternative is not \"no flags\", it is manual review of everything, which has its own error rate and does not scale past a few hundred items. Automated tagging over a large corpus at 50% precision with a known base rate is far more useful than reading 30 transcripts by hand and generalising, because you can correct a known bias and you cannot correct an unknown sample. The failure mode is not using the flags; it is quoting their output as a count without ever having measured what the count means.\n\n## Related Resources\n\n- [Screener Accuracy: Why Most People Who Pass Your Screener Are Not Who You Wanted](/docs/screener-accuracy-positive-predictive-value) — the same arithmetic on a recruiting screener\n- [Tightening Your Screener Makes the Sample Purer and the Findings Worse](/docs/screener-tightening-false-negatives) — what happens when you move a threshold without pricing both errors\n- [You Cannot Spot-Check Your Way to Data Quality](/docs/research-quality-inspection-sampling) — the inspect-or-not decision this measurement feeds\n- [Structured Questions Guide](/docs/structured-questions-guide) — the six question types, and how asking directly removes the need to classify\n- [Singleton Themes: Why One-Off Comments Are the Only Estimate You Have of What You Missed](/docs/singleton-themes-unseen-coverage) — the sibling failure in thematic coverage\n- [AI Auto-Tagging for Customer Interviews](/docs/ai-auto-tagging-customer-interviews) — how the tags whose precision you are auditing get produced\n- [When the Human Baseline Is Wrong](/docs/imperfect-gold-standard-ai-analysis-validation) — what happens when the reference you audit against is itself wrong\n- [Why Complaint Counts Cannot Become Rates](/docs/complaint-counts-cannot-be-rates) — the missing-denominator problem in your inbound feedback channel\n","category":"Analysis & Synthesis","lastModified":"2026-11-11T15:50:12.221562+00:00","metaTitle":"Classifier Precision and Base Rate in Research (2026)","metaDescription":"Why precision is uncomputable without prevalence, why accuracy rises as precision falls, and the two audits that give you a number you can quote.","keywords":["classifier precision","base rate","class imbalance precision","AI tag accuracy","precision vs accuracy","prevalence estimate audit","false positive rate research"],"aiSummary":"Precision is a joint property of a classifier and the population it runs over, so a flag with fixed recall and specificity halves in precision when the base rate falls from 6% to 2% while its accuracy rises from 94.1% to 94.7%. Saito and Rehmsmeier showed ROC curves and AUC are unchanged by that shift while precision moves from 75.8% to 23.8%. The fix is two random audits, one of the flagged items and one of the unflagged pile, reported with the prevalence attached.","aiPrerequisites":["An automated tag, score or flag already running over research data","Basic familiarity with precision and recall"],"aiLearningOutcomes":["Explain why precision depends on prevalence and accuracy does not reveal it","Run the flagged and unflagged audits that estimate precision, recall and base rate","Publish a precision footer that survives being quoted out of context","Choose a flag threshold by pricing both errors in the decision it feeds"],"aiDifficulty":"advanced","aiEstimatedTime":"13 min"}],"pagination":{"total":1,"returned":1,"offset":0}}