The Base Rate Nobody Measured: Why Every Flag in Your Research Stack Has an Unknown Precision (2026)

Every automated flag in your research stack — the AI theme tag, the sentiment label, the churn-risk score, the "this is a feature request" classifier, the quality gate, the fraud detector — is a diagnostic test. Its usefulness is governed by the prevalence of the thing it flags, and almost nobody measures that prevalence. Which means the number you care about, precision, is not merely unknown: it is uncomputable from anything on your dashboard. Worse, the metric your dashboard does show — accuracy — moves in the opposite direction when prevalence falls. Below, a tagger holds its recall and specificity constant while the rare class gets rarer, and its accuracy improves from 94.1% to 94.7% while its precision collapses from 50.5% to 24.6%.

This is the general form of the screening problem covered in screener accuracy, and it applies to every flag you did not personally validate.

The one thing a flag cannot tell you

A classifier has two properties you can measure without knowing anything about the world: recall (of the items that really belong to the class, what share does it flag?) and specificity (of the items that do not, what share does it correctly leave alone?). Both are estimated by testing the flag against labelled examples.

What you actually want to know runs the other way: given that this item is flagged, what is the probability it belongs to the class? That is precision, and it depends on a third quantity that is not a property of the classifier at all — the base rate of the class in the stream you are running it over.

Work it through on 10,000 coded interview responses where 6% are genuinely feature requests, using a tagger with 80% recall and 95% specificity:

Precision is 480 / 950 = 50.5%. Half of the items in your "feature requests" bucket are not feature requests, from a tagger that catches four out of five real ones and correctly ignores 95% of everything else.

Now change nothing about the tagger and run it on a stream where feature requests are 2% instead of 6% — a different product area, a different quarter, a support channel instead of an interview corpus:

Precision is 160 / 650 = 24.6%. The tag became twice as wrong, and nothing about the model changed. The only thing that changed was the population you pointed it at.

Accuracy is the metric that hides this

Here is the part that makes the failure durable rather than merely common. Compute accuracy for both tables above:

Accuracy went up while precision halved. This is not a paradox or a trick; it is arithmetic. As the positive class gets rarer, the huge and easy negative class dominates the accuracy calculation, so a classifier looks better precisely as its output becomes less trustworthy. Push it to the limit and the point is obvious: on a stream where 4% of items belong to the class, a classifier that flags absolutely nothing scores 96% accuracy.

The same blind spot afflicts the two other metrics teams reach for. Saito and Rehmsmeier demonstrated it directly for ROC curves and AUC, in a paper whose abstract states that "the visual interpretability of ROC plots in the context of imbalanced datasets can be deceptive with respect to conclusions about the reliability of classification performance, owing to an intuitive but wrong interpretation of specificity" (Saito T, Rehmsmeier M. PLOS ONE 2015;10(3):e0118432).

Their simulation makes it concrete. They generated a balanced sample of 1,000 positives and 1,000 negatives, and an imbalanced sample of 1,000 positives and 10,000 negatives. At one operating point the classifier produced 500 true positives and 160 false positives in the balanced case, and 500 true positives and 1,600 false positives in the imbalanced case. The ROC curves are identical — same true positive rate of 0.5, same false positive rate of 0.16 — and, as the authors report, all AUC scores are unchanged. Precision, meanwhile, went from 75.8% to 23.8%.

The authors' reason for preferring precision-recall plots is the whole point of this article: they "evaluate the fraction of true positives among positive predictions." That fraction is the only one your team will act on, and it is the only one that moves when the base rate moves.

Why nobody notices

Three ordinary practices conspire to keep the base rate invisible.

You only ever look at the flagged items. A researcher opens the "churn risk" list, the "confused about pricing" theme, the flagged-for-review queue. Those are the true positives and false positives together. The false negatives are, by construction, in the pile nobody opens. You can spot a wrong item in a list; you cannot spot a missing one.

The vendor's numbers were measured on a different population. Every published accuracy figure — for a sentiment model, a tagger, a fraud score — was measured on some evaluation set with some class distribution. That distribution is not yours. A model reported at 92% precision on a benchmark where the class runs at 30% will not deliver 92% on your corpus where it runs at 3%, and the gap is not a defect in the model.

The ground truth is itself wrong. Even careful, expensive labelling carries error, so measured precision is an estimate against a noisy standard. The canonical study of this found that when confident-learning algorithms flagged candidate label errors in ten widely used benchmarks and humans checked them, only 51% of the algorithmically-flagged candidates turned out to be genuinely mislabelled (Northcutt C, Athalye A, Mueller J, NeurIPS 2021, arXiv:2103.14749) — a 51% precision, for a flag, in the most carefully studied labelling context that exists. The consequences for building evaluation sets are covered in evaluation datasets for AI products.

How to actually get the number

You need two measurements, and only one of them is the obvious one.

Measurement 1: audit a random sample of the flagged items. Pull 50 items at random from what the flag caught, have a human classify each one against a written definition, and the share that are genuine is your measured precision, with a confidence interval you can compute. Most teams do a version of this informally. It is necessary and it is not sufficient, because it tells you nothing about the base rate or about what you missed.

Measurement 2: audit a random sample of the unflagged items. This is the step almost everyone skips, and it is the one that unlocks everything else. Pull 100 items at random from the pile the flag ignored and classify them the same way. The share of genuine cases in that sample estimates your false negative rate; combined with measurement 1, it gives you recall and the base rate itself.

With a 6% base rate and a flag that catches four in five, a 100-item audit of the unflagged pile should turn up roughly one missed case. That is a small number, which is exactly why the audit has to be a deliberate, scheduled, random draw rather than something you do when you happen to feel suspicious — a suspicion-triggered review of the unflagged pile is not a random sample, and it will systematically over-find in the areas you already worried about.

Then attach the prevalence to the number. Report precision as "51% precision at a 6% base rate, measured on 2026 Q2 interview transcripts", not "51% precision". The second version will be quoted six months later against a corpus with a different composition, and it will be wrong in a way nobody can trace.

The precision footer

The practical output of all this is a short block that travels with any report built on automated tags. Six lines:

1. What the flag claims to identify, in one sentence with a written inclusion rule.
2. Base rate, estimated from a random audit, with the sample size and date.
3. Measured precision, from the flagged audit, with the sample size.
4. Measured recall, from the unflagged audit, with the sample size.
5. The population it was measured on, named specifically enough that a reader can tell whether their question is about the same one.
6. When it was last re-measured. Base rates drift as your product, your channels and your customer mix change, and a precision figure ages faster than anyone expects.

A report that carries this footer can be argued with. A report that says "the AI found 340 pricing complaints" cannot, because there is no way for a reader to know whether that is 340 of a real 400 or 340 of a real 900 with 200 of the 340 misfiled.

Where the threshold comes from

Every flag has a threshold, explicit or implicit, and it encodes a trade you probably never made deliberately. Moving it toward recall finds more real cases and lowers precision; moving it toward precision does the reverse. There is no neutral setting.

The right way to choose is to price the two errors in the decision the flag feeds:

• When a flag triggers expensive human work — routing a session for manual review, escalating an account to a CSM — a false positive costs real hours, so precision dominates and a high threshold is correct.
• When a flag feeds a count that a decision rests on — "37% of interviews mentioned onboarding friction" — both errors distort the number and, critically, they do not cancel. Under-detection understates the theme, over-detection inflates it, and the net is whichever is larger at your base rate.
• When a flag is a safety net — catching harmful model outputs, spotting a compliance issue — a missed case costs far more than a wasted review, so recall dominates and you accept a low precision on purpose. State that you did, so nobody reads the flagged list as an inventory of real problems.

Any threshold defended as "the default" is a decision that got made by whoever wrote the library.

How this differs from spot-checking

The all-or-none rule for research QA answers a different question: given a known defect rate and known costs, should you inspect every item or none of them? That is the economics of inspection. This article is about a prior question — whether you know the defect rate at all, and whether the metric you are reading can tell you. You need this measurement before that decision has any inputs.

It is also distinct from the base rate for a product bet, which is about reference classes for forecasting how often a feature succeeds. Same phrase, different object: there the base rate is a prior on an outcome, here it is the prevalence of a class in a stream.

The modern approach: flags you can audit

Most tooling makes measurement 2 impossible. If your feedback tool shows you a themed dashboard but not the underlying corpus, you cannot draw a random sample of what it ignored, so you cannot ever estimate recall or the base rate — you are structurally confined to admiring the flagged pile.

Koji is built so both audits are cheap:

• Every tag traces to a transcript. Themes and tags in Koji resolve to the verbatim exchanges that produced them, so auditing a flagged item takes seconds rather than a data request. See understanding themes and patterns and AI auto-tagging for customer interviews for how the tags are produced.
• The unflagged pile is right there. Because the full corpus of interviews stays in the platform rather than being reduced to a dashboard, drawing a random sample of untagged responses for the second audit is a normal operation, not an export project.
• Structured questions remove entire classes of inference. The six question types — open_ended, scale, single_choice, multiple_choice, ranking and yes_no — mean that anything you can ask directly does not need to be inferred by a classifier at all. A yes_no or single_choice answer has a base rate you can count exactly rather than estimate, and a scale item gives you a distribution instead of a flag. The strongest way to improve a classifier's precision is to stop needing it for that variable. The structured questions guide covers when each type replaces an inference.
• Quality scores are stated, not hidden. Koji's 1-5 interview quality score is visible per session, so the flag that gates your data is one you can audit against your own reading of the transcripts rather than a black box with a reported accuracy.

The pattern generalises beyond tooling. The demand you cannot see is a recurring failure in research — nobody experienced the average, no respondent knows the answer, nobody counted the one-offs. This one is the same shape: nobody measured the base rate, so every downstream number inherits an error with no upper bound and no alarm attached.

Frequently asked questions

What is the difference between precision and accuracy in plain terms?

Precision answers "of the things the flag caught, how many are real?" Accuracy answers "of everything, how often did the flag get it right?" When the class you are flagging is rare, almost everything is a negative, so a classifier scores well on accuracy simply by leaving the big easy pile alone. That makes accuracy nearly useless for rare classes and makes it dangerous, because it moves in the wrong direction: as the class gets rarer, accuracy rises while precision falls. If someone quotes an accuracy figure for a flag over a rare class, they have not told you anything you can act on.

How large should my audit samples be?

Fifty flagged items and 100 unflagged items is a serviceable starting point for a monthly cadence. Fifty flagged items pins precision to roughly plus or minus 14 points at 50% precision, which is enough to tell "about half" from "about nine in ten" — the distinction that changes what you do. The unflagged sample needs to be larger because misses are rare, and if it turns up zero missed cases that is still informative: it bounds your miss rate rather than proving it is zero.

The vendor published precision and recall for their model. Is that enough?

No, because those numbers were measured on their evaluation set, not on your corpus, and precision is a joint property of the classifier and the population. A model reported at 90% precision on a benchmark where the class runs at 30% can easily deliver 40% on your data where it runs at 3%, with no defect in the model. Recall and specificity travel across populations reasonably well; precision does not travel at all. Treat vendor precision as an upper bound measured under favourable conditions and measure your own.

How often do I need to re-measure?

Whenever the population changes, which is more often than it feels. A new acquisition channel, a pricing change, a seasonal spike, a new market or a model version update all shift either the base rate or the classifier's behaviour. A quarterly re-audit is a reasonable default for a stable product, and any of those events should trigger one immediately. The re-measurement is also the only way to detect drift, since a drifting classifier looks perfectly healthy on accuracy right up until someone checks the flagged pile.

Does this apply to human coders as well as AI?

Entirely. A human coder applying a codebook is a classifier with a recall and a specificity, and their precision depends on the same base rate. The difference is that human coders can be asked why they made a call, and their errors tend to be systematic and correctable through a written codebook rather than opaque. Both benefit from the same two audits. Agreement metrics between coders are a related but different measurement — they tell you about consistency, not correctness, and two coders can agree perfectly while both being wrong about a rare class.

If precision is unknowable without an audit, should I stop using automated flags?

No — the alternative is not "no flags", it is manual review of everything, which has its own error rate and does not scale past a few hundred items. Automated tagging over a large corpus at 50% precision with a known base rate is far more useful than reading 30 transcripts by hand and generalising, because you can correct a known bias and you cannot correct an unknown sample. The failure mode is not using the flags; it is quoting their output as a count without ever having measured what the count means.

Related Resources

• Screener Accuracy: Why Most People Who Pass Your Screener Are Not Who You Wanted — the same arithmetic on a recruiting screener
• Tightening Your Screener Makes the Sample Purer and the Findings Worse — what happens when you move a threshold without pricing both errors
• You Cannot Spot-Check Your Way to Data Quality — the inspect-or-not decision this measurement feeds
• Structured Questions Guide — the six question types, and how asking directly removes the need to classify
• Singleton Themes: Why One-Off Comments Are the Only Estimate You Have of What You Missed — the sibling failure in thematic coverage
• AI Auto-Tagging for Customer Interviews — how the tags whose precision you are auditing get produced