Precision, Recall, and the Research Nobody Can Find

Your research repository has two performance numbers, and you can only ever see one of them. Precision is the share of what your search returned that was actually useful — cheap to check, because the results are on the screen in front of you. Recall is the share of everything relevant in the repository that your search actually returned — and measuring it honestly means reading the whole repository. So teams measure precision, feel good, and quietly assume recall took care of itself. It did not. The best evidence we have says trained professionals searching a corpus they knew well, with high stakes and unlimited motivation, retrieved about a fifth of what was there while believing they had retrieved three quarters.

This article explains why that gap is structural rather than a sign of a bad tool, what it means for a research repository specifically, and what you can actually do about it.

The two numbers, and why you only ever have one

Take a question you have asked your repository: what do we know about why trial users never invite a teammate? You run the search. Nine results come back. You skim them, and six are genuinely on point.

Precision is 6/9, about 67 percent. You computed it in ninety seconds, because computing it only required looking at what came back.

Recall is 6 divided by the number of relevant findings that exist anywhere in the repository — and you do not know that number. You cannot know it without opening every study you have ever run and judging each one against this question. That is not a tooling limitation. It is arithmetic: precision is bounded by the size of your result list, and recall is bounded by the size of your corpus.

Put real numbers on it. Suppose your repository holds 1,200 findings and a careful relevance judgment takes thirty seconds. Checking precision on the top ten results costs five minutes. Checking recall for that same single question costs ten hours — one and a quarter working days. Do it for twenty questions and you have spent 200 hours, five working weeks, measuring the search box. The cost ratio for a single query is 120 to 1.

That asymmetry is the whole problem. It is not that teams are lazy about recall. It is that recall is priced out of reach, so it goes unmeasured, and unmeasured quantities get assumed to be fine.

Three decades of measurements, and they keep saying the same thing

The foundational study is Blair and Maron's 1985 evaluation of a full-text retrieval system supporting litigation, published in Communications of the ACM. The U.S. National Institute of Standards and Technology, summarising it in the overview of its 2006 TREC Legal Track, describes the result plainly: the study "found that while attorneys believed they had found 75% of the relevant documents for litigation involving a train accident, in fact only an estimated 20% of relevant documents were discovered. The authors attributed this to the inherent ambiguity of language."

Sit with the two numbers. Believed 75 percent. Achieved 20 percent. That is an overestimate by a factor of 3.75, by professionals whose careers depended on the answer.

The obvious response is that 1985 was a long time ago and search has improved enormously since. So look at the modern replication. In the 2007 TREC Legal Track, NIST evaluated retrieval over a large document collection using a statistical sampling method rather than guesswork, across 43 topics. The queries under test were not casual — each was a Boolean query negotiated between two opposing parties, both of whom had every incentive to get it right. NIST reports: "The mean estimated recall of the reference Boolean run (refL07B) was just 22%. Hence the final negotiated boolean query was missing about 78% of the relevant documents (on average across all topics)."

Twenty-two years apart, with vastly better technology, an adversarially negotiated query and a rigorous sampling methodology: 20 percent, then 22 percent.

Two further details from that 2007 evaluation matter for anyone running a repository. First, the variance was enormous — NIST notes that estimated recall "varied considerably per topic, from 0% (topic 77) to 100% (topic 84)." Your average recall tells you almost nothing about the recall of the specific question you asked this morning. Second, retrieving vastly more did not rescue it: the best run scored 47 percent recall even when allowed to return 25,000 documents. Retrieving deeper helps, and it does not get you to completeness.

Why the trap is the pool, not the search box

Here is the part that catches even careful teams: the standard method for evaluating search systems has this same blind spot baked in, and it is honest about it.

Large-scale retrieval evaluations use pooling. You cannot judge millions of documents, so you take the union of what all the participating systems returned, judge that pool, and treat everything outside it as not relevant. NIST describes the practice on its own relevance-judgments page, and note the scare quotes it puts around the key word: "The relevance judgments are considered 'complete' for that particular set of documents. By 'complete' we mean that enough results have been assembled and judged to assume that most relevant documents have been found."

Complete means assumed complete. A document that no system surfaced is scored as irrelevant by construction, not by judgment. In the 2006 Legal Track overview NIST states the consequence directly, that "our pool-based effectiveness measures do not provide a measure of the absolute effectiveness of any of the participating systems."

Now translate that to your repository. When someone reports that they searched the repository and there are three prior studies on this, they have described their pool. They have made a statement about their query, their vocabulary and their patience. They have made no statement whatsoever about what the repository contains. The finding that used different words, sat under a different tag, or was written by someone who left last year is scored as non-existent by exactly the same mechanism — silently, and with no error message.

This is worth naming because the failure mode is invisible in a way that most research failures are not. A bad sample announces itself in the demographics table. A leading question shows up when someone reads the guide. A missed finding renders as an empty space on a screen that looks completely normal.

What relevance even means for a research repository

Retrieval evaluation requires a definition of relevance, and TREC's working definition is a good one to steal. NIST states it as: "If you were writing a report on the subject of the topic and would use the information contained in the document in the report, then the document is relevant."

That framing is useful because it is about the decision, not about topical similarity. A finding about onboarding friction is relevant to your pricing question if you would cite it in the pricing memo. Most repository search is tuned for topical similarity, which is why it returns nine articles that are all about the thing you typed and none of the three that would have changed your mind.

Research repositories also make the recall problem harder than a document collection does, in three specific ways:

• The unit is small and the language is loose. A single interview can contain a dozen distinct findings, described in the participant's words, the researcher's words, and the summary's words — three different vocabularies for one idea.
• The corpus grows monotonically and nothing ever gets retired. Every study you run makes every future search harder, because the number of relevant items per question grows while the length of the results list does not.
• The searcher is usually not the author. The person who most needs a two-year-old finding is typically someone who was not in the room and does not know it exists.

How Koji changes the shape of the problem

Koji cannot repeal the arithmetic. Nothing can — recall over a corpus costs corpus-sized effort to verify. What an AI-native platform can do is attack the three drivers that make measured recall low in the first place, and make sampling-based measurement cheap enough that you actually do it.

Retrieval that is not word matching. Legacy repositories and survey tools like SurveyMonkey give you keyword search over a text field, which means a finding is only reachable through the exact words someone happened to type. Koji indexes meaning, so a search for users abandon during setup reaches a transcript where a participant said I gave up before I got anywhere. Blair and Maron attributed their result to "the inherent ambiguity of language"; semantic retrieval is a direct attack on that cause rather than on its symptoms.

Consistent description at scale. Koji's automatic thematic analysis assigns themes across every interview using the same criteria every time, so the vocabulary of the index does not drift with whoever wrote the summary that week. Manual tagging drifts by definition — different people, different weeks, different words.

Structure that does not need retrieving at all. This is the underrated one. Koji's structured questions come in six types — open_ended, scale, single_choice, multiple_choice, ranking, and yes_no — and the five non-open types produce values, not prose. A scale answer is a number in a field. A single_choice answer is a known option. You do not search for those; you filter and aggregate them, which has a recall of exactly 100 percent because nothing is hiding in the phrasing. Every question you can express as structured rather than open_ended is a question permanently removed from the recall problem. Open_ended questions still carry the depth, and they are the ones that need semantic retrieval and thematic analysis to stay findable.

Cheap re-asking. Traditional research economics say a missed finding costs you a six-week study to recover. With AI-moderated interviews running in parallel, and voice interviews collecting depth without scheduling a single call, the cost of re-answering a question drops far enough that an occasional retrieval miss stops being a disaster. That does not excuse low recall — it makes an honest recall number safe to look at, which is a precondition for measuring it at all.

A protocol you can actually run

You cannot afford to measure recall exactly. You can absolutely afford to estimate it, and NIST's own answer to this problem was statistical sampling rather than exhaustive judgment. Do the same thing at your scale:

1. Pick 40 findings you know exist. Take them from studies across at least two years, chosen by someone who did not run them.
2. Write the question each finding answers, phrased the way a product manager would ask it — not the way the report titled it.
3. Have someone who did not run those studies search for each one, with a fixed time limit of two minutes per question.
4. Count the hits. The proportion found is a genuine estimate of your repository's recall, with a margin of about ±15 percentage points at 40 probes. That is enough to distinguish a repository at 45 percent from one at 80 percent, which is the decision you actually need to make.
5. Read the misses, not the hits. Every miss names a specific vocabulary gap, and fixing named gaps is tractable work in a way that "improve findability" is not.

Forty probes is roughly an afternoon. Compare that with the five weeks that exhaustive measurement of twenty queries would cost, and note what you gave up: a confidence interval instead of a point estimate. That is a very cheap thing to give up in exchange for being able to do the measurement at all.

The honest summary

Precision is what you can see, recall is what decides whether the repository was worth building, and the two have almost nothing to do with each other. Every team that has measured recall carefully — in 1985 with lawyers and in 2007 with NIST's sampling methodology — has found a number between 20 and 25 percent while the people doing the searching believed it was three or four times higher.

The correct response is not despair and not a new tool purchase. It is to stop treating I searched and found nothing as evidence of absence, to estimate your recall by sampling rather than assuming it, and to move as many of your questions as possible into structured formats where retrieval is not required. Start by assuming your repository's recall is about a third of what your team thinks it is, then go measure it.

Frequently asked questions

What is the difference between precision and recall in a research repository?

Precision is the proportion of returned results that are actually relevant to your question. Recall is the proportion of all relevant material in the repository that your search actually returned. Precision is measurable in minutes because you only inspect what came back. Recall requires judging the entire corpus, which for a 1,200-item repository at thirty seconds per judgment is about ten hours per question.

Why can't I just measure recall directly?

Because the cost scales with the size of your repository rather than the size of your results list. Measuring precision on ten results costs about five minutes; measuring true recall for the same question costs roughly 120 times more. The practical answer is not exhaustive measurement but sampling — a known-item test on about 40 findings gives you a usable estimate to within roughly ±15 percentage points in an afternoon.

Is 20 percent recall really typical?

It is what careful measurement keeps producing. Blair and Maron measured about 20 percent in 1985 with attorneys who believed they had found 75 percent. NIST's 2007 TREC Legal Track measured mean estimated recall of 22 percent for adversarially negotiated Boolean queries across 43 topics. Your repository may differ, but the burden of proof sits with the claim that it is much higher, and that claim is testable.

Does semantic or AI search fix the recall problem?

It attacks the largest single cause, which Blair and Maron identified as the ambiguity of language, and it meaningfully raises recall over keyword matching. It does not make recall measurable and it does not make it 100 percent. Treat semantic retrieval as a large improvement to be verified by sampling, not as a reason to stop checking.

How is this different from auditing for duplicate studies?

A duplicate-study audit counts how often the same question was commissioned twice and assigns ownership for preventing it. That is governance over studies. This article is about instrumenting the search system itself — whether a given query can reach material that demonstrably exists. The two are complementary: retrieval failure is one of the mechanisms that produces duplicate studies in the first place.

What is the single most useful thing to do first?

Run the 40-probe known-item test described above, then read only the misses. Each miss is a concrete, named vocabulary or structure gap you can fix, whereas an aggregate recall score on its own tells you the size of the problem without telling you where it is.

Related Resources

• Structured Questions Guide — the six question types, and why structured answers never need retrieving
• Research Repository Guide — what to store and how to set a repository up
• Insight Repository Methodology — taxonomy, atomic insights and governance
• Search Interview Transcripts — the search modes and filters available in Koji
• Re-Research Audit: Duplicate Studies — measuring how often the same question gets asked twice
• Study-Level Description for Findability — the description layer that makes studies reachable