Vocabulary Mismatch: Why the Searches That Feel Best Miss the Most

When a search returns junk, everyone does the same thing: they add another word to narrow it. That reflex is backwards. Adding a required term raises precision, which you can see, and multiplies your miss rate, which you cannot. The underlying reason is a measured property of language: when two people name the same thing independently, they agree less than one time in five. So the searches that feel best — tight, clean, obviously on-topic — are systematically the ones missing the most. The fix is not a better word. It is refusing to require any single word at all.

The measurement that should have ended keyword search

In 1987, Furnas, Landauer, Gomez and Dumais published "The Vocabulary Problem in Human-System Communication" in Communications of the ACM. They tested how people spontaneously name things across five different application domains. Their finding, stated in the paper's own abstract: "We studied spontaneous word choice for objects in five application-related domains, and found the variability to be surprisingly large. In every case two people favored the same term with probability <0.20."

Under 0.20. In every domain they tested.

The authors then drew out the design consequence, and it is blunt: "the popular approach in which access is via one designer's favorite single word will result in 80-90 percent failure rates in many common situations."

That is not a claim about bad search engines. It is a claim about people. Two competent colleagues, looking at the same concept, will pick different words about four times out of five. Any system that requires the searcher to guess the author's word is therefore starting from a base failure rate of roughly 80 percent — before anything technical goes wrong.

This is the mechanism behind the recall numbers that keep showing up in retrieval studies. Summarising Blair and Maron's classic finding that attorneys retrieved about 20 percent of relevant documents while believing they had retrieved 75 percent, NIST notes that "the authors attributed this to the inherent ambiguity of language." Same cause, measured twice, two decades apart.

The inversion: narrowing multiplies the miss

Here is where the sign flips, and it flips inside a single arithmetic step.

Model a search where a concept has roughly a 20 percent chance of being expressed with the word you chose — Furnas's number. If you require one term, you reach the material about 20 percent of the time. Now do the thing everyone does when results look noisy, and require a second term as well. If the two word choices are roughly independent, your chance of matching becomes 0.20 × 0.20 = 4 percent. Add a third required term and it is 0.8 percent.

Now run the same model in the opposite direction. Instead of requiring words, accept them — treat several different terms as equivalent routes to the same concept, so a match on any one of them retrieves the item:

Two things are worth saying about these tables. They assume independence between word choices, which is a simplification — in practice terms correlate, so treat the numbers as the shape of the effect rather than a forecast for your repository. And the shape is exactly what Furnas derived from his data: he named the optimal strategy "unlimited aliasing" and reported it "capable of several-fold improvements." The 3.4× at five aliases in the table above is an independent arrival at the same conclusion from the same starting probability.

The inversion, stated plainly: the same action that improves what you can see degrades what you cannot. Narrowing raises precision, which is visible on screen, and cuts recall by a multiplicative factor, which produces no feedback at all. Broadening does the reverse — it makes the results page look worse and the retrieval genuinely better. Every incentive in the interaction points the wrong way.

What a real miss looks like

Abstractions about vocabulary are easy to nod at and hard to feel, so here is a documented case from NIST's 2007 TREC Legal Track evaluation.

The topic sought scientific studies referencing health effects tied to indoor air quality. The negotiated Boolean query was (scien! OR stud! OR research) AND ("air quality" w/15 health) — a reasonable, professionally constructed query with stemming and a proximity operator. It missed a document that assessors judged relevant. NIST explains why: the document "did not contain required Boolean terms such as 'air' or 'health'," but was judged relevant because it referred to the "largest study ever" on whether "secondary smoke causes cancer" and to the "carcinogenic effects" of gas released from volatile organic compounds in shower water.

Read that back. The document is about air quality and health. It simply does not use the words "air" or "health." No amount of care in constructing the query would have caught it, because the failure is not in the query — it is in the assumption that a shared concept implies a shared word.

Across all 43 topics in that evaluation, NIST reports mean estimated recall of just 22 percent for these negotiated queries, "missing about 78% of the relevant documents (on average across all topics)."

Why your repository is worse than a document collection

A research repository has every vocabulary problem a document collection has, plus three of its own.

Three vocabularies per finding. A participant says I gave up. The researcher writes abandonment during setup. The summary says onboarding friction. These are the same finding in three registers, and a searcher will typically use a fourth: why do trials stall?

The searcher is the wrong person. The people who most need a two-year-old finding are the ones who were not in the room. They have no memory of the study's framing, its title, or the words the team was using that quarter, which are precisely the handles a keyword index offers.

Terminology drifts under you. Products get renamed, segments get redefined, and the phrase your team used confidently in 2024 reads as jargon by 2026. Old findings do not update their vocabulary. They just quietly stop being reachable.

The uncomfortable implication for taxonomies

The standard prescription for all of this is a controlled vocabulary — agree on the canonical term for each concept, tag everything with it, enforce it. This is genuinely worth doing, and a stable taxonomy is a real pillar of a working repository.

But notice what Furnas's result says about it: a controlled vocabulary is a decision to designate one favourite word per concept, which is the exact configuration measured at an 80 to 90 percent failure rate for people who did not participate in choosing it. A taxonomy disciplines the people who apply tags. It does nothing for the person typing into the search box two years later, who never saw the taxonomy and is using their own words.

So the taxonomy is necessary and insufficient, and the missing half is aliasing. Furnas's own conclusion was not "choose the right word." It was to accept many words as routes to the same thing. In practice that means every canonical tag needs a list of synonyms, participant phrasings, deprecated product names and adjacent terms attached to it — and that list needs to grow every time someone fails to find something.

How Koji attacks the vocabulary problem

Maintaining unlimited aliasing by hand is exactly the librarian bottleneck that kills repositories. Nobody has time to write ten synonyms for every theme, and the synonyms that matter most are the ones nobody thought of. This is where an AI-native platform does something a legacy tool structurally cannot.

Semantic retrieval instead of word matching. Koji indexes what text means rather than which characters it contains, which is aliasing without an alias list — the transcript that says I gave up before I got anywhere is reachable from a search for users abandon during setup without anyone having connected those phrases in advance. Traditional survey platforms such as SurveyMonkey give you keyword search over response text, which is precisely the one-favourite-word configuration Furnas measured at 80 to 90 percent failure.

Thematic analysis that applies one vocabulary consistently. Koji's automatic thematic analysis assigns themes across every interview by the same criteria each time. Human tagging drifts between people and across months; automated tagging gives the index a stable spine, which the searcher's synonyms can then be matched against.

Customizable AI consultants that carry your language. You can configure the AI consultant with your product names, your segment definitions and your internal shorthand, so the vocabulary of the index tracks the vocabulary of your company rather than a generic model's defaults.

And the structural fix: ask fewer questions in prose. Koji's structured questions come in six types — open_ended, scale, single_choice, multiple_choice, ranking, and yes_no. Five of those six produce values rather than sentences. A single_choice answer is a known option, a scale answer is a number, a ranking is an order. None of them has a vocabulary problem, because there is no wording to guess: you filter and aggregate instead of searching. Open_ended questions remain where depth lives, and they are exactly the ones that need semantic retrieval to stay reachable. The practical discipline is to notice which of your open_ended questions were only ever going to be counted, and convert those — every one you convert is a permanent exit from the vocabulary problem rather than a mitigation of it.

What to do on Monday

• Stop narrowing first. When results look noisy, broaden and then filter by study, date or segment. Filters cut on facts; extra required words cut on someone else's word choice.
• Search for the concept three ways before concluding nothing exists — the participant's likely words, the researcher's likely words, and the executive's likely words.
• Let Koji harvest your misses. Every time someone finally finds a finding after failing twice, record the failed query as an alias on that finding. Failed queries are the only direct evidence you get about the gap between your index and your colleagues' vocabulary.
• Treat a zero-result search as data, not as an answer. It is a statement about wording until proven otherwise.
• Convert countable open_ended questions to structured types at study design time, before the vocabulary problem exists.

Koji cannot make two colleagues choose the same words, and no platform can. What it can do is stop requiring them to. The one-sentence version: your repository is not failing because people chose bad words. It is failing because it requires them to choose the right one, and the measured probability of that is under 0.20.

Frequently asked questions

What is vocabulary mismatch?

Vocabulary mismatch is the phenomenon where different people name the same concept differently, so a searcher's term fails to match the author's term for material that is genuinely relevant. Furnas and colleagues measured it across five domains in 1987 and found that two people favoured the same term with probability under 0.20 in every case.

Why does adding search terms make things worse?

Each additional required term must independently match the author's word choice. Starting from roughly a 20 percent chance per term, requiring two terms drops the chance of matching a relevant item to about 4 percent and three terms to under 1 percent. The results you do get look cleaner, so precision appears to improve while recall collapses invisibly.

Doesn't a controlled vocabulary solve this?

Only for the people applying tags. A controlled vocabulary designates one canonical term per concept, which is the configuration Furnas associated with 80 to 90 percent failure rates for anyone who did not help choose it. A taxonomy needs to be paired with aliasing — synonyms, participant phrasings and deprecated names — so that the searcher's word reaches the canonical one.

Does semantic search make this go away?

It removes the requirement to guess exact words, which addresses the dominant cause, and it substantially raises recall over keyword matching. It does not reach 100 percent, and it can still miss material that is conceptually relevant but framed very differently. Verify with a known-item test rather than assuming the problem is solved.

How do I know if vocabulary mismatch is hurting my team?

Take findings you know exist, have colleagues who did not run those studies try to find them, and record every failed query. A low hit rate combined with failed queries that are obviously reasonable phrasings is direct evidence of mismatch rather than of missing content.

Which questions should be structured rather than open-ended?

Any question whose answers you were only ever going to count or compare. Satisfaction, preference between options, relative priority and yes or no decisions belong in scale, single_choice, multiple_choice, ranking and yes_no formats. Reserve open_ended for questions where the reasoning matters, since those are the ones worth paying the retrieval cost for.

Related Resources

• Structured Questions Guide — the six question types and when to use each
• AI Auto-Tagging Customer Interviews — emergent and codebook-guided tagging
• Insight Repository Methodology — building a taxonomy that survives contact with searchers
• Search Interview Transcripts — search modes, filters and query patterns in Koji
• Qualitative Research Codebook — defining codes consistently across a team
• Information Architecture Research Guide — testing whether your categories match users' mental models