Diagnostic Overshadowing: Why One Known Problem Hides Every New One (2026)

The short answer

Diagnostic overshadowing is what happens when a problem you already know about absorbs the evidence for a problem you do not. A segment, an account, or a feature acquires a reputation, and from then on every new report arriving from it gets filed under the established explanation. The new issue is not merely missed. Its reports are converted into additional evidence for the old one, so the wrong problem looks better supported every quarter.

The term comes from clinical medicine. Jones, Howard and Thornicroft named it in Acta Psychiatrica Scandinavica in 2008 (volume 118, issue 3, pages 169 to 171) to describe a specific and lethal pattern: patients with a psychiatric diagnosis who reported physical symptoms had those symptoms attributed to their mental illness rather than investigated. A 2023 systematic review in the Journal of Clinical Nursing by Molloy, Brand, Munro and Pope defines it as a phenomenon that "occurs when physical symptoms reported by mental health consumers are misattributed to mental disorders by health professionals." The label attached to the source changed how the signal from that source was read.

Feedback analysis has the same structure, minus the mortality. This article explains the mechanism, shows why it is arithmetically worse than it feels, separates it from confirmation bias, and gives you four checks that detect it in your own analysis.

What overshadowing actually is

The mechanism has three parts, and all three must be present:

1. A durable label attached to a source. Not a hypothesis about the world, but a property assigned to a channel: this enterprise account complains about performance, this segment always wants integrations, this feature generates confusion tickets.
2. A new signal that is textually compatible with the label. Compatibility is doing the work here, not similarity. A silent data-sync failure produces reports that read as "the app is slow." Slowness is exactly what the existing label predicts.
3. An assignment step that resolves ambiguity toward the established category. A human coder, an auto-tagger, or a support agent picks the code that already exists and already fits.

Notice what is absent from that list. Nobody needs to want a particular conclusion. Nobody needs to be defending a hypothesis. The label does the work on its own, which is why this survives every debiasing intervention aimed at motivated reasoning.

Why this is not confirmation bias

Confirmation bias is about what you are looking for. You have a belief, so you seek and weight evidence that supports it. The corrective is structural: neutral moderation, pre-committed analysis plans, questions that could return the answer you do not want.

Overshadowing is about where evidence gets filed once it arrives. You can hold no hypothesis at all, want nothing in particular, and still route a novel report into an existing bucket because that bucket fits and exists. An analyst with perfect motives and a mature codebook is more exposed than a sloppy one, for reasons the next section makes precise.

The two also fail differently under scrutiny. Ask someone with confirmation bias to justify a conclusion and they will produce the supporting evidence they gathered. Ask someone who has overshadowed and they will produce a large, internally consistent, growing pile of reports for the dominant theme, all of which are genuinely in the data. Nothing looks wrong. The count is real. It is the attribution that is wrong.

The arithmetic: absorption, not frequency, decides what you discover

This is the part that surprises people. Whether a new problem becomes visible is not primarily a function of how often it occurs. It is a function of how compatible its reports are with the label already in place.

Work an example. An enterprise segment sends 200 pieces of feedback a quarter. It carries an established code, Performance, which is textually compatible with roughly 70 percent of the free-text reports that segment produces. A new data-integrity bug ships and generates 12 reports.

Of those 12, suppose 9 describe the symptom in terms the Performance code comfortably accepts: things felt slow, the page took a long time to settle, records appeared late. Three describe it in terms Performance cannot accept: a number was wrong.

• Observed count for the new bug: 3
• Observed count for Performance: 140 plus 9, so 149

If your triage rule needs five mentions before an issue gets a ticket, the data-integrity bug is invisible at 3. Meanwhile Performance has just gained nine reports and looks like it is getting worse. The team schedules performance work. The bug that corrupts customer data is not on the list, and the evidence for it is now sitting inside the case for something else.

Three consequences follow, and the third is the one worth remembering:

• The new problem is undercounted by a factor set by compatibility, not by severity. A data-loss bug and a cosmetic bug with identical compatibility are hidden identically.
• The old problem is overcounted by exactly the amount the new one lost. This is a transfer, not a leak. The total stays right, which is why volume checks never catch it.
• A more mature codebook absorbs more. A broad, well-established, well-documented code is precisely one that a coder can confidently apply to an ambiguous report. Codebook maturity and novelty detection pull against each other, and nobody warns you about that when you are told to standardise your codes.

Why the research literature on it is thin, and what that tells you

There is a detail in the clinical evidence base worth borrowing. A 2024 scoping review in the International Journal of Mental Health Nursing by Preston, Christmass, Lim, McGough and Heslop searched six databases, identified 1,995 records, and included 42 studies. Of those 42, only 3 addressed direct causes of overshadowing. The remaining 39 addressed background factors. The review reports six key themes, with communication barriers, stigma and knowledge deficiencies the most prominent.

Three studies out of 42 on direct causes, after a search of nearly 2,000 records. That ratio is not an accident of a young field. A direct cause of overshadowing is only observable if you know what the correct attribution should have been, which means you need the case where the misattribution was eventually caught. Those cases are rare in the record precisely because the mechanism suppresses them.

The same constraint applies to you. You cannot audit overshadowing by looking at your themes, because in your themes it is invisible by construction. You have to go at it from the assignment step.

Four checks that actually detect it

Each of these attacks the assignment step rather than the output.

1. The residual rate. Track the share of reports that land in no existing code, per period, per source. A healthy analysis has a stable and non-trivial residual. A residual that falls toward zero as a codebook matures is the signature of absorption, not of a complete codebook. If every new report fits, your codes have become too accommodating to be informative.

2. The blind re-code. Take a sample of reports from a labelled source, strip every identifier that reveals the source, and have them coded by someone who does not know which account or segment they came from. Compare against the original coding. Disagreements cluster exactly where overshadowing lives. This is the closest thing to a controlled test available, and it costs an afternoon.

3. The compatibility audit. For each dominant code, write down what fraction of your typical free-text reports that code could plausibly accept. Any code that could accept more than about half of what arrives is a sink, not a category. Split it.

4. The disconfirming-detail sweep. For every dominant theme, search its assigned reports for details the theme does not predict. In the worked example above, "a number was wrong" is not something a performance problem predicts. Those details are the three reports you needed. They are already in your data, filed under the wrong heading.

Croskerry, writing in Academic Medicine in 2003 (volume 78, issue 8, pages 775 to 780), catalogued the cognitive failures behind diagnostic error and argued the principal remedy is metacognition, which he describes as "a reflective approach to problem solving that involves stepping back from the immediate problem to examine and reflect on the thinking process." All four checks above are mechanical substitutes for that reflection, which is the point. Reflection does not scale across 400 interviews. A residual-rate chart does.

How Koji handles this

Koji is an AI-native research platform, and AI analysis is genuinely double-edged here. An auto-tagger applying an existing codebook is an absorption machine: it is optimised to find the best-fitting existing code, which is the exact failure mode described above. Koji is built so that you do not have to choose between speed and novelty detection.

• Emergent and codebook-guided analysis are separate modes, not one blended pass. Koji can code a set of interviews against your existing codebook and, independently, derive themes from the transcripts with no codebook at all. Running both over the same corpus and diffing them is a blind re-code you get for free, at full sample size rather than on an afternoon sample.
• Every extracted item stays grounded in the transcript. Koji ties analysis output back to the source passage, so a disconfirming-detail sweep is a search over evidence rather than a re-reading of 400 transcripts.
• Structured questions give you signal that cannot be absorbed. Koji supports six question types: open_ended, scale, single_choice, multiple_choice, ranking, and yes_no. Free text is what overshadowing feeds on, because free text is ambiguous enough to be compatible with an established code. A single_choice question asking which of five specific things went wrong, or a yes_no asking whether any value displayed was incorrect, produces an answer that no label can quietly reinterpret. This is the cheapest structural defence available: put one unambiguous item next to your open-ended probe.
• AI moderation follows the participant instead of the file. Koji's AI interviewer probes what the participant actually said in the moment. It does not arrive carrying a reputation for the account, which is the human moderator's hardest problem in a renewal conversation with a customer everyone already has an opinion about.
• Interview quality scoring is independent of theme assignment. Koji scores each interview on a 1 to 5 scale against the research goals, so a session can be flagged as high value even when its content does not map onto any theme you already track.

The traditional alternative is a survey tool plus a spreadsheet plus an analyst under deadline. Legacy survey platforms such as SurveyMonkey hand you counts per predefined option and free text nobody has time to read, which means the label assigned upstream is the only label that will ever exist. Koji makes the competing interpretation cheap enough to actually generate.

Common mistakes

• Treating a falling residual rate as progress. It is the main warning sign. Celebrate a stable residual, not a vanishing one.
• Auditing themes instead of assignments. The output is where the evidence has already been destroyed.
• Assuming a good codebook protects you. It increases exposure. Maturity and absorption rise together.
• Confusing this with a volume problem. Overshadowing is a transfer between categories, so totals reconcile perfectly. Any check based on report volume will pass.
• Blaming the coder. The mechanism needs no bad judgment, no motive, and no fatigue. It needs a label and an ambiguous report.

Frequently asked questions

What is diagnostic overshadowing in feedback analysis?

It is the pattern where a problem already associated with a source absorbs the evidence for a new problem from that same source. Reports about the new issue get filed under the established theme because they are textually compatible with it. The new issue is undercounted and the old one is overcounted by the same amount, so the wrong problem appears to be getting worse.

How is diagnostic overshadowing different from confirmation bias?

Confirmation bias governs what you look for and how you weight what you find, and it requires you to hold a belief you are motivated to protect. Overshadowing governs where a report gets filed after it arrives, and it needs no belief and no motive at all. An analyst with a mature codebook and entirely neutral intentions is more exposed, not less, because a well-established code is easier to apply confidently to an ambiguous report.

Does AI thematic analysis make overshadowing better or worse?

Both, depending on how it is configured. An AI that only applies your existing codebook is an absorption engine, since finding the best-fitting existing code is exactly its objective. But AI also makes the corrective affordable: you can code the same corpus twice, once against the codebook and once with no codebook at all, and diff the results. That comparison is impractical by hand and routine in Koji.

How do I detect overshadowing in my own analysis?

Use four checks. Track the residual rate, meaning the share of reports fitting no existing code, and treat a decline toward zero as a warning. Run a blind re-code on source-stripped reports. Audit each dominant code for how much of your typical feedback it could accept, and split any code that could take more than about half. Finally, search each dominant theme's reports for details that theme does not predict.

Why does a better codebook make the problem worse?

Because the quality that makes a code useful, being clearly defined and broadly understood, is the same quality that lets a coder apply it confidently to a report it does not really explain. Codebook maturity raises the probability that an ambiguous new signal finds a comfortable existing home. Standardisation advice rarely mentions this tension, but novelty detection and codebook maturity genuinely pull in opposite directions.

Can structured questions prevent overshadowing?

They prevent it for the specific things you ask about. Overshadowing feeds on ambiguity, so an answer with no interpretive slack cannot be reassigned. A yes_no item asking whether any displayed value was incorrect, or a single_choice item naming five concrete failure modes, produces evidence that survives a coder's preference for an existing label. Free text remains essential for discovering what you did not think to ask, which is why Koji pairs both in one study.

Related Resources

• Confirmation Bias in User Research - the desire-driven failure this one is often mistaken for
• How to Build a Qualitative Research Codebook - building the codes whose maturity raises your exposure
• AI Auto-Tagging for Customer Interviews - emergent versus codebook-guided coding, and why running both matters
• Customer Feedback Categorization - taxonomy design, including how wide a category should be allowed to get
• When a Loud Complaint Hides a Quiet One - the volume-driven cousin of this failure
• Structured Questions Guide - the six question types and when an unambiguous item beats free text