{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-11-11T19:58:06.566Z"},"content":[{"type":"documentation","id":"9c4aa797-07b1-4018-a163-18d45a20c7ec","slug":"surveillance-bias-detection-research","title":"Surveillance Bias: Why the Team That Measures Best Looks Worst (2026)","url":"https://www.koji.so/docs/surveillance-bias-detection-research","summary":"Surveillance bias occurs when detection intensity differs between compared groups, so the better-instrumented group posts worse event rates. Bilimoria et al 2013 JAMA showed hospitals in the highest imaging quartile reporting 13.5 clots per 1,000 versus 5.0 in the lowest, and higher-quality hospitals scoring worse. Lead time bias inflates survival by detecting earlier without changing outcomes, as in the Mayo Lung Project where median survival was 16.0 versus 5.0 years while mortality was unchanged. Observed event rate equals true incidence times detection intensity times reporting propensity. The test is whether bad outcomes fell or only the detection-to-outcome interval grew.","content":"# Surveillance Bias: Why the Team That Measures Best Looks Worst\n\n**Answer first:** Any metric that counts detected events measures three things multiplied together: how often the event happens, how hard you look for it, and whether anyone records it. Only the first is what you care about. When you compare teams, products or periods with different detection intensity, you are mostly measuring the looking. This is surveillance bias, and its close relative lead time bias means an intervention can improve every survival number you report while changing no outcome at all. The test that separates them is simple, and almost nobody runs it.\n\n## The finding that should end cross-team metric comparisons\n\nIn 2013, Karl Bilimoria and colleagues published a study in *JAMA* (310(14):1482-1489) that is the cleanest demonstration of this problem ever produced. They merged 2010 Hospital Compare and American Hospital Association data from 2,838 hospitals, then used 2009-2010 Medicare claims covering **954,926 surgical patient discharges** from 2,786 hospitals across 11 major operations.\n\nPostoperative venous thromboembolism, a blood clot, is a widely reported hospital quality metric that was slated for use in pay-for-performance programmes. The logic is intuitive: good hospitals prevent clots, so good hospitals should report fewer clots.\n\nThe data said the opposite.\n\n| Hospital structural quality quartile | Clot-prevention adherence | Risk-adjusted clot rate |\n| --- | --- | --- |\n| Lowest quality quartile | 93.3 percent | 4.8 per 1,000 |\n| Highest quality quartile | 95.5 percent | 6.4 per 1,000 |\n\nHospitals with more accreditations and more quality initiatives did better on prevention and **worse** on the outcome metric, both differences significant at P < .001. Greater adherence to prevention was weakly associated with worse risk-adjusted event rates.\n\nThen the authors found the cause. They sorted hospitals by how often they ordered the imaging that detects clots:\n\n| Imaging use quartile | Diagnostic imaging rate | Risk-adjusted clot rate |\n| --- | --- | --- |\n| Lowest | 32 studies per 1,000 | 5.0 per 1,000 |\n| Highest | 167 studies per 1,000 | 13.5 per 1,000 |\n\nA five-fold difference in looking produced a 2.7-fold difference in finding, rising in a clean stepwise fashion across quartiles (P < .001). The authors' conclusion was blunt: surveillance bias limits the usefulness of the measure both for hospitals trying to improve and for patients trying to choose one.\n\nAs Haut and Pronovost put it in their *JAMA* commentary on the phenomenon (305(23):2462-2463, 2011), the principle is that the greater the intensity of the search for a condition, the greater the likelihood it will be found.\n\n## Why this is your metrics dashboard\n\nSubstitute freely. The structure survives every substitution:\n\n| Your metric | The team that looks harder | What the metric actually rewards |\n| --- | --- | --- |\n| Production errors per release | The squad with the best error monitoring and alerting | Turning off Sentry |\n| Support tickets per 1,000 accounts | The team that made it easiest to file a ticket | Hiding the help widget |\n| At-risk accounts flagged | The CS org with the best health scoring | Scoring nothing |\n| Accessibility defects found | The team that actually runs an audit | Not auditing |\n| Security incidents reported | The org with a functioning speak-up culture | Discouraging reports |\n| Usability problems per study | The researcher who ran more sessions with tougher tasks | Running easier tasks |\n\nEvery row is a case where the better-performing team posts the worse number, and where the cheapest way to improve the metric is to degrade the measurement. If your quarterly review compares detected-event counts across teams whose instrumentation differs, the review is ranking instrumentation.\n\n## The sign inversion: two biases pulling opposite ways\n\nIt is worth being precise about how this differs from [immortal time bias](/docs/immortal-time-bias-retention-analysis), because the two are mirror images and teams routinely have both at once.\n\n| | Immortal time bias | Surveillance bias |\n| --- | --- | --- |\n| What is unearned | Survival time | Detection |\n| Direction of error | Flatters the treatment | Penalises the better-measured group |\n| Where it lives | The group definition | The measurement process |\n| Fixed by | Realigning the clocks | Measuring or equalising the looking |\n\nOne makes your feature look better than it is. The other makes your best team look worse than it is. Neither is visible in the metric itself, and no amount of statistical adjustment on the outcome fixes either one, because in both cases the corruption entered before the outcome was recorded.\n\n## Lead time bias: improving the number without improving anything\n\nThe second half of this problem is subtler and, in product work, more expensive.\n\nThe Mayo Lung Project was a randomised trial of lung cancer screening in 9,211 male smokers conducted between 1971 and 1983. The intervention arm was offered chest x-ray and sputum cytology every four months for six years. Marcus and colleagues published extended follow-up in the *Journal of the National Cancer Institute* (92(16):1308-1316, 2000), with a median follow-up of 20.5 years.\n\nHere is what screening achieved.\n\n**Survival**, measured from diagnosis, was clearly better in the screened arm (P = .0039). Among patients with resected early-stage disease, **median survival was 16.0 years in the screened arm versus 5.0 years in the usual-care arm**. On the metric that every hospital brochure reports, screening looked like a triumph: it more than tripled survival.\n\n**Mortality**, the rate at which people actually died of lung cancer, was 4.4 per 1,000 person-years in the screened arm and 3.9 in the usual-care arm (P = .09). Screening did not reduce deaths. If anything the point estimate ran the wrong way.\n\nBoth numbers are correct. Survival is measured from the moment of diagnosis, and screening moves the moment of diagnosis earlier. If you find the disease three years sooner and the person dies on exactly the same day, survival from diagnosis rises by three years and nothing whatsoever has improved. That gap is the lead time.\n\nA follow-up analysis by the same group (*JNCI* 98(11):748-756, 2006) tracked lung cancer incidence through 1999 and found 585 cases in the intervention arm against 500 in usual care. The excess persisted after 16 additional years of follow-up, which is the signature of **overdiagnosis**: finding disease that would never have caused a problem.\n\n## The three-way decomposition\n\nThe generalisable idea underneath all of this is worth stating as a formula, because it makes the failure mode obvious:\n\n**Observed event rate = true incidence x detection intensity x reporting propensity**\n\nYou care about the first term. You control the second and third. Any change in your looking or your logging moves the observed rate without moving reality, and the observed rate is the only one you can see.\n\nThis yields three rules:\n\n1. **Never compare detected-event rates across units with different instrumentation.** Compare within a unit over a period when instrumentation did not change, or explicitly measure and adjust for the looking.\n2. **When a detection improvement ships, expect the metric to get worse, and say so before it happens.** Announce the expected step change in advance. A team that predicts its own metric regression retains credibility; a team that explains it afterwards does not.\n3. **Report the denominator of looking alongside the numerator.** Bilimoria's contribution was not discovering that clots vary. It was measuring the imaging rate. Publish \"errors per release\" next to \"percentage of code paths instrumented\" and the comparison becomes interpretable.\n\n## The one test that separates a real win from lead time\n\nWhen someone claims an early-detection programme worked, ask a single question:\n\n**Did the number of bad outcomes fall, or did the time between detection and the bad outcome grow?**\n\nOnly the first is a win. The second is arithmetic.\n\nConcretely, if a customer success team says \"accounts we flag survive 11 months from flag date, versus 4 months for unflagged accounts from when their problems started\", they have reported a survival statistic and proved nothing. The correct measure is the churn rate of the whole eligible population before and after the programme, ideally with a randomised holdout of accounts that are scored but not contacted.\n\nThe holdout is what kills the argument, and it is also what almost nobody is willing to run, because it means deliberately not saving some accounts. That reluctance is exactly why so many retention programmes have been running for years with no evidence they do anything. The equivalent of the overdiagnosis finding is the account flagged as at-risk that was never going to churn: every one of those is a \"save\" your team gets credit for and a cost your business absorbed for nothing.\n\n## Surveillance bias inside qualitative research\n\nThis is not only a quantitative problem, and the version that hides inside interview studies is the one researchers most often miss.\n\nA moderator runs 20 interviews over three weeks: 10 with enterprise customers and 10 with SMB customers. By week two, the enterprise conversations have surfaced something interesting, so the moderator probes those threads harder, asks more follow-ups, and spends longer on the topic. The report concludes that enterprise customers have more integration pain.\n\nDo they? Or were they simply searched harder? Detection intensity differed systematically between the comparison groups, introduced by the researcher, and the finding is not separable from the method that produced it. Unequal probing depth across segments is surveillance bias with a human moderator as the imaging machine, and it appears in the limitations section of approximately no research reports.\n\nThe same mechanism operates across time. A researcher who runs a study before and after a redesign knows what changed and probes the changed areas harder in round two. More problems surface in the changed areas. That is not evidence the redesign made things worse.\n\n## The modern approach: how Koji helps\n\nThe fix for surveillance bias in research is a constant instrument, applied with the same intensity to every group and every round. That is difficult for humans, who get curious, tired, and biased toward what they already believe, and straightforward for a well-specified AI moderator.\n\n- **Identical probing depth across segments.** A Koji study runs the same brief for every participant. Enterprise and SMB respondents get the same questions with the same follow-up logic, so a difference between segments is a difference in what they said, not in how hard someone dug.\n- **Structured questions give you a detection-independent baseline.** The six types matter here specifically because closed types are immune to probing depth. A `scale` rating, a `single_choice` selection, a `multiple_choice` set, a `ranking` order and a `yes_no` answer mean the same thing regardless of how long the conversation ran. `open_ended` questions with AI follow-up are where depth varies, so pair them: if the open-ended themes diverge between segments but the scale and ranking data do not, you are looking at a probing artifact rather than a real difference. The [structured questions guide](/docs/structured-questions-guide) covers how to build that pairing.\n- **Consistency across rounds.** Re-running the identical brief before and after a change means round two is not searched harder than round one. Manual longitudinal research almost never achieves this, because the moderator has learned things in between.\n- **Automatic thematic analysis applies one standard.** Every transcript is coded by the same process, so a theme is not more likely to be recorded in the segment the researcher found interesting.\n- **Real-time reporting exposes the volume difference.** You can see immediately whether one segment produced more content than another, which is the diagnostic signal for unequal detection.\n\nAgainst legacy tooling this is a structural advantage rather than a speed one. A SurveyMonkey field applies a constant instrument but cannot probe at all, so it never gets the mechanism. A human interview programme probes well but cannot hold intensity constant across 40 conversations. AI moderation is the only approach that delivers both: adaptive follow-up within a fixed, auditable instruction set. Teams using AI-assisted research report substantially faster time-to-insight, but the more durable benefit here is that the instrument does not drift between the groups you are comparing.\n\nA caveat worth stating plainly, because it is the honest version of this claim: adaptation in the probe is not the same as inconsistency in the instrument, provided the instruction set was fixed before fielding. What varies between two Koji interviews is the realisation of a rule, not the rule.\n\n## Frequently asked questions\n\n### What is surveillance bias?\n\nSurveillance bias is the distortion that occurs when the intensity of searching for an event differs between the groups being compared. Because looking harder finds more, the more closely monitored group posts higher event rates regardless of true incidence. Bilimoria and colleagues demonstrated it across 954,926 surgical discharges: hospitals in the highest imaging quartile reported 13.5 clots per 1,000 versus 5.0 in the lowest.\n\n### What is the difference between surveillance bias and lead time bias?\n\nSurveillance bias inflates event counts because you searched harder. Lead time bias inflates survival times because you detected the event earlier without changing the outcome. In the Mayo Lung Project, screening produced median survival of 16.0 years versus 5.0 years for resected early-stage disease while lung cancer mortality was 4.4 versus 3.9 per 1,000 person-years, a non-significant difference.\n\n### How do I know whether my early-warning programme actually works?\n\nAsk whether the number of bad outcomes fell, or whether only the time between detection and the outcome grew. Measure the churn rate of the entire eligible population before and after, not survival from the flag date. The definitive design is a randomised holdout of accounts that are scored but not contacted.\n\n### Can I fix surveillance bias with statistical adjustment?\n\nNot on the outcome alone. The corruption enters before the event is recorded, so adjusting the outcome cannot recover it. The workable fixes are to measure detection intensity explicitly and report it alongside the event rate, to compare only within units whose instrumentation did not change, or to standardise the looking across groups.\n\n### Does surveillance bias affect qualitative research?\n\nYes, and it is commonly missed. When a moderator probes one segment harder than another, more issues surface in the harder-probed segment, and the finding cannot be separated from the method. The defence is a constant instrument: the same brief, the same follow-up logic, and closed question types whose answers do not depend on conversation length.\n\n### Why did better hospitals score worse on the quality measure?\n\nBecause the measure counted detected clots. Higher-quality hospitals had better prevention adherence, 95.5 percent versus 93.3 percent, but also imaged far more, and imaging is what turns a clot into a recorded event. Their risk-adjusted rates were 6.4 versus 4.8 per 1,000. The metric ranked surveillance intensity while appearing to rank quality.\n\n## Related Resources\n\n- [The Stepped Wedge Rollout](/docs/stepped-wedge-rollout-research) - keeping probing depth identical across rollout periods\n- [Structured questions guide](/docs/structured-questions-guide) - the six question types, and which ones are immune to probing depth\n- [Immortal time bias](/docs/immortal-time-bias-retention-analysis) - the mirror problem, where unearned time flatters your feature\n- [The Bradford Hill criteria for product research](/docs/bradford-hill-criteria-product-research) - the framework that names both of these as alternative explanations\n- [Correlation vs causation](/docs/correlation-vs-causation-research) - why metric movements mislead\n- [Survivorship bias in customer research](/docs/survivorship-bias-customer-research) - who you never reach in the first place\n- [Panel conditioning and repeat participants](/docs/panel-conditioning-repeat-participants) - how repeated measurement changes what you measure\n- [Blind analysis in research](/docs/blind-analysis-research) - keeping expectations out of the detection process\n- [Measuring statistical power and minimum detectable effect](/docs/statistical-power-minimum-detectable-effect) - deciding what size of change you can actually see","category":"Research Methods","lastModified":"2026-08-15T03:26:15.112546+00:00","metaTitle":"Surveillance Bias and Lead Time Bias in Product Metrics (2026 Guide)","metaDescription":"Looking harder finds more. Learn how surveillance bias makes well-instrumented teams look worse, how lead time bias fakes success, and the one test that separates them.","keywords":["surveillance bias","lead time bias","detection bias","overdiagnosis","product metrics bias","quality measure validity","early warning programme","research instrument consistency"],"aiSummary":"Surveillance bias occurs when detection intensity differs between compared groups, so the better-instrumented group posts worse event rates. Bilimoria et al 2013 JAMA showed hospitals in the highest imaging quartile reporting 13.5 clots per 1,000 versus 5.0 in the lowest, and higher-quality hospitals scoring worse. Lead time bias inflates survival by detecting earlier without changing outcomes, as in the Mayo Lung Project where median survival was 16.0 versus 5.0 years while mortality was unchanged. Observed event rate equals true incidence times detection intensity times reporting propensity. The test is whether bad outcomes fell or only the detection-to-outcome interval grew.","aiPrerequisites":["Familiarity with product or operational metrics","Basic understanding of rates and denominators"],"aiLearningOutcomes":["Recognise surveillance bias in cross-team metric comparisons","Distinguish lead time bias from a genuine outcome improvement","Apply the incidence times detection times reporting decomposition","Design an early-warning evaluation with a randomised holdout","Hold probing depth constant across segments in qualitative research"],"aiDifficulty":"intermediate","aiEstimatedTime":"12 min"}],"pagination":{"total":1,"returned":1,"offset":0}}