{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-08-14T17:24:08.178Z"},"content":[{"type":"documentation","id":"ff098596-5c20-4885-afe1-40e07221873c","slug":"collider-bias-product-research","title":"Collider Bias: When Adding a Control Variable Creates the Correlation (2026)","url":"https://www.koji.so/docs/collider-bias-product-research","summary":"A collider is a variable influenced by two other variables. Conditioning on it induces a correlation between those variables inside the restricted sample even when they are independent in the population. Sackett showed relative odds of 4.06 in a hospital sample versus 1.06 in the general population. Griffith and colleagues showed 32 percent of 2,556 phenotypes were associated with merely being tested for COVID-19 in UK Biobank, explaining the implausible protective effect of smoking. In product research the common form is analysing only churned accounts, only survey responders, or only users who upgraded. Sample size does not help; only changing how the sample is drawn does.","content":"**Every other article about research bias tells you the same thing: you failed to account for something, so add a control, match your groups, or segment more carefully. Collider bias is the exact inverse. It is the case where adding the control is what creates the false finding.** Filtering to a subgroup, matching on a variable, or simply analysing the users who ended up in your dataset can manufacture a strong, statistically significant, entirely fictional relationship between two things that are unrelated in the real population.\n\nThis matters more than it sounds, because conditioning is the most common operation in product analytics. Every time you write \"among users who upgraded\", \"among accounts that responded\", or \"restricting to active users\", you are conditioning. Most of the time it is harmless. When the variable you conditioned on is caused by both of the things you are comparing, it is not.\n\n## The mechanism in one paragraph\n\nA collider is a variable influenced by two other variables. Draw the arrows and they collide on it: A points at C, B points at C. In the full population A and B may be completely independent. **But once you restrict your analysis to a particular value of C, A and B become correlated inside that restricted group, and the correlation is an artefact of your own filter.**\n\nThe intuition is easier with a concrete rule. Suppose a nightclub admits anyone who is either an excellent dancer or extremely charming. Among the general public those two traits are unrelated. Inside the club they are strongly negatively correlated, because a poor dancer who got in must be charming. You did not discover anything about dancing and charm. You discovered your own door policy.\n\n## The classic demonstration\n\nThe oldest version in the literature is hospital admission bias, described by Joseph Berkson in 1946 (*Limitations of the application of fourfold table analysis to hospital data*, Biometrics 1946;2(3):47-53). The Catalog of Bias defines it as a distortion in the assessed relation between an exposure and a disease \"which arises because the subjects studied had been admitted to hospital\".\n\nDavid Sackett quantified it in 1979 with a comparison that is still the cleanest teaching case available:\n\n| Population studied | Relative odds, respiratory vs locomotor disease |\n| --- | --- |\n| Hospital sample | 4.06 |\n| General population | 1.06 |\n\nIn the general population there is essentially no association. In the hospital sample the association is fourfold and would pass any significance test you cared to run. The explanation is that having either condition raises your chance of admission, and having both raises it further, so the admitted group is enriched for people with both. **The entire effect was produced by the selection rule, and no amount of extra hospital patients would have corrected it.**\n\n## The modern demonstration, at scale\n\nThe most instructive recent example concerns COVID-19. Early in the pandemic several studies reported that smoking appeared to *protect* against infection, a finding that was biologically implausible and widely circulated anyway.\n\nGriffith and colleagues showed why, in *Nature Communications* (2020;11:5749). Analysing UK Biobank, they compared the 486,967 participants in the wider cohort against the 1,410 who had been tested for COVID-19 as of April 2020. Of 2,556 phenotypes examined for association with simply *being tested*, **811, or 32 percent, were significantly associated with test status.** The tested sample was heavily selected on genetic, behavioural, cardiovascular, demographic and anthropometric traits.\n\nThat selection was the collider. Healthcare workers were tested at high rates regardless of severity, while among non-workers mainly severe cases were tested. Conditioning on having been tested induced associations between traits that were independent in the population, and smoking picked up a spurious protective signal. As the authors put it, if the factors that influence sample selection themselves influence the variables of interest, the relationship between those variables can become distorted.\n\n**Note what makes this frightening rather than merely interesting: the sample was enormous, the analysis was competent, and the result was confidently wrong in a direction that had public health consequences.** Sample size is not a defence. A larger conditioned sample estimates the artefact more precisely.\n\n## Where this happens in product research\n\nProduct teams condition constantly, usually without noticing it is an analytical choice.\n\n| The analysis | What you conditioned on | Why it is a collider |\n| --- | --- | --- |\n| \"Among accounts that upgraded, heavy API use predicts churn\" | Upgrading | Both API use and churn risk drive upgrade decisions |\n| \"Among survey responders, satisfaction is unrelated to tenure\" | Responding | Both satisfaction and tenure drive willingness to respond |\n| \"Among churned accounts, price complaints and bugs are inversely related\" | Churning | Either complaint alone can cause churn, so churned accounts with one tend to lack the other |\n| \"Among users who booked an interview, feature interest is unrelated to seniority\" | Booking | Both interest and available time drive booking |\n| \"Among support ticket filers, product knowledge predicts satisfaction\" | Filing a ticket | Both a problem and a propensity to report drive filing |\n\nThe third row is the one that catches experienced teams. **Analysing only churned accounts to find out what causes churn is conditioning on the outcome**, and it reliably produces inverse relationships between the various causes of churn. If either a pricing problem or a reliability problem is enough to make an account leave, then among the accounts that left, the ones with pricing problems will disproportionately *not* have reliability problems. You will conclude the two are alternatives when they are simply independent routes to the same exit.\n\nThat is also why exit interviews, valuable as they are for understanding mechanism, cannot rank causes of churn on their own. The [case-control design](/docs/case-control-research-churn-lost-deals) exists precisely to solve this by drawing controls from the same source population rather than reasoning inside the churned group alone.\n\n## The rule, and why the usual advice inverts here\n\nThe operative rule is short:\n\n> **Never condition on a variable that both your suspected cause and your outcome influence. If you must, you no longer have an estimate of the relationship you set out to measure.**\n\nThis puts collider bias in direct tension with the standard advice about confounding. Confounders are common *causes* and you should adjust for them. Colliders are common *effects* and adjusting for them introduces bias where none existed. The two look identical in a regression table. **Nothing in your output distinguishes a helpful control from a harmful one, which is why the decision has to be made from a diagram of what causes what, before the model is fitted, not from model fit afterwards.**\n\nThree practical tests before adding a variable or a filter:\n\n1. **Does the outcome influence this variable?** If churn influences whether an account appears in your dataset, that filter is a collider. Timing gives it away: anything measured after the outcome began is suspect.\n2. **Does my suspected cause also influence it?** If both arrows point at it, stop.\n3. **Would this filter change who is in the analysis, or only which rows I display?** Cosmetic filters are safe. Filters that determine membership are not.\n\nA fourth check is worth running on any surprising result: **if the finding is counterintuitive and appears only inside a subgroup, suspect the subgroup definition before you believe the finding.** The smoking result and the Sackett result were both discoveries about a selection rule dressed as discoveries about the world.\n\n## What you can do about it\n\nCollider bias cannot be removed after the fact by a smarter model, because the information required is not in the conditioned sample. The remedies are all upstream, at the point where the sample is defined.\n\n- **Define the population before the filter.** Decide who the claim is about, then check whether your dataset represents that population or a selected slice of it. This is the same discipline as [defining the survey universe](/docs/survey-universe-definition).\n- **Sample the unconditioned frame.** If the question is what causes churn, you need retained accounts too. If the question is about all users, you need the ones who never respond.\n- **Prefer randomised assignment when it is available**, since randomisation makes the exposure independent of everything, including the drivers of selection. When the change ships to everyone and user-level randomisation seems impossible, [the stepped wedge rollout](/docs/stepped-wedge-rollout-research) recovers a randomised comparison from the release order.\n- **Treat any \"among X\" headline as provisional** until someone has drawn the arrows.\n\n## The modern approach: reaching the unconditioned population\n\nThe reason teams condition is almost never methodological conviction. It is that the conditioned group is the reachable one. Churned accounts who agree to an exit call, users who answer surveys, and customers who book interviews are all defined by their willingness to engage, and building a study around them is simply cheaper than building one around the population you actually want.\n\nThat cost asymmetry is what Koji changes:\n\n- **AI-moderated interviews run at a cost per conversation that makes the comparison group affordable**, so you can talk to retained accounts as well as churned ones, and non-responders as well as responders.\n- **Voice interviews and asynchronous scheduling** reach the people who never book a calendar slot, which is exactly the group that conditioning silently excludes.\n- **Automatic thematic analysis** lets you run both arms at full size instead of shrinking the comparison group to whatever the analyst has time to read.\n- **Real-time reporting** puts the comparison in front of the team while the decision is still open.\n- **Customizable AI consultants** can be briefed to run an identical protocol across both arms, which is what makes the comparison interpretable.\n\nThe six structured question types matter for a specific reason here. Koji supports `open_ended`, `scale`, `single_choice`, `multiple_choice`, `ranking` and `yes_no`. Because the closed types produce the same measurement regardless of who is answering or how long they talk, they give you a way to **compare your conditioned group against the unconditioned frame directly.** Field the same `scale` and `single_choice` items to survey responders and to a sample of non-responders, and the difference between them is a measurement of your selection effect rather than a guess about it. The [structured questions guide](/docs/structured-questions-guide) covers how to build that instrument.\n\n## Frequently asked questions\n\n### What is collider bias in plain terms?\n\nA collider is a variable that two other variables both influence. When you restrict your analysis to people who share a particular value of that variable, the two influencing variables become correlated within your restricted group even if they are unrelated in the wider population. The correlation is created by your selection rule rather than discovered in the world.\n\n### How is a collider different from a confounder?\n\nA confounder is a common cause of both your exposure and your outcome, and you should adjust for it. A collider is a common effect of both, and adjusting for it introduces bias rather than removing it. They are indistinguishable in a regression output, so the decision must come from reasoning about causal direction before you fit the model.\n\n### Does a bigger sample help?\n\nNo, and this is the most dangerous misconception. Conditioning bias does not shrink with sample size. A larger conditioned sample simply estimates the artefact more precisely and with a narrower confidence interval. The UK Biobank example involved hundreds of thousands of participants and still produced an implausible protective effect for smoking.\n\n### Is analysing only churned customers always wrong?\n\nNot always, but it cannot support claims about what causes churn or about the relative importance of different causes. Interviewing churned accounts is excellent for understanding mechanism and for generating hypotheses about how a problem unfolds. Ranking causes requires a comparison group of accounts that did not churn, drawn from the same source population.\n\n### How do I know whether I have conditioned on a collider?\n\nAsk whether the outcome you are studying could influence membership of the group you are analysing. If churn affects whether an account is in the dataset, or if satisfaction affects whether someone answered, you have conditioned on something downstream of the outcome. Timing is the practical tell: any filter based on something that happened after the outcome started is a candidate.\n\n### Can I fix collider bias in the analysis?\n\nGenerally no. The information needed to undo the selection is missing from the conditioned sample, which is why the problem is structural rather than statistical. Methods exist to bound or correct it when you have good external data on the selection process, but the reliable fix is to change how the sample is drawn, not how it is modelled.\n\n## Related Resources\n\n- [The Healthy Adherer Effect](/docs/healthy-adherer-effect-product-research) - the confounding problem this one inverts\n- [The Stepped Wedge Rollout](/docs/stepped-wedge-rollout-research) - buying a randomised comparison instead of repairing one\n- [Case-Control Research for Churn](/docs/case-control-research-churn-lost-deals) - why controls must come from the same source population\n- [Sampling Bias](/docs/sampling-bias-research) - the broader family of selection problems\n- [Survey Universe Definition](/docs/survey-universe-definition) - deciding who the claim is about before you filter\n- [Correlation vs Causation](/docs/correlation-vs-causation-research) - the foundations this builds on\n- [Structured Questions Guide](/docs/structured-questions-guide) - comparable closed measures across conditioned and unconditioned groups\n\n**Check your next \"among users who...\" finding against the unconditioned population.** Koji gives you 10 free interview credits, which is enough to field the same instrument to the group your filter left out.","category":"Research Methods","lastModified":"2026-08-14T03:22:39.89955+00:00","metaTitle":"Collider Bias in Product Research: When Controls Create Correlations","metaDescription":"Collider bias is the case where adding a control variable, filtering to a subgroup, or analysing only churned users manufactures a correlation that does not exist. How to spot it before it reaches a roadmap.","keywords":["collider bias","conditioning on a collider","Berkson bias","admission rate bias","selection bias mechanism","confounder vs collider","causal diagram product analytics"],"aiSummary":"A collider is a variable influenced by two other variables. Conditioning on it induces a correlation between those variables inside the restricted sample even when they are independent in the population. Sackett showed relative odds of 4.06 in a hospital sample versus 1.06 in the general population. Griffith and colleagues showed 32 percent of 2,556 phenotypes were associated with merely being tested for COVID-19 in UK Biobank, explaining the implausible protective effect of smoking. In product research the common form is analysing only churned accounts, only survey responders, or only users who upgraded. Sample size does not help; only changing how the sample is drawn does.","aiPrerequisites":["Understanding of correlation versus causation","Familiarity with segmentation and cohort filtering"],"aiLearningOutcomes":["Distinguish a collider from a confounder before fitting a model","Recognise the common conditioning traps in product analytics","Explain why larger samples do not reduce conditioning bias","Redesign a study to sample the unconditioned population"],"aiDifficulty":"advanced","aiEstimatedTime":"12 min read"}],"pagination":{"total":1,"returned":1,"offset":0}}