{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-08-14T17:24:02.460Z"},"content":[{"type":"documentation","id":"9ee5166b-66fe-4fc0-a2d6-21981a9abfbc","slug":"healthy-adherer-effect-product-research","title":"The Healthy Adherer Effect: Why Users Who Finish Onboarding Always Retain Better (2026)","url":"https://www.koji.so/docs/healthy-adherer-effect-product-research","summary":"The healthy adherer effect is the tendency for people who comply with a behaviour to have better outcomes for reasons unrelated to the behaviour. In the Coronary Drug Project, placebo adherers had 15.1 percent five-year mortality versus 28.3 percent for non-adherers, and adjustment for 40 baseline characteristics left about 10 points standing. In product research this means onboarding completion, integration setup and invite behaviour are markers of unobserved organisational disposition rather than causes of retention. Adjustment does not fix it; randomising the prompt or interviewing both arms identically does.","content":"**Users who complete your onboarding checklist retain better than users who do not. This is true, it is reproducible, and it is almost never evidence that the checklist works.** The act of completing something is itself a marker of the kind of customer who was going to succeed anyway. Epidemiologists have a name for this and sixty years of evidence about how badly it misleads: the healthy adherer effect.\n\nThe evidence that settles the argument is not subtle. In a randomised trial where one group took a real drug and the other took a sugar pill, the people who faithfully took the sugar pill lived substantially longer than the people who did not. Nothing was in the pill. Adherence was measuring the person, not the treatment.\n\nThis article covers what the effect is, the trial that proved it, why statistical adjustment does not remove it, how it differs from the sampling biases you already know about, and the one method that actually resolves it.\n\n## The claim your dashboard is really making\n\nAlmost every growth team has shipped a version of this sentence:\n\n- Accounts that connect an integration in week one churn 40 percent less.\n- Users who complete the onboarding checklist are 3x more likely to still be active at day 90.\n- Teams that invite three or more colleagues have double the retention.\n- Customers who attend onboarding training renew at a higher rate.\n\nEach sentence has the same grammar. It compares people who did a thing against people who did not, and it treats the doing as the cause. But nobody assigned the behaviour. Customers sorted themselves, and whatever made them sort themselves is still in the data, unmeasured, doing work you are attributing to your feature.\n\n## The sugar pill result\n\nThe Coronary Drug Project was a large randomised trial of lipid-lowering drugs in men who had survived a heart attack. In 1980 its investigators published an analysis that has been unsettling trialists ever since (Coronary Drug Project Research Group, *New England Journal of Medicine* 1980;303(18):1038-1041).\n\nThey split each arm by adherence, using a threshold of taking at least 80 percent of the assigned dose, and compared five-year mortality.\n\n| Arm | Good adherers | Poor adherers | Significance |\n| --- | --- | --- | --- |\n| Clofibrate (active drug) | 15.0% | 24.6% | P = 0.00011 |\n| Placebo (sugar pill) | 15.1% | 28.3% | P = 4.7 x 10^-16 |\n\nRead the second row again. Among men taking a pill with no active ingredient, the diligent takers died at roughly half the rate of the non-diligent ones, and the result was overwhelmingly significant. The placebo arm alone had 1,813 adherers and 882 non-adherers, so this is not a small-sample curiosity.\n\nThe investigators then did what any careful analyst would do. They adjusted for 40 baseline characteristics. The gap shrank from about 13 percentage points to about 10. It did not go away.\n\nTheir conclusion is the sentence to remember, because it generalises perfectly to product data: adherers and non-adherers **must be different in ways not accounted for by the available data, because greater adherence to a placebo is not expected to have a causal effect on mortality risk.**\n\nThe pattern replicates. A meta-analysis by Simpson and colleagues in the *BMJ* in 2006 pooled 21 studies covering 46,847 participants, including eight studies with placebo arms covering 19,633 participants. Good adherence to placebo was associated with an odds ratio for mortality of 0.56 (95% CI 0.43 to 0.74). Good adherence to a genuinely beneficial drug gave 0.55 (95% CI 0.49 to 0.62). **The sugar pill and the real medicine produced the same adherence benefit, which is the cleanest possible demonstration that the benefit belongs to the adherer and not to the pill.** Those authors named it the healthy adherer effect and described adherence as a surrogate marker for overall healthy behaviour.\n\n## Your onboarding checklist is the sugar pill\n\nThe translation into product research is direct. Completing an onboarding checklist, connecting an integration, inviting teammates and attending training are all adherence behaviours. They are things a customer does when they have already decided the product matters, when someone internally owns the rollout, when the budget is secure and when the team has the slack to invest a Tuesday afternoon.\n\nEvery one of those conditions independently predicts retention. None of them is in your event data.\n\n| The claim | The adherer explanation you have not ruled out |\n| --- | --- |\n| The checklist drives retention | Accounts with an internal owner finish checklists and also renew |\n| Integrations cause stickiness | Teams with engineering capacity connect integrations and also survive reorgs |\n| Inviting teammates increases retention | Products already chosen as a team standard get invitations |\n| Training attendance improves renewal | Companies that fund training have budget and executive sponsorship |\n| Reading the docs predicts expansion | Motivated power users read docs and were already expanding |\n\nThe practical consequence is expensive. A team that believes the checklist is causal will spend a quarter nagging the accounts that did not finish it. Those accounts are not one checklist away from retaining. They are missing the sponsor, the budget or the capacity that would have produced both the completion and the renewal, and prompting them to tick boxes changes the marker without touching the thing the marker was standing in for.\n\n## Why adjusting for covariates does not save you\n\nThe instinct is to control for company size, plan tier, industry and seat count, then claim the residual effect is causal. The Coronary Drug Project investigators had richer baseline data than your CRM and adjusted for 40 characteristics, and roughly three quarters of the gap survived.\n\nThe reason is structural rather than statistical. Adjustment can only remove variation in things you measured. The disposition that drives adherence is a bundle of internal facts about the customer organisation, including whether a champion still works there, whether a competing migration is consuming the team, and whether anyone senior has staked credibility on the purchase. These are not columns. Adding more of the columns you happen to have does not approximate them, and a model that reports a smaller coefficient after adjustment has not demonstrated that the remainder is causal.\n\nThere is a second trap here worth naming, because the fix for one bias can create another. Conditioning on variables that sit downstream of both the behaviour and the outcome makes matters actively worse, not merely incomplete. That mechanism has its own article: see [collider bias](/docs/collider-bias-product-research).\n\n## How this differs from biases you already track\n\nThe healthy adherer effect is frequently mislabelled as selection bias, which obscures what makes it hard. The distinction that matters is **who is missing from your data versus who chose the behaviour inside your data.**\n\n| Bias | Who is affected | What went wrong | Does better recruiting fix it? |\n| --- | --- | --- | --- |\n| [Sampling and self-selection bias](/docs/sampling-bias-research) | People who answer your research | The respondents differ from the population | Yes, largely |\n| [Survivorship bias](/docs/survivorship-bias-customer-research) | People you never reach | Churned accounts are absent from the frame | Yes, reach them |\n| [Nonresponse bias](/docs/nonresponse-bias) | People who ignore the invitation | Responders differ from non-responders | Yes, partly |\n| [Immortal time bias](/docs/immortal-time-bias-retention-analysis) | Everyone you observed | Follow-up time was misallocated to the adopter group | No, it is a time-alignment error |\n| **Healthy adherer effect** | Everyone you observed | The behaviour is a marker of unobserved disposition | **No, it is a confounding error** |\n\nThe first three are problems of coverage, and a better frame or a cheaper channel fixes them. The last two are not. In immortal time bias you already have all the people and you misallocated their time. In the healthy adherer effect you already have all the people, the time is correct, and the group definition itself encodes the answer. **No amount of additional respondents helps, because the people you add will sort themselves the same way.**\n\n## What actually resolves it\n\nThere are exactly two honest routes, and they sit at different costs.\n\n**Route one: randomise the behaviour rather than observing it.** If the checklist is assigned rather than chosen, the disposition that drove completion is balanced across arms by construction. In practice you rarely get to randomise whether a customer completes something, but you can very often randomise the *prompt*: half of new accounts get the checklist surfaced prominently, half do not, and you compare all of the prompted group against all of the unprompted group regardless of who finished. That comparison is causal about the prompt, which is the only lever you actually control. When user-level randomisation is impossible because the change ships to everyone, [the stepped wedge rollout](/docs/stepped-wedge-rollout-research) recovers a randomised comparison from the rollout schedule instead.\n\n**Route two: ask what the marker was standing in for.** The adherer explanation is a claim about unobserved disposition, and disposition is precisely the thing event data cannot hold and a conversation can. This is not a soft supplement to the analysis. It is the only instrument that reaches the variable.\n\nThe design that works is narrow and specific. Interview **both** groups, ask them **the same questions**, and probe the conditions that existed before the behaviour rather than the behaviour itself:\n\n- Who inside the company decided this tool would be used, and are they still here?\n- What else was competing for the team calendar in the first month?\n- Was there a budget or headcount decision attached to this purchase?\n- What would have had to be true for you to finish setup in week one?\n- What happened in the two weeks before you stopped setting it up?\n\nIf the non-completers systematically report a departed champion, a competing migration or an absent sponsor, you have found your confounder and the checklist is exonerated as a cause. If they report that they tried and the setup was confusing, you have a genuine product problem and the completion metric was, in that instance, telling the truth. **Only the interview distinguishes those two worlds, and they call for opposite roadmaps.**\n\n## The modern approach: making the non-adopter arm affordable\n\nThe reason teams skip this is arithmetic rather than ignorance. The design demands interviews with the group nobody wants to talk to and nobody has a relationship with, asked identically to the group that loves you. Traditionally that means recruiting two matched samples, scheduling across time zones and moderating twenty conversations, which is two to three weeks of research capacity for one dashboard claim.\n\nKoji changes the cost structure of exactly that arm:\n\n- **AI-moderated interviews run both arms in parallel**, so the non-completers are reached in the same window as the completers rather than being dropped when the deadline arrives.\n- **Identical moderation across groups.** A human moderator naturally probes the enthusiastic adopter harder, which manufactures a depth difference that looks like a finding. An AI moderator runs the same brief and the same follow-up logic in both arms, which is what makes the comparison legitimate. The same argument applies to [surveillance bias](/docs/surveillance-bias-detection-research).\n- **Voice interviews reach operators who will never book a Zoom**, which is disproportionately the non-completer population.\n- **Automatic thematic analysis** surfaces whether the reasons for non-completion cluster on capacity and sponsorship (confounding) or on product friction (a real defect).\n- **Customizable AI consultants** can be briefed to press specifically on pre-behaviour conditions rather than on satisfaction.\n- **Real-time reporting** means the confounder check lands before the roadmap is committed rather than after.\n\nThe structured question layer matters more here than usual. Koji supports six question types: `open_ended`, `scale`, `single_choice`, `multiple_choice`, `ranking` and `yes_no`. For an adherer investigation the closed types are the load-bearing ones, because they mean the same thing regardless of how long the conversation ran. A `yes_no` on whether an executive sponsor was named, a `single_choice` on who owned the rollout, and a `scale` on how much competing work the team faced give you comparable rates across completers and non-completers. If those closed measures differ sharply between the groups while your open-ended themes look similar, **the disposition gap is real and your checklist finding is confounded.** See the [structured questions guide](/docs/structured-questions-guide) for how to build that instrument.\n\n## A working checklist\n\n1. Write the causal sentence your metric implies, out loud, in full.\n2. Ask whether anyone assigned the behaviour. If not, treat the comparison as a hypothesis rather than a finding.\n3. List the dispositions that could produce both the behaviour and the outcome. Sponsorship, capacity, budget, prior commitment.\n4. Check whether any of them are columns you hold. Usually none are.\n5. Do not adjust and declare victory. Note that 40 covariates left 10 points standing in the CDP.\n6. Randomise the prompt if you can, at user level or through the rollout order.\n7. Interview both arms identically, probing pre-behaviour conditions.\n8. Report the finding with the disposition explanation named and addressed, or not at all.\n\n## Frequently asked questions\n\n### What is the healthy adherer effect?\n\nIt is the tendency for people who comply with a treatment or a recommended behaviour to have better outcomes than non-compliers for reasons unrelated to the treatment itself. Adherence acts as a marker of an underlying disposition, such as general conscientiousness or, in a business context, organisational commitment and capacity. The definitive demonstration is that adherence to a placebo predicts survival about as strongly as adherence to an active drug.\n\n### Does this mean onboarding checklists are useless?\n\nNo, and this is the most common misreading. The effect means your observational comparison cannot tell you whether the checklist works. The checklist may well be valuable. The point is that the retention gap between completers and non-completers is not the measurement of that value, and it will substantially overstate it. To measure the real effect, randomise who gets prompted and compare everyone who was prompted against everyone who was not.\n\n### Can I fix it by controlling for company size and plan tier?\n\nNot reliably. Adjustment removes only the variation you measured, and the drivers of adherence are mostly unmeasured organisational facts. The Coronary Drug Project investigators adjusted for 40 baseline characteristics and about three quarters of the mortality gap survived. Your CRM holds far fewer and far weaker covariates than that trial did.\n\n### How is this different from survivorship bias?\n\nSurvivorship bias is about who is missing from your data, typically the churned accounts you never interviewed. The healthy adherer effect operates on a complete dataset where nobody is missing. Everyone is present and correctly measured, and the problem is that membership of the adopter group was chosen by the customers according to an unobserved disposition. Better reach fixes the first and does nothing for the second.\n\n### How is it different from immortal time bias?\n\nBoth inflate the apparent benefit of adoption, but through different machinery. Immortal time bias is a time-alignment error in which the adopter group is credited with follow-up time during which it was impossible for them to churn. The healthy adherer effect is a confounding error in which the adopter group is composed of systematically different customers. A study can suffer from both at once, and the fixes are unrelated: landmark analysis for the first, randomisation or interviews for the second.\n\n### What is the fastest way to check whether my adoption finding is confounded?\n\nRun a short identical study across adopters and non-adopters that asks about the conditions preceding adoption rather than about the product. Use closed question types so the rates are comparable between the groups. If sponsorship, ownership and available capacity differ sharply between them, your adoption metric is largely reading disposition. This takes days with an AI-moderated study rather than the weeks required to recruit and moderate two matched samples by hand.\n\n## Related Resources\n\n- [Immortal Time Bias](/docs/immortal-time-bias-retention-analysis) - the time-alignment error that inflates the same adoption charts\n- [Collider Bias](/docs/collider-bias-product-research) - why adding a control variable can create the correlation\n- [The Stepped Wedge Rollout](/docs/stepped-wedge-rollout-research) - recovering a randomised comparison from your release schedule\n- [Sampling Bias](/docs/sampling-bias-research) - the coverage problem this is often confused with\n- [Survivorship Bias in Customer Research](/docs/survivorship-bias-customer-research) - who is missing from your data\n- [Quasi-Experimental Design](/docs/quasi-experimental-design-guide) - estimating effects when assignment was not random\n- [Structured Questions Guide](/docs/structured-questions-guide) - building the comparable closed measures this method needs\n\n**Test your next adoption claim before you build a quarter on it.** Koji gives you 10 free interview credits, which is enough to ask your non-adopters what was actually happening when they stopped.","category":"Research Methods","lastModified":"2026-08-14T03:22:39.89955+00:00","metaTitle":"The Healthy Adherer Effect in Product Research (2026)","metaDescription":"Users who finish onboarding retain better - and so do users who adhere to a placebo. Why adoption metrics overstate feature impact, why covariate adjustment fails, and what to do instead.","keywords":["healthy adherer effect","adherence bias","onboarding completion retention","feature adoption causation","healthy user effect","confounding in product analytics"],"aiSummary":"The healthy adherer effect is the tendency for people who comply with a behaviour to have better outcomes for reasons unrelated to the behaviour. In the Coronary Drug Project, placebo adherers had 15.1 percent five-year mortality versus 28.3 percent for non-adherers, and adjustment for 40 baseline characteristics left about 10 points standing. In product research this means onboarding completion, integration setup and invite behaviour are markers of unobserved organisational disposition rather than causes of retention. Adjustment does not fix it; randomising the prompt or interviewing both arms identically does.","aiPrerequisites":["Basic familiarity with retention and cohort metrics","Understanding of correlation versus causation"],"aiLearningOutcomes":["Recognise adherence-marker claims in your own adoption dashboards","Explain why covariate adjustment leaves most of the bias in place","Distinguish the healthy adherer effect from sampling, survivorship and immortal time bias","Design a two-arm interview that tests the disposition explanation"],"aiDifficulty":"intermediate","aiEstimatedTime":"12 min read"}],"pagination":{"total":1,"returned":1,"offset":0}}