The Healthy Adherer Effect: Why Users Who Finish Onboarding Always Retain Better (2026)

Users who complete your onboarding checklist retain better than users who do not. This is true, it is reproducible, and it is almost never evidence that the checklist works. The act of completing something is itself a marker of the kind of customer who was going to succeed anyway. Epidemiologists have a name for this and sixty years of evidence about how badly it misleads: the healthy adherer effect.

The evidence that settles the argument is not subtle. In a randomised trial where one group took a real drug and the other took a sugar pill, the people who faithfully took the sugar pill lived substantially longer than the people who did not. Nothing was in the pill. Adherence was measuring the person, not the treatment.

This article covers what the effect is, the trial that proved it, why statistical adjustment does not remove it, how it differs from the sampling biases you already know about, and the one method that actually resolves it.

The claim your dashboard is really making

Almost every growth team has shipped a version of this sentence:

• Accounts that connect an integration in week one churn 40 percent less.
• Users who complete the onboarding checklist are 3x more likely to still be active at day 90.
• Teams that invite three or more colleagues have double the retention.
• Customers who attend onboarding training renew at a higher rate.

Each sentence has the same grammar. It compares people who did a thing against people who did not, and it treats the doing as the cause. But nobody assigned the behaviour. Customers sorted themselves, and whatever made them sort themselves is still in the data, unmeasured, doing work you are attributing to your feature.

The sugar pill result

The Coronary Drug Project was a large randomised trial of lipid-lowering drugs in men who had survived a heart attack. In 1980 its investigators published an analysis that has been unsettling trialists ever since (Coronary Drug Project Research Group, New England Journal of Medicine 1980;303(18):1038-1041).

They split each arm by adherence, using a threshold of taking at least 80 percent of the assigned dose, and compared five-year mortality.

Read the second row again. Among men taking a pill with no active ingredient, the diligent takers died at roughly half the rate of the non-diligent ones, and the result was overwhelmingly significant. The placebo arm alone had 1,813 adherers and 882 non-adherers, so this is not a small-sample curiosity.

The investigators then did what any careful analyst would do. They adjusted for 40 baseline characteristics. The gap shrank from about 13 percentage points to about 10. It did not go away.

Their conclusion is the sentence to remember, because it generalises perfectly to product data: adherers and non-adherers must be different in ways not accounted for by the available data, because greater adherence to a placebo is not expected to have a causal effect on mortality risk.

The pattern replicates. A meta-analysis by Simpson and colleagues in the BMJ in 2006 pooled 21 studies covering 46,847 participants, including eight studies with placebo arms covering 19,633 participants. Good adherence to placebo was associated with an odds ratio for mortality of 0.56 (95% CI 0.43 to 0.74). Good adherence to a genuinely beneficial drug gave 0.55 (95% CI 0.49 to 0.62). The sugar pill and the real medicine produced the same adherence benefit, which is the cleanest possible demonstration that the benefit belongs to the adherer and not to the pill. Those authors named it the healthy adherer effect and described adherence as a surrogate marker for overall healthy behaviour.

Your onboarding checklist is the sugar pill

The translation into product research is direct. Completing an onboarding checklist, connecting an integration, inviting teammates and attending training are all adherence behaviours. They are things a customer does when they have already decided the product matters, when someone internally owns the rollout, when the budget is secure and when the team has the slack to invest a Tuesday afternoon.

Every one of those conditions independently predicts retention. None of them is in your event data.

The practical consequence is expensive. A team that believes the checklist is causal will spend a quarter nagging the accounts that did not finish it. Those accounts are not one checklist away from retaining. They are missing the sponsor, the budget or the capacity that would have produced both the completion and the renewal, and prompting them to tick boxes changes the marker without touching the thing the marker was standing in for.

Why adjusting for covariates does not save you

The instinct is to control for company size, plan tier, industry and seat count, then claim the residual effect is causal. The Coronary Drug Project investigators had richer baseline data than your CRM and adjusted for 40 characteristics, and roughly three quarters of the gap survived.

The reason is structural rather than statistical. Adjustment can only remove variation in things you measured. The disposition that drives adherence is a bundle of internal facts about the customer organisation, including whether a champion still works there, whether a competing migration is consuming the team, and whether anyone senior has staked credibility on the purchase. These are not columns. Adding more of the columns you happen to have does not approximate them, and a model that reports a smaller coefficient after adjustment has not demonstrated that the remainder is causal.

There is a second trap here worth naming, because the fix for one bias can create another. Conditioning on variables that sit downstream of both the behaviour and the outcome makes matters actively worse, not merely incomplete. That mechanism has its own article: see collider bias.

How this differs from biases you already track

The healthy adherer effect is frequently mislabelled as selection bias, which obscures what makes it hard. The distinction that matters is who is missing from your data versus who chose the behaviour inside your data.

The first three are problems of coverage, and a better frame or a cheaper channel fixes them. The last two are not. In immortal time bias you already have all the people and you misallocated their time. In the healthy adherer effect you already have all the people, the time is correct, and the group definition itself encodes the answer. No amount of additional respondents helps, because the people you add will sort themselves the same way.

What actually resolves it

There are exactly two honest routes, and they sit at different costs.

Route one: randomise the behaviour rather than observing it. If the checklist is assigned rather than chosen, the disposition that drove completion is balanced across arms by construction. In practice you rarely get to randomise whether a customer completes something, but you can very often randomise the prompt: half of new accounts get the checklist surfaced prominently, half do not, and you compare all of the prompted group against all of the unprompted group regardless of who finished. That comparison is causal about the prompt, which is the only lever you actually control. When user-level randomisation is impossible because the change ships to everyone, the stepped wedge rollout recovers a randomised comparison from the rollout schedule instead.

Route two: ask what the marker was standing in for. The adherer explanation is a claim about unobserved disposition, and disposition is precisely the thing event data cannot hold and a conversation can. This is not a soft supplement to the analysis. It is the only instrument that reaches the variable.

The design that works is narrow and specific. Interview both groups, ask them the same questions, and probe the conditions that existed before the behaviour rather than the behaviour itself:

• Who inside the company decided this tool would be used, and are they still here?
• What else was competing for the team calendar in the first month?
• Was there a budget or headcount decision attached to this purchase?
• What would have had to be true for you to finish setup in week one?
• What happened in the two weeks before you stopped setting it up?

If the non-completers systematically report a departed champion, a competing migration or an absent sponsor, you have found your confounder and the checklist is exonerated as a cause. If they report that they tried and the setup was confusing, you have a genuine product problem and the completion metric was, in that instance, telling the truth. Only the interview distinguishes those two worlds, and they call for opposite roadmaps.

The modern approach: making the non-adopter arm affordable

The reason teams skip this is arithmetic rather than ignorance. The design demands interviews with the group nobody wants to talk to and nobody has a relationship with, asked identically to the group that loves you. Traditionally that means recruiting two matched samples, scheduling across time zones and moderating twenty conversations, which is two to three weeks of research capacity for one dashboard claim.

Koji changes the cost structure of exactly that arm:

• AI-moderated interviews run both arms in parallel, so the non-completers are reached in the same window as the completers rather than being dropped when the deadline arrives.
• Identical moderation across groups. A human moderator naturally probes the enthusiastic adopter harder, which manufactures a depth difference that looks like a finding. An AI moderator runs the same brief and the same follow-up logic in both arms, which is what makes the comparison legitimate. The same argument applies to surveillance bias.
• Voice interviews reach operators who will never book a Zoom, which is disproportionately the non-completer population.
• Automatic thematic analysis surfaces whether the reasons for non-completion cluster on capacity and sponsorship (confounding) or on product friction (a real defect).
• Customizable AI consultants can be briefed to press specifically on pre-behaviour conditions rather than on satisfaction.
• Real-time reporting means the confounder check lands before the roadmap is committed rather than after.

The structured question layer matters more here than usual. Koji supports six question types: open_ended, scale, single_choice, multiple_choice, ranking and yes_no. For an adherer investigation the closed types are the load-bearing ones, because they mean the same thing regardless of how long the conversation ran. A yes_no on whether an executive sponsor was named, a single_choice on who owned the rollout, and a scale on how much competing work the team faced give you comparable rates across completers and non-completers. If those closed measures differ sharply between the groups while your open-ended themes look similar, the disposition gap is real and your checklist finding is confounded. See the structured questions guide for how to build that instrument.

A working checklist

1. Write the causal sentence your metric implies, out loud, in full.
2. Ask whether anyone assigned the behaviour. If not, treat the comparison as a hypothesis rather than a finding.
3. List the dispositions that could produce both the behaviour and the outcome. Sponsorship, capacity, budget, prior commitment.
4. Check whether any of them are columns you hold. Usually none are.
5. Do not adjust and declare victory. Note that 40 covariates left 10 points standing in the CDP.
6. Randomise the prompt if you can, at user level or through the rollout order.
7. Interview both arms identically, probing pre-behaviour conditions.
8. Report the finding with the disposition explanation named and addressed, or not at all.

Frequently asked questions

What is the healthy adherer effect?

It is the tendency for people who comply with a treatment or a recommended behaviour to have better outcomes than non-compliers for reasons unrelated to the treatment itself. Adherence acts as a marker of an underlying disposition, such as general conscientiousness or, in a business context, organisational commitment and capacity. The definitive demonstration is that adherence to a placebo predicts survival about as strongly as adherence to an active drug.

Does this mean onboarding checklists are useless?

No, and this is the most common misreading. The effect means your observational comparison cannot tell you whether the checklist works. The checklist may well be valuable. The point is that the retention gap between completers and non-completers is not the measurement of that value, and it will substantially overstate it. To measure the real effect, randomise who gets prompted and compare everyone who was prompted against everyone who was not.

Can I fix it by controlling for company size and plan tier?

Not reliably. Adjustment removes only the variation you measured, and the drivers of adherence are mostly unmeasured organisational facts. The Coronary Drug Project investigators adjusted for 40 baseline characteristics and about three quarters of the mortality gap survived. Your CRM holds far fewer and far weaker covariates than that trial did.

How is this different from survivorship bias?

Survivorship bias is about who is missing from your data, typically the churned accounts you never interviewed. The healthy adherer effect operates on a complete dataset where nobody is missing. Everyone is present and correctly measured, and the problem is that membership of the adopter group was chosen by the customers according to an unobserved disposition. Better reach fixes the first and does nothing for the second.

How is it different from immortal time bias?

Both inflate the apparent benefit of adoption, but through different machinery. Immortal time bias is a time-alignment error in which the adopter group is credited with follow-up time during which it was impossible for them to churn. The healthy adherer effect is a confounding error in which the adopter group is composed of systematically different customers. A study can suffer from both at once, and the fixes are unrelated: landmark analysis for the first, randomisation or interviews for the second.

What is the fastest way to check whether my adoption finding is confounded?

Run a short identical study across adopters and non-adopters that asks about the conditions preceding adoption rather than about the product. Use closed question types so the rates are comparable between the groups. If sponsorship, ownership and available capacity differ sharply between them, your adoption metric is largely reading disposition. This takes days with an AI-moderated study rather than the weeks required to recruit and moderate two matched samples by hand.

Related Resources

• Immortal Time Bias - the time-alignment error that inflates the same adoption charts
• Collider Bias - why adding a control variable can create the correlation
• The Stepped Wedge Rollout - recovering a randomised comparison from your release schedule
• Sampling Bias - the coverage problem this is often confused with
• Survivorship Bias in Customer Research - who is missing from your data
• Quasi-Experimental Design - estimating effects when assignment was not random
• Structured Questions Guide - building the comparable closed measures this method needs

Test your next adoption claim before you build a quarter on it. Koji gives you 10 free interview credits, which is enough to ask your non-adopters what was actually happening when they stopped.