All frameworks and cheat sheets

Checklist

Immortal time bias checklist for retention analysis

Checklist of six questions to spot immortal time bias in a retention comparison, from group membership rules to shortening the window.
Run any adopters vs non-adopters chart through these six questions before it reaches a roadmap.

Six questions that reveal immortal time bias in a product retention comparison, such as adopters versus non-adopters of a feature. They check whether group membership needs an action, whether both clocks start at the same event, how long the window is, where churn sits and whether the effect shrinks with a shorter window. The last check is the cheapest: rerun the chart at 14, 30 and 90 days.

Where this appears

What's in this checklist

  • Does group membership require an action? That time is immortal.
  • Do both groups start the clock at the same event?
  • Could a user churn before they can be put in a group?
  • How long is the adoption window? Longer means more bias.
  • Is churn front-loaded? That makes the bias worse.
  • Does the effect shrink when you shorten the window?

Use this image

Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:

html
<a href="https://www.koji.so/visuals/immortal-time-bias-retention-checklist"><img src="https://www.koji.so/visuals/immortal-time-bias-retention-checklist/immortal-time-bias-retention-checklist.png" alt="Checklist of six questions to spot immortal time bias in a retention comparison, from group membership rules to shortening the window." width="1200" height="630" /></a>
<p>Source: <a href="https://www.koji.so/visuals/immortal-time-bias-retention-checklist">Immortal time bias checklist for retention analysis</a>, Koji</p>