Checklist
Quasi-experiment checklist in nine steps
A nine-step checklist for evaluating a launch, pricing change or policy without an A/B test: pick the design, write the expected effect first, check the pre-period, plot pre-trends, list concurrent events, sweep for confounders, model seasonality, confirm mechanism and report assumptions. Koji can run the confounder sweep as AI interviews in hours.
Where this appears
What's in this checklist
- Ask what decided who got treated: everyone, a threshold, or a group
- Write the impact model before you look at post data
- Check you have enough pre-period: 8 to 12 points for a time series
- Plot the pre-trends, and stop if they diverge
- List concurrent events: releases, campaigns, pricing, competitor moves
- Run a confounder sweep with the treated group, and the comparison group
- Fit the model with seasonality and autocorrelation handled
- Check mechanism: can treated users describe the change?
- Report the assumption behind the estimate along with the number
Use this image
Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:
html
<a href="https://www.koji.so/visuals/quasi-experiment-checklist-nine-steps"><img src="https://www.koji.so/visuals/quasi-experiment-checklist-nine-steps/quasi-experiment-checklist-nine-steps.png" alt="Checklist of nine steps for a quasi-experiment, from naming the design and writing the impact model to checking mechanism and reporting assumptions." width="1080" height="1080" /></a>
<p>Source: <a href="https://www.koji.so/visuals/quasi-experiment-checklist-nine-steps">Quasi-experiment checklist in nine steps</a>, Koji</p>
