{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-08-07T14:23:20.157Z"},"content":[{"type":"documentation","id":"95945c7a-c3c8-4960-b718-3aec4709bd25","slug":"quasi-experimental-design-guide","title":"Quasi-Experimental Design: How to Measure Impact When You Cannot Run an A/B Test (2026)","url":"https://www.koji.so/docs/quasi-experimental-design-guide","summary":"Quasi-experimental designs estimate causal effects when random assignment is unavailable. The design follows from what determined treatment: universal rollout points to interrupted time series, threshold assignment to regression discontinuity, and group assignment to difference-in-differences. Because only about a third of ideas at Microsoft improved their target metric, uncontrolled before-and-after evaluations systematically overstate success. All three designs fail on unmeasured concurrent events, which are discoverable only by asking people who lived through the period - making a qualitative confounder sweep the highest-value complement to the quantitative estimate.","content":"**A quasi-experimental design estimates the effect of an intervention when you cannot randomly assign who receives it.** Instead of buying your counterfactual through randomisation, you construct it - from the pre-existing trend, from an untreated comparison group, or from units that sat just the wrong side of an assignment threshold.\n\nThe short answer to \"which design should I use?\" is determined by one question: **what decided who got treated?** Three answers, three designs.\n\n| What determined who got treated | Design | The counterfactual comes from |\n| --- | --- | --- |\n| Nothing - everyone got it at once (a launch, a price change, a policy) | **Interrupted time series** | The pre-intervention trend, projected forward |\n| A threshold or rule (accounts over $50k get a CSM; scores under 6 get outreach) | **Regression discontinuity** | Units just below the cutoff versus just above |\n| A group you did not choose (one region, one platform, one segment) | **Difference-in-differences** | An untreated comparison group over the same period |\n\nIf none of the three fits, you are usually looking at a straight before-and-after comparison, which is not a quasi-experiment at all - it is the design that Donald Campbell and Julian Stanley classified as pre-experimental in their 1963 monograph *Experimental and Quasi-Experimental Designs for Research*, precisely because it leaves nearly every threat to internal validity wide open.\n\n## Why this matters more than most teams assume\n\nThe reason to care is a base rate, and it is a brutal one.\n\n**Only one third of the ideas tested on the Experimentation Platform at Microsoft improved the metric or metrics they were designed to improve** (Kohavi, Crook and Longbotham 2009, reported in Kohavi and Longbotham's overview of online controlled experiments). Success rates are lower still in well-optimised domains. Jim Manzi reported that at Google, only about 10% of controlled experiments led to business changes.\n\nSit with that. In organisations with world-class experimentation infrastructure, where every idea is generated by expert teams and reviewed before it ships, **most ideas do not work**. Your prior probability that any given change improved its target metric should start well below half.\n\nNow consider what an uncontrolled before-and-after evaluation does. It compares the metric after the change to the metric before, and reports the difference. It has no mechanism to return \"no effect\" other than the metric happening not to move - and metrics always move. So the evaluation almost always confirms success, against a base rate that says success is the exception.\n\n**An uncontrolled before-and-after is not a weak test. It is a test that mostly returns the answer you brought to it.** That is the case for quasi-experimental design: not methodological purity, but the fact that without one you will systematically overstate how often your work worked.\n\n## Design 1: Interrupted time series\n\nUse it when the intervention hit everyone at a clearly defined moment: a redesign shipped to all users, a pricing change, a new policy, a regulation.\n\nThe logic is that your pre-intervention data points define a trend, and that trend projected forward is the counterfactual - what the metric would have done in the absence of the intervention. You then test for a break at the intervention point: a **level change** (the series jumps), a **slope change** (the rate of change shifts), or both.\n\nThe definitive practical treatment is Bernal, Cummins and Gasparrini, \"Interrupted time series regression for the evaluation of public health interventions: a tutorial,\" published in the *International Journal of Epidemiology* in 2017 and cited over 3,000 times. Their worked example is instructive because it maps cleanly onto product situations: Italy banned smoking in public places from 10 January 2005 - a sharp, unambiguous, universal intervention - and the authors used segmented regression on monthly hospital admissions for acute coronary events. The model found a **reduction of 11% (relative risk 0.894, 95% CI 0.864 to 0.925, P < 0.001)**.\n\nFour things they insist on, all of which product teams routinely skip:\n\n**Propose the impact model a priori.** Bernal and colleagues call this the essential yet often omitted step. Before you look at post-intervention data, write down which shape you expect - level change, slope change, or both - and when it should appear. If you expect a redesign to shift conversion immediately, that is a level change. If you expect an onboarding improvement to compound as more cohorts pass through, that is a slope change. **This is the difference between an evaluation and a narration.** Picking the shape after seeing the data means you will always find one that fits.\n\n**Adjust for seasonality.** If your pre and post windows contain a different mix of months, weekdays or billing cycles, the comparison is contaminated before you start. This bites hardest on short series.\n\n**Handle autocorrelation and over-dispersion.** Observations close in time resemble each other, which violates the assumptions of ordinary regression and makes your confidence intervals too narrow. Standard corrections exist; ignoring the issue produces confident nonsense.\n\n**Name the time-varying confounders.** The most dangerous are other events occurring near the intervention. In the smoking-ban example, the authors flag that a new troponin test had been progressively rolled out in Italy since 2000 - a change in *how the outcome is detected* that could mimic an effect. Come back to that example; it is the crux of this whole guide.\n\n**Practical requirement:** you need enough pre-intervention points to establish a trend. Fewer than 8-12 periods on each side and you are fitting a line through noise.\n\n## Design 2: Regression discontinuity\n\nUse it when treatment was assigned by a rule with a cutoff. These rules are everywhere in software businesses and almost nobody exploits them:\n\n- Accounts above a revenue threshold get a dedicated customer success manager\n- Users with a health score below a number enter a save programme\n- Trials over a certain usage level get a sales call\n- Customers past a tenure date get grandfathered pricing\n\nThe insight, which dates to Thistlethwaite and Campbell in 1960, is that an account just above the threshold and one just below are, in every respect other than treatment, essentially the same account. The rule assigned them differently for an arbitrary reason. So the **discontinuity in the outcome at the cutoff** is the treatment effect, estimated locally.\n\nThis design has the strongest internal validity of the three - close to a randomised experiment in the neighbourhood of the cutoff - and it needs no new data collection, because the rule already ran. Two cautions:\n\n- **Nothing else can change at the same threshold.** If accounts over $50k get a CSM *and* a different SLA *and* quarterly business reviews, you are estimating the effect of the bundle, not the CSM.\n- **Units must not be able to manipulate their position.** If sales can nudge a deal over the threshold to unlock the CSM, the units just above are no longer comparable to those just below.\n\nThe estimate is local. It tells you what the CSM did for accounts near $50k, not for a $500k account.\n\n## Design 3: Difference-in-differences\n\nUse it when the intervention landed on a group you did not select - one region, one platform, one customer segment, one office.\n\nYou measure the change over time in the treated group and subtract the change over the same period in an untreated comparison group. Whatever moved both groups - seasonality, macro conditions, a competitor's launch - cancels out.\n\nThe canonical study is Card and Krueger's 1994 analysis of the New Jersey minimum wage. New Jersey raised its minimum from $4.25 to $5.05 on 1 April 1992; neighbouring Pennsylvania did not. The authors surveyed **331 fast-food restaurants in New Jersey and 79 in eastern Pennsylvania** in February 1992 and again in November 1992, yielding a difference-in-differences estimate of about +2.75 full-time-equivalent employees - a result that contradicted the standard prediction and reshaped a literature.\n\nThe design rests on one assumption that carries everything: **parallel trends.** In the absence of the intervention, the two groups would have moved together. You cannot prove this, but you can make it credible by plotting both series for several periods *before* the intervention and showing they tracked. If the pre-period lines already diverge, the design is not available and no amount of statistics rescues it.\n\nFor product teams, the most common valid comparison groups are: the same segment in a market where you did not launch, a platform that did not receive the change (web versus mobile), and a cohort excluded for operational rather than behavioural reasons.\n\n## The evidence ladder\n\nRanked by internal validity, with the cost of each rung.\n\n| Design | Internal validity | Needs | Common blocker |\n| --- | --- | --- | --- |\n| Randomised experiment | Highest | Ability to assign randomly | Politically or technically impossible for launches, pricing, policy |\n| Regression discontinuity | Very high near the cutoff | An existing threshold rule and data either side | Other things change at the same threshold |\n| Difference-in-differences | High if trends were parallel | A credible untreated comparison group | No comparable group; divergent pre-trends |\n| Interrupted time series | Moderate to high | 8-12+ periods each side; a clean intervention date | Concurrent events; seasonality; short series |\n| Before-and-after, no control | Low | Two data points | Nearly every validity threat, including regression to the mean |\n| Before-and-after on a group selected for being extreme | Lowest - actively misleading | Two data points | Guaranteed bias; see [regression to the mean](/docs/regression-to-the-mean-research) |\n\nNote the bottom row. Selecting the worst-performing cohort and re-measuring it is not a weaker version of a quasi-experiment; it is a design that produces improvement whether or not anything worked.\n\n## The threat that breaks every design on this list\n\nEvery quasi-experimental design controls for what it can measure. All of them share the same fatal exposure: **something else happened at the same time that you did not know about.** Campbell and Stanley called this the *history* threat, and it is the reason quasi-experiments are quasi.\n\nDifference-in-differences fails if something hit the treated group and not the comparison group. Interrupted time series fails if another change coincided with yours. Regression discontinuity fails if a second rule shares the cutoff.\n\nAnd here is the thing worth noticing about the troponin example from the smoking-ban tutorial: **a new blood test being rolled out across Italian hospitals is not in the dataset.** No amount of statistical care extracts it from monthly admission counts. It entered the paper because a person who understood the domain knew it had happened and wrote it down.\n\nThat is the general case. **Confounders are not discovered in data. They are discovered from people who lived through the period.**\n\n## The qualitative counterfactual\n\nThis gives quasi-experimental work a second, independent evidence source that most teams never use, and it is the one that fails least often.\n\n**Ask the treated units what else changed.**\n\nA structured, short study fielded to people inside the treated group - customers, users, the affected segment - asking what changed for them over the evaluation window, is the cheapest confounder detector available. It routinely surfaces things no dashboard holds:\n\n- A competitor changed pricing in that market six weeks before your launch\n- The affected segment had a seasonal budget cycle you did not model\n- Your own sales team ran an unannounced campaign into the treated group\n- A partner integration broke and was silently fixed mid-window\n- The metric definition changed because an engineer fixed an events bug\n\nAny one of these invalidates a difference-in-differences estimate. All of them are trivially discoverable by asking twenty people, and effectively undiscoverable otherwise.\n\nTwo further roles for qualitative evidence in a quasi-experiment:\n\n**Mechanism confirmation.** A quasi-experiment gives you an effect size, not an explanation. If the estimate says the change worked but no treated user can describe encountering it, the estimate is measuring something else. Effect without mechanism is a warning.\n\n**Impact-model specification.** The a priori impact model requires you to predict a shape and a timing. That prediction is much better informed by ten conversations about how people would actually encounter the change than by a guess in a planning document. **Interviews before the launch improve the test; interviews after the launch validate it.**\n\n## The modern approach: making the qualitative arm affordable\n\nEverything above has been methodologically well understood since the 1960s. The reason product teams still ship uncontrolled before-and-after evaluations is not ignorance. It is that the qualitative arm - the confounder sweep, the mechanism check, the pre-launch impact model interviews - has historically cost two to three weeks of recruiting, scheduling, moderating and analysing, and evaluation windows do not wait.\n\n**The binding constraint is not analysis. It is the speed of talking to the treated group.**\n\nWith Koji, that arm runs in hours. AI-moderated interviews field in parallel, so 25 conversations with your treated segment complete in about the time it takes to run one traditional session, and thematic analysis is produced automatically rather than after a week of transcript coding.\n\nThree capabilities matter for quasi-experimental work specifically:\n\n**Structured questions plus open-ended follow-up in one instrument.** Koji supports six structured question types - `open_ended`, `scale`, `single_choice`, `multiple_choice`, `ranking`, and `yes_no`. For a confounder sweep this combination is exactly right: `yes_no` and `multiple_choice` questions give you comparable rates across the treated and comparison groups (\"did anything else about your setup change in the last quarter?\"), while the AI moderator follows up conversationally on every yes to find out *what*. A conventional survey tool gives you the yes without the what, and the what is the entire value. See our [structured questions guide](/docs/structured-questions-guide) for building this instrument.\n\n**Identical moderation across arms.** If you are interviewing both the treated and the comparison group - which you should, since a confounder that hit both groups is not a confounder - the two arms must be asked the same way. Human moderators cannot guarantee that; an AI moderator asks the same core questions identically in both arms, so any difference you find is a difference in the world rather than in the interviewer. See [interviewer bias](/docs/interviewer-bias).\n\n**Speed that matches the evaluation window.** A confounder sweep is only useful before the readout is written. Fielding within 48 hours of the window closing is what makes it an input to the analysis rather than an autopsy.\n\n**The honest limitation:** interviews do not produce the causal estimate. The estimate comes from the segmented regression, the discontinuity, or the difference-in-differences. What interviews do is tell you whether the assumptions those methods rest on - no concurrent events, parallel trends, nothing else at the cutoff - actually held. Those assumptions are where quasi-experiments fail, and they are not testable in the data.\n\n## A working checklist\n\n1. **Ask what determined who got treated.** Universal, threshold-assigned, or group-assigned. That names your design.\n2. **Write the impact model before you look at post data.** Level change, slope change, or both, and when. Draw it on a blank chart.\n3. **Check you have enough pre-period.** 8-12 observations minimum for a time series; several parallel pre-periods for difference-in-differences.\n4. **Plot the pre-trends.** For difference-in-differences this is not optional; divergent pre-trends kill the design.\n5. **List concurrent events from memory.** Releases, campaigns, incidents, pricing, competitor moves, instrumentation changes.\n6. **Field a confounder sweep to the treated group** - and the comparison group if you have one.\n7. **Fit the model with seasonality and autocorrelation handled.**\n8. **Check mechanism.** Can treated users describe encountering the change?\n9. **Report the assumption, not just the estimate.** State the counterfactual you constructed and what would have to be true for it to hold.\n\n## Frequently asked questions\n\n### When should I use a quasi-experimental design instead of an A/B test?\n\nWhenever random assignment is unavailable, which is more often than experimentation advocates suggest. Pricing changes, brand campaigns, policy changes, regulatory compliance work, sales process changes, and anything shipped to all users at once are usually impossible to randomise. Quasi-experimental designs are not a downgrade from A/B testing in those situations - they are the only defensible option, and they are a large upgrade from a before-and-after comparison. Our guide to [A/B testing versus user research](/docs/ab-testing-vs-user-research) covers when each is appropriate.\n\n### How much pre-intervention data do I need for an interrupted time series?\n\nEnough to establish a trend and to see the seasonal pattern. As a practical floor, aim for at least 8 to 12 observations before and after the intervention, and more if the series is noisy or strongly seasonal - for monthly data that means at least two years if you want to model seasonality properly. With four pre-period points you are fitting a line through noise, and the projected counterfactual will carry uncertainty wide enough to accommodate almost any result.\n\n### What if I have no comparable control group for difference-in-differences?\n\nLook harder before giving up: another geography, another platform, a segment excluded for operational rather than behavioural reasons, or a similar cohort from an earlier period. If none exists, fall back to interrupted time series, which needs no control group but is more exposed to concurrent events. Whichever you choose, the confounder sweep matters more, not less, because you have lost the mechanism that would otherwise have cancelled out shared shocks.\n\n### Can qualitative research alone establish causation?\n\nNot in the sense of an effect size, and claims that it can should be treated sceptically. What qualitative research establishes is mechanism and the presence or absence of competing explanations - both of which are prerequisites for a causal claim that the quantitative estimate cannot supply. The strongest evidence combines them: a defensible quasi-experimental estimate, plus interview evidence that treated users encountered the change, plus a confounder sweep showing nothing else material happened in the window.\n\n### How does regression to the mean interact with quasi-experimental designs?\n\nIt attacks any design where the treated units were selected because their outcome was extreme. Interrupted time series on a whole population is largely safe. Difference-in-differences is safe if the treated group was defined by geography or platform rather than by performance, and badly compromised if the treated group was the worst-performing segment. Regression discontinuity is generally safe because assignment is by an arbitrary rule rather than by a noisy performance measure - unless the running variable is itself a noisy score. See [regression to the mean](/docs/regression-to-the-mean-research) for the full treatment.\n\n### What is the single biggest mistake teams make with these designs?\n\nChoosing the impact model after seeing the data. Once you have looked at the post-intervention series, you will find a shape that fits - a level change here, a delayed slope change there - and you will believe it, because it is genuinely present in the data. Writing the expected shape and timing down beforehand converts the exercise from storytelling into a test that can fail. It costs ten minutes and it is the highest-return step in this entire guide.\n\n## Related Resources\n\n- [Structured Questions Guide](/docs/structured-questions-guide) - building a confounder sweep with the six question types\n- [Regression to the Mean](/docs/regression-to-the-mean-research) - the bias that contaminates any design using extreme selection\n- [A/B Testing vs User Research](/docs/ab-testing-vs-user-research) - when to randomise and when to ask\n- [Statistical Power and Minimum Detectable Effect](/docs/statistical-power-minimum-detectable-effect) - sizing the effect you could actually detect\n- [Statistical Significance in Survey Research](/docs/statistical-significance-survey-research) - interpreting the estimate once you have it\n- [Program Evaluation Surveys](/docs/program-evaluation-survey-guide) - the instrument side of proving impact\n- [Longitudinal Research](/docs/longitudinal-research-guide) - collecting the repeated measures these designs depend on\n- [Panel Conditioning](/docs/panel-conditioning-repeat-participants) - why repeated measurement changes the people you are measuring\n\n---\n\n**Run the confounder sweep on your next launch.** Koji gives you 10 free interview credits - enough to ask your treated segment what else changed before you write the readout.","category":"Research Methods","lastModified":"2026-08-07T03:24:50.49997+00:00","metaTitle":"Quasi-Experimental Design: Measure Impact Without an A/B Test (2026)","metaDescription":"Cannot randomise? Learn which of the three quasi-experimental designs your situation calls for - interrupted time series, regression discontinuity, or difference-in-differences - and how to test their assumptions.","keywords":["quasi-experimental design","interrupted time series","difference in differences","regression discontinuity","pre post study design","measure impact without ab test","causal inference product","counterfactual analysis"],"aiSummary":"Quasi-experimental designs estimate causal effects when random assignment is unavailable. The design follows from what determined treatment: universal rollout points to interrupted time series, threshold assignment to regression discontinuity, and group assignment to difference-in-differences. Because only about a third of ideas at Microsoft improved their target metric, uncontrolled before-and-after evaluations systematically overstate success. All three designs fail on unmeasured concurrent events, which are discoverable only by asking people who lived through the period - making a qualitative confounder sweep the highest-value complement to the quantitative estimate.","aiPrerequisites":["Basic familiarity with product metrics","Understanding of averages, trends and control groups"],"aiLearningOutcomes":["Choose between interrupted time series, regression discontinuity and difference-in-differences by asking what determined treatment","Write an a priori impact model specifying expected shape and timing","Test the parallel-trends assumption before relying on a difference-in-differences estimate","Run a qualitative confounder sweep to detect concurrent events no dataset contains","Rank available evidence by internal validity and report the assumption alongside the estimate"],"aiDifficulty":"advanced","aiEstimatedTime":"14 min"}],"pagination":{"total":1,"returned":1,"offset":0}}