Quasi-Experimental Design: How to Measure Impact When You Cannot Run an A/B Test (2026)

A quasi-experimental design estimates the effect of an intervention when you cannot randomly assign who receives it. Instead of buying your counterfactual through randomisation, you construct it - from the pre-existing trend, from an untreated comparison group, or from units that sat just the wrong side of an assignment threshold.

The short answer to "which design should I use?" is determined by one question: what decided who got treated? Three answers, three designs.

If none of the three fits, you are usually looking at a straight before-and-after comparison, which is not a quasi-experiment at all - it is the design that Donald Campbell and Julian Stanley classified as pre-experimental in their 1963 monograph Experimental and Quasi-Experimental Designs for Research, precisely because it leaves nearly every threat to internal validity wide open.

Why this matters more than most teams assume

The reason to care is a base rate, and it is a brutal one.

Only one third of the ideas tested on the Experimentation Platform at Microsoft improved the metric or metrics they were designed to improve (Kohavi, Crook and Longbotham 2009, reported in Kohavi and Longbotham's overview of online controlled experiments). Success rates are lower still in well-optimised domains. Jim Manzi reported that at Google, only about 10% of controlled experiments led to business changes.

Sit with that. In organisations with world-class experimentation infrastructure, where every idea is generated by expert teams and reviewed before it ships, most ideas do not work. Your prior probability that any given change improved its target metric should start well below half.

Now consider what an uncontrolled before-and-after evaluation does. It compares the metric after the change to the metric before, and reports the difference. It has no mechanism to return "no effect" other than the metric happening not to move - and metrics always move. So the evaluation almost always confirms success, against a base rate that says success is the exception.

An uncontrolled before-and-after is not a weak test. It is a test that mostly returns the answer you brought to it. That is the case for quasi-experimental design: not methodological purity, but the fact that without one you will systematically overstate how often your work worked.

Design 1: Interrupted time series

Use it when the intervention hit everyone at a clearly defined moment: a redesign shipped to all users, a pricing change, a new policy, a regulation.

The logic is that your pre-intervention data points define a trend, and that trend projected forward is the counterfactual - what the metric would have done in the absence of the intervention. You then test for a break at the intervention point: a level change (the series jumps), a slope change (the rate of change shifts), or both.

The definitive practical treatment is Bernal, Cummins and Gasparrini, "Interrupted time series regression for the evaluation of public health interventions: a tutorial," published in the International Journal of Epidemiology in 2017 and cited over 3,000 times. Their worked example is instructive because it maps cleanly onto product situations: Italy banned smoking in public places from 10 January 2005 - a sharp, unambiguous, universal intervention - and the authors used segmented regression on monthly hospital admissions for acute coronary events. The model found a reduction of 11% (relative risk 0.894, 95% CI 0.864 to 0.925, P < 0.001).

Four things they insist on, all of which product teams routinely skip:

Propose the impact model a priori. Bernal and colleagues call this the essential yet often omitted step. Before you look at post-intervention data, write down which shape you expect - level change, slope change, or both - and when it should appear. If you expect a redesign to shift conversion immediately, that is a level change. If you expect an onboarding improvement to compound as more cohorts pass through, that is a slope change. This is the difference between an evaluation and a narration. Picking the shape after seeing the data means you will always find one that fits.

Adjust for seasonality. If your pre and post windows contain a different mix of months, weekdays or billing cycles, the comparison is contaminated before you start. This bites hardest on short series.

Handle autocorrelation and over-dispersion. Observations close in time resemble each other, which violates the assumptions of ordinary regression and makes your confidence intervals too narrow. Standard corrections exist; ignoring the issue produces confident nonsense.

Name the time-varying confounders. The most dangerous are other events occurring near the intervention. In the smoking-ban example, the authors flag that a new troponin test had been progressively rolled out in Italy since 2000 - a change in how the outcome is detected that could mimic an effect. Come back to that example; it is the crux of this whole guide.

Practical requirement: you need enough pre-intervention points to establish a trend. Fewer than 8-12 periods on each side and you are fitting a line through noise.

Design 2: Regression discontinuity

Use it when treatment was assigned by a rule with a cutoff. These rules are everywhere in software businesses and almost nobody exploits them:

• Accounts above a revenue threshold get a dedicated customer success manager
• Users with a health score below a number enter a save programme
• Trials over a certain usage level get a sales call
• Customers past a tenure date get grandfathered pricing

The insight, which dates to Thistlethwaite and Campbell in 1960, is that an account just above the threshold and one just below are, in every respect other than treatment, essentially the same account. The rule assigned them differently for an arbitrary reason. So the discontinuity in the outcome at the cutoff is the treatment effect, estimated locally.

This design has the strongest internal validity of the three - close to a randomised experiment in the neighbourhood of the cutoff - and it needs no new data collection, because the rule already ran. Two cautions:

• Nothing else can change at the same threshold. If accounts over $50k get a CSM and a different SLA and quarterly business reviews, you are estimating the effect of the bundle, not the CSM.
• Units must not be able to manipulate their position. If sales can nudge a deal over the threshold to unlock the CSM, the units just above are no longer comparable to those just below.

The estimate is local. It tells you what the CSM did for accounts near $50k, not for a $500k account.

Design 3: Difference-in-differences

Use it when the intervention landed on a group you did not select - one region, one platform, one customer segment, one office.

You measure the change over time in the treated group and subtract the change over the same period in an untreated comparison group. Whatever moved both groups - seasonality, macro conditions, a competitor's launch - cancels out.

The canonical study is Card and Krueger's 1994 analysis of the New Jersey minimum wage. New Jersey raised its minimum from $4.25 to $5.05 on 1 April 1992; neighbouring Pennsylvania did not. The authors surveyed 331 fast-food restaurants in New Jersey and 79 in eastern Pennsylvania in February 1992 and again in November 1992, yielding a difference-in-differences estimate of about +2.75 full-time-equivalent employees - a result that contradicted the standard prediction and reshaped a literature.

The design rests on one assumption that carries everything: parallel trends. In the absence of the intervention, the two groups would have moved together. You cannot prove this, but you can make it credible by plotting both series for several periods before the intervention and showing they tracked. If the pre-period lines already diverge, the design is not available and no amount of statistics rescues it.

For product teams, the most common valid comparison groups are: the same segment in a market where you did not launch, a platform that did not receive the change (web versus mobile), and a cohort excluded for operational rather than behavioural reasons.

The evidence ladder

Ranked by internal validity, with the cost of each rung.

Note the bottom row. Selecting the worst-performing cohort and re-measuring it is not a weaker version of a quasi-experiment; it is a design that produces improvement whether or not anything worked.

The threat that breaks every design on this list

Every quasi-experimental design controls for what it can measure. All of them share the same fatal exposure: something else happened at the same time that you did not know about. Campbell and Stanley called this the history threat, and it is the reason quasi-experiments are quasi.

Difference-in-differences fails if something hit the treated group and not the comparison group. Interrupted time series fails if another change coincided with yours. Regression discontinuity fails if a second rule shares the cutoff.

And here is the thing worth noticing about the troponin example from the smoking-ban tutorial: a new blood test being rolled out across Italian hospitals is not in the dataset. No amount of statistical care extracts it from monthly admission counts. It entered the paper because a person who understood the domain knew it had happened and wrote it down.

That is the general case. Confounders are not discovered in data. They are discovered from people who lived through the period.

The qualitative counterfactual

This gives quasi-experimental work a second, independent evidence source that most teams never use, and it is the one that fails least often.

Ask the treated units what else changed.

A structured, short study fielded to people inside the treated group - customers, users, the affected segment - asking what changed for them over the evaluation window, is the cheapest confounder detector available. It routinely surfaces things no dashboard holds:

• A competitor changed pricing in that market six weeks before your launch
• The affected segment had a seasonal budget cycle you did not model
• Your own sales team ran an unannounced campaign into the treated group
• A partner integration broke and was silently fixed mid-window
• The metric definition changed because an engineer fixed an events bug

Any one of these invalidates a difference-in-differences estimate. All of them are trivially discoverable by asking twenty people, and effectively undiscoverable otherwise.

Two further roles for qualitative evidence in a quasi-experiment:

Mechanism confirmation. A quasi-experiment gives you an effect size, not an explanation. If the estimate says the change worked but no treated user can describe encountering it, the estimate is measuring something else. Effect without mechanism is a warning.

Impact-model specification. The a priori impact model requires you to predict a shape and a timing. That prediction is much better informed by ten conversations about how people would actually encounter the change than by a guess in a planning document. Interviews before the launch improve the test; interviews after the launch validate it.

The modern approach: making the qualitative arm affordable

Everything above has been methodologically well understood since the 1960s. The reason product teams still ship uncontrolled before-and-after evaluations is not ignorance. It is that the qualitative arm - the confounder sweep, the mechanism check, the pre-launch impact model interviews - has historically cost two to three weeks of recruiting, scheduling, moderating and analysing, and evaluation windows do not wait.

The binding constraint is not analysis. It is the speed of talking to the treated group.

With Koji, that arm runs in hours. AI-moderated interviews field in parallel, so 25 conversations with your treated segment complete in about the time it takes to run one traditional session, and thematic analysis is produced automatically rather than after a week of transcript coding.

Three capabilities matter for quasi-experimental work specifically:

Structured questions plus open-ended follow-up in one instrument. Koji supports six structured question types - open_ended, scale, single_choice, multiple_choice, ranking, and yes_no. For a confounder sweep this combination is exactly right: yes_no and multiple_choice questions give you comparable rates across the treated and comparison groups ("did anything else about your setup change in the last quarter?"), while the AI moderator follows up conversationally on every yes to find out what. A conventional survey tool gives you the yes without the what, and the what is the entire value. See our structured questions guide for building this instrument.

Identical moderation across arms. If you are interviewing both the treated and the comparison group - which you should, since a confounder that hit both groups is not a confounder - the two arms must be asked the same way. Human moderators cannot guarantee that; an AI moderator asks the same core questions identically in both arms, so any difference you find is a difference in the world rather than in the interviewer. See interviewer bias.

Speed that matches the evaluation window. A confounder sweep is only useful before the readout is written. Fielding within 48 hours of the window closing is what makes it an input to the analysis rather than an autopsy.

The honest limitation: interviews do not produce the causal estimate. The estimate comes from the segmented regression, the discontinuity, or the difference-in-differences. What interviews do is tell you whether the assumptions those methods rest on - no concurrent events, parallel trends, nothing else at the cutoff - actually held. Those assumptions are where quasi-experiments fail, and they are not testable in the data.

A working checklist

1. Ask what determined who got treated. Universal, threshold-assigned, or group-assigned. That names your design.
2. Write the impact model before you look at post data. Level change, slope change, or both, and when. Draw it on a blank chart.
3. Check you have enough pre-period. 8-12 observations minimum for a time series; several parallel pre-periods for difference-in-differences.
4. Plot the pre-trends. For difference-in-differences this is not optional; divergent pre-trends kill the design.
5. List concurrent events from memory. Releases, campaigns, incidents, pricing, competitor moves, instrumentation changes.
6. Field a confounder sweep to the treated group - and the comparison group if you have one.
7. Fit the model with seasonality and autocorrelation handled.
8. Check mechanism. Can treated users describe encountering the change?
9. Report the assumption, not just the estimate. State the counterfactual you constructed and what would have to be true for it to hold.

Frequently asked questions

When should I use a quasi-experimental design instead of an A/B test?

Whenever random assignment is unavailable, which is more often than experimentation advocates suggest. Pricing changes, brand campaigns, policy changes, regulatory compliance work, sales process changes, and anything shipped to all users at once are usually impossible to randomise. Quasi-experimental designs are not a downgrade from A/B testing in those situations - they are the only defensible option, and they are a large upgrade from a before-and-after comparison. Our guide to A/B testing versus user research covers when each is appropriate.

How much pre-intervention data do I need for an interrupted time series?

Enough to establish a trend and to see the seasonal pattern. As a practical floor, aim for at least 8 to 12 observations before and after the intervention, and more if the series is noisy or strongly seasonal - for monthly data that means at least two years if you want to model seasonality properly. With four pre-period points you are fitting a line through noise, and the projected counterfactual will carry uncertainty wide enough to accommodate almost any result.

What if I have no comparable control group for difference-in-differences?

Look harder before giving up: another geography, another platform, a segment excluded for operational rather than behavioural reasons, or a similar cohort from an earlier period. If none exists, fall back to interrupted time series, which needs no control group but is more exposed to concurrent events. Whichever you choose, the confounder sweep matters more, not less, because you have lost the mechanism that would otherwise have cancelled out shared shocks.

Can qualitative research alone establish causation?

Not in the sense of an effect size, and claims that it can should be treated sceptically. What qualitative research establishes is mechanism and the presence or absence of competing explanations - both of which are prerequisites for a causal claim that the quantitative estimate cannot supply. The strongest evidence combines them: a defensible quasi-experimental estimate, plus interview evidence that treated users encountered the change, plus a confounder sweep showing nothing else material happened in the window.

How does regression to the mean interact with quasi-experimental designs?

It attacks any design where the treated units were selected because their outcome was extreme. Interrupted time series on a whole population is largely safe. Difference-in-differences is safe if the treated group was defined by geography or platform rather than by performance, and badly compromised if the treated group was the worst-performing segment. Regression discontinuity is generally safe because assignment is by an arbitrary rule rather than by a noisy performance measure - unless the running variable is itself a noisy score. See regression to the mean for the full treatment.

What is the single biggest mistake teams make with these designs?

Choosing the impact model after seeing the data. Once you have looked at the post-intervention series, you will find a shape that fits - a level change here, a delayed slope change there - and you will believe it, because it is genuinely present in the data. Writing the expected shape and timing down beforehand converts the exercise from storytelling into a test that can fail. It costs ten minutes and it is the highest-return step in this entire guide.

Related Resources

• Structured Questions Guide - building a confounder sweep with the six question types
• Regression to the Mean - the bias that contaminates any design using extreme selection
• A/B Testing vs User Research - when to randomise and when to ask
• Statistical Power and Minimum Detectable Effect - sizing the effect you could actually detect
• Statistical Significance in Survey Research - interpreting the estimate once you have it
• Program Evaluation Surveys - the instrument side of proving impact
• Longitudinal Research - collecting the repeated measures these designs depend on
• Panel Conditioning - why repeated measurement changes the people you are measuring

---

Run the confounder sweep on your next launch. Koji gives you 10 free interview credits - enough to ask your treated segment what else changed before you write the readout.