Instrumental Variables and Encouragement Designs: Measuring an Effect You Cannot Assign (2026)

When you cannot assign the treatment, randomise the encouragement to take it. That single move converts an observational mess into a real experiment -- and delivers an effect that applies only to the people the encouragement moved, a group you can describe but never list.

The two previous designs in this cluster both work hard at the same problem. Synthetic control constructs a comparison unit when none exists. Propensity score matching tries to make treated and untreated groups comparable and, past a point, makes them worse. Both accept the same premise: that the path to a causal estimate runs through building a group comparable to the treated one.

Instrumental variables abandons that premise. It concedes that adopters and non-adopters are incomparable, that the difference lives in things you never measured, and that no amount of adjustment will fix it. Then it finds something else to randomise.

The setup

You want the effect of adopting a feature on retention. You cannot randomise adoption, because adoption is a choice. What you can randomise is the prompt: half your eligible users see an in-app invitation to try the feature, half do not.

Randomisation gives you an unimpeachable comparison between prompted and unprompted groups. That comparison answers a real question -- what did shipping this prompt achieve? -- and it is the intention to treat estimate.

But it understates the effect of the feature, because most prompted users ignored the prompt. If the prompt raised adoption from 10 percent to 30 percent and raised 90-day retention by 2 points, the feature did not produce 2 points among everyone. It produced roughly 2 divided by 0.20, or 10 points, among the 20 percent of users whose adoption the prompt actually caused.

That division is instrumental variables. The prompt is the instrument, the ratio is the estimate, and it goes by the name Angrist and Imbens gave it: the local average treatment effect.

What "local" is doing in that name

Split your users into four types by how they respond to encouragement.

Always-takers adopt in both arms, so they contribute nothing to the difference. Never-takers adopt in neither, so they contribute nothing either. Defiers -- users who adopt only when not prompted -- would break the arithmetic, and the monotonicity assumption rules them out. That leaves compliers, and the estimate is theirs alone.

The Nobel committee's scientific background for the 2021 prize to Angrist, Imbens and Card describes the contribution precisely: Angrist and Imbens showed that "an average causal effect can be estimated among those who complied with the assignment generated by the experiment or the quasi-experiment, and that this effect is identified by instrumental variables." They named it "the local average treatment effect -- sometimes it is also referred to as the complier average causal effect," and in doing so "made clear exactly what one can say about treatment effects in settings where the source of the heterogeneity is not known."

That last clause is the honest core of the method. It does not model away the heterogeneity between adopters and non-adopters. It states which slice of it the answer covers.

The capstone problem: you cannot name a complier

Here is the part that gets omitted from most write-ups, and it is the reason this design deserves careful handling rather than enthusiasm.

Compliance is defined by two outcomes -- what a user would do if prompted and what they would do if not -- and you only ever observe one of them for any given user. A prompted user who adopted might be a complier or an always-taker. An unprompted user who did not adopt might be a complier or a never-taker. The data cannot separate them.

The Nobel background states it in a footnote, and the sentence is worth quoting exactly: "Note that 'compliers' is a conceptual construct. We cannot tell whether an individual is a complier."

So you now hold a defensible causal estimate about a group whose members cannot be enumerated. You cannot filter them in your warehouse, cannot build a segment from them, cannot email them, and cannot check whether they resemble your most valuable customers.

This is a different kind of limitation from the previous two designs. Synthetic control gives you a number without a mechanism. Matching gives you a number that may encode selection. Instrumental variables gives you a clean number about an unidentifiable population -- and if compliers happen to be your most marginal, least committed users, a large effect among them may be a poor guide to what happens if you ship the feature to everyone.

The canonical example makes the risk concrete. Angrist and Krueger used quarter of birth, interacting with compulsory schooling laws, as an instrument for years of education, and estimated a causal return of roughly 9 percent per additional year -- slightly higher than the naive regression. But as the Nobel background notes, "the quasi-experimental variation mainly affected those with a high probability of dropping out of school as soon as possible," so "the returns to schooling in this part of the population are not representative of the overall population." The instrument was valid and the estimate was right. It was right about a specific group that the instrument selected, and that group was unusual.

The same structure appears in product work constantly. A prompt shown in-app moves users who are already in the product, already paying some attention, and already close to the decision. The effect you measure belongs to them, not to the disengaged users you were hoping to activate.

The three conditions, and the one that usually fails

An instrument must satisfy three requirements.

Relevance. The encouragement must actually shift take-up. A prompt that moves adoption from 10 percent to 11 percent gives you a denominator near zero, which makes the estimate wildly unstable and biased toward the naive comparison. Design encouragements expected to move take-up substantially, and be willing to abandon the analysis when the realised shift is small.

Monotonicity. No defiers. Usually reasonable, but not automatic: an aggressive or badly timed prompt can create genuine reactance, where users who would otherwise have adopted refuse because they were pushed.

The exclusion restriction. The encouragement must affect the outcome only through take-up. This is the one that fails, and product encouragements are close to purpose-built to violate it.

Consider what an in-app prompt does besides driving adoption. It teaches -- the user now knows the feature exists, and may use that knowledge weeks later through another route. It reminds the user your product exists at all, which can lift engagement on its own. It signals that your company invested in this area, which can shift perceived product quality. Any of those paths moves retention without the user ever adopting, and every one of them violates exclusion.

The Nobel background is blunt that this is the general difficulty: "Almost inevitably, when we try to imagine any candidate z, this exclusion restriction becomes a concern."

The low-risk row is the design worth engineering toward. If access itself is randomly staged and the communication is held constant across arms, the instrument shifts take-up while conveying almost nothing else. That is a much stronger footing than a prompt whose entire job is persuasion.

Two reporting rules follow. Report intention to treat alongside the instrumental variables estimate, always -- intention to treat is assumption-light and answers what shipping the encouragement did, which is often the decision actually on the table. And state the exclusion restriction as a written, arguable claim, not a checkbox: name the alternative pathways and say why you think each is small.

What research can recover that the data cannot

The situation looks closed. The estimate applies to compliers; compliers cannot be identified; therefore you cannot know who the estimate is about.

But the Nobel footnote does not stop at the sentence usually quoted. It continues: "the share of compliers in the population can be determined, and their characteristics can be described."

That is an opening, and it is a research job rather than an analytics one.

The complier share is straightforward: it is the difference in take-up between the arms. If the prompt moved adoption from 10 to 30 percent, compliers are 20 percent of your population. Reporting that number alongside the effect changes how the result reads. A 10-point retention gain among 20 percent of users is a very different roadmap input than a 10-point gain among everyone, and most decks present the second while the analysis supports the first.

Their characteristics are where interviews do work nothing in the warehouse can. You cannot label an individual as a complier, but you can go to the prompted arm, find the users who adopted, and ask a question the data cannot hold: would you have found this without the prompt? An always-taker will tell you they were already looking for it. A complier will tell you the prompt arrived at a moment when something else made it relevant. Self-report is imperfect and the boundary is fuzzy, but it is real evidence about a population that is otherwise entirely dark.

The follow-up question is the valuable one: what made the prompt land this time? Compliers are by definition the marginal users -- close enough to adopting that a nudge decided it. What tipped them is the most direct information you will ever get about the real barrier to adoption, and it is exactly what a take-up rate cannot express.

The modern approach: describing an unnameable population

This used to be impractical. Interviewing enough adopters across both arms, fast enough to inform the rollout decision, was a research programme competing against a number that already existed in the dashboard. Teams shipped the number.

Koji changes that calculation. Field an AI-moderated interview to a sample from each arm and get to analysis in hours rather than weeks. The AI moderator matters more here than in most studies, because the discriminating information sits behind a follow-up: "would you have found it anyway?" gets a confident yes from almost everyone, and only a probe into what they had actually tried before separates a real always-taker from a complier telling a flattering story about themselves. A static form takes the first answer. Voice interviews work well for this, because the hesitation before the answer is itself informative.

Use structured questions for the parts that must be comparable across arms. Koji supports six types, and a complier-characterisation study uses each of them deliberately:

• yes_no for the classifying fact: had they looked for this capability before the prompt?
• single_choice to place them into an adoption-path category
• scale for prior intent, which is the closest measurable proxy for complier status
• multiple_choice for which alternatives they had already tried
• ranking to order what actually decided it when several factors applied
• open_ended, with AI follow-up, to surface the trigger nobody listed

Automatic thematic analysis then does the comparison that matters: themes present among adopters in the prompted arm but absent among adopters in the unprompted arm. Adopters in the unprompted arm are always-takers by construction, which gives you a clean contrast group. Whatever is unique to the prompted arm's adopters is your description of compliers -- assembled from real respondents, not inferred from a model.

You still cannot label an individual. You can now say what kind of user the estimate is about, which is the question every stakeholder asks and which the econometrics alone cannot answer.

A working checklist

1. Confirm the treatment genuinely cannot be randomised. If it can, run a normal experiment.
2. Design an encouragement that shifts take-up substantially while conveying as little else as possible.
3. Randomise the encouragement and verify the arms are balanced.
4. Report intention to treat first.
5. Report take-up in both arms; the difference is your complier share.
6. Divide to get the local average treatment effect, and label it as applying to compliers only.
7. Write the exclusion restriction as an argued claim, naming each alternative pathway.
8. Check for reactance before leaning on monotonicity.
9. Interview adopters in both arms to characterise compliers.
10. Report the effect, the complier share, and the complier description together. Never the first alone.

Frequently asked questions

What is an encouragement design?

You randomise an invitation to take a treatment rather than the treatment itself. Half of users get a prompt to try a feature, half do not. Because the prompt was randomised, the difference in outcomes is causal, and dividing it by the difference in take-up rescales it into an effect of the feature on the people the prompt moved.

What are compliers, always-takers, and never-takers?

Compliers take the treatment when encouraged and not otherwise. Always-takers take it regardless, never-takers refuse regardless, and defiers do the opposite of what they are encouraged to do. Monotonicity assumes defiers do not exist. Instrumental variables estimates apply only to compliers.

Can I tell which of my users are compliers?

Not individually. The Nobel committee's background paper puts it plainly: compliers are a conceptual construct and we cannot tell whether an individual is a complier. You can, however, estimate what share of the population they are and describe their characteristics, which is enough to judge whether the estimate generalises.

What is the exclusion restriction and why does it fail in product work?

It requires the encouragement to affect the outcome only by changing take-up. Product prompts routinely violate it, because a prompt also teaches, reminds, and signals that the company considers the feature important. Any of those can move retention without the user ever adopting anything.

How is this different from intention to treat?

Intention to treat compares everyone as randomised and answers what the encouragement achieved in practice. Instrumental variables rescales that by take-up to estimate the effect of the treatment itself on compliers. Both are legitimate; they answer different questions and should be reported together.

What makes an encouragement weak, and why does that matter?

If the prompt barely shifts take-up, the denominator of the rescaling is close to zero and the estimate becomes unstable and biased. A useful rule of thumb is to design encouragements expected to move take-up by a large, clearly detectable margin, and to abandon the analysis if the realised shift is small.

Does this replace synthetic control or matching?

It replaces the assumption they depend on. Both of those require that treated and untreated units can be made comparable. An encouragement design does not, because randomisation handles comparability. The trade is that its answer applies to a narrower and unnameable group.

Related Resources

• Synthetic Control -- building a comparison unit when none exists
• Propensity Score Matching and the Pruning Paradox -- why forcing comparability can backfire
• Intention to Treat vs Per Protocol -- which responses belong in the analysis
• Quasi-Experimental Design -- the designs to try when randomisation of any kind is available
• The Healthy Adherer Effect -- the selection problem this design routes around
• Structured Questions in AI Interviews -- the six question types and how to characterise a population with them