Propensity Score Matching and the Pruning Paradox: When Better Balance Makes Your Estimate Worse (2026)

Propensity score matching promises to turn observational data into something resembling an experiment by pairing treated and untreated units that were equally likely to be treated. King and Nielsen showed in 2019 that beyond a certain point the method reverses: continued pruning increases imbalance, model dependence and bias rather than reducing them. They call it the PSM paradox, and it matters because the workflow it describes is the one most product analytics teams run by default.

This is an uncomfortable result, and it cuts against the grain of nearly every other measurement lesson. Elsewhere, more care converges on the truth: more respondents narrow the interval, more sources corroborate the finding, tighter controls reduce the noise. Here, more effort past the turning point makes the answer worse, and the diagnostic most teams look at gets better while it happens.

The problem matching is trying to solve

Some users adopted the feature. Others did not. Adopters retain better. You cannot conclude the feature causes retention, because the people who adopt differ from the people who do not in ways that also drive retention -- they are more engaged, larger, further into onboarding, on a better plan.

The instinct is to compare like with like. Find, for each adopter, a non-adopter with the same plan, company size, tenure, and usage level, and compare only those pairs. Any remaining difference is attributable to adoption.

With four covariates this is feasible. With twenty it is not: exact matches stop existing. Rosenbaum and Rubin's 1983 result offered an elegant way out. Estimate each unit's probability of treatment given its covariates -- the propensity score -- and match on that single number instead. If treatment assignment depends only on observed covariates, matching on the score is sufficient to remove the confounding those covariates carry. Twenty dimensions collapse into one.

The elegance is why the method spread. King and Nielsen quantified how far: they sampled 1,000 randomly selected English-language articles from JSTOR published between 1983 and 2015 that reference PSM, narrowed to the 230 that applied it to real data, and found that about 80 percent used one-to-one greedy matching, the simplest possible version. Whatever refinements exist in the methodological literature, the version in the field is the plain one.

The paradox

King and Nielsen's abstract states the finding without hedging:

"We show that propensity score matching (PSM), an enormously popular method of preprocessing data for causal inference, often accomplishes the opposite of its intended goal -- thus increasing imbalance, inefficiency, model dependence, and bias."

The mechanism is a target problem. There are two experimental designs you might try to imitate. A completely randomised experiment flips a coin for each unit; treated and control groups match in expectation, but in any single realisation they differ by chance. A fully blocked experiment first groups units into strata of identical or near-identical units and randomises within each block; the groups match exactly on the blocking variables, which is strictly more informative.

Propensity score matching targets the first. That is the whole problem. As King and Nielsen put it, PSM "is thus uniquely blind to the often large portion of imbalance that can be eliminated by approximating full blocking with other matching methods."

The reason is that the score is a compression. Many different covariate profiles produce the same propensity score. A large, old, low-usage account and a small, new, high-usage account can both have a 0.4 probability of adoption. PSM regards them as an acceptable match. They are not remotely similar, and the differences it ignored are exactly the ones that drive the outcome.

The paradox follows. Once your data are balanced enough to look like a completely randomised experiment -- either because they started that way or because pruning got them there -- PSM has nothing left to target. It begins pruning approximately at random. And random pruning, as the authors show, "increases imbalance even relative to the original data."

Their summary of the condition is the sentence to remember: "the more balanced the data, or the more balanced it becomes by pruning some observations through matching, the more likely PSM will degrade inferences."

Why this bites product teams specifically

The paradox is worst when data are already fairly balanced, and product data often are. If you compare users on the same plan, in the same cohort, in the same region, the treated and untreated groups may already be close on observables. That is precisely the regime where PSM has nothing useful to do and starts pruning noise.

The failure is quiet, because the diagnostic teams check is the wrong one. Standard practice is to compare the propensity score distributions before and after matching and confirm they now overlap. They will. Matching on the score guarantees the score balances. Balance on the score is not balance on the covariates, and it is the covariates that drive the outcome.

Three habits make it worse in practice:

• Iterating to a good-looking result. Respecify the score model, rerun, check balance, repeat. Each pass adds a researcher degree of freedom, and the analytic flexibility that creates is the same machinery described in p-hacking and researcher degrees of freedom.
• Treating pruning as free. Dropping unmatched units is presented as conservative. Past the turning point it is actively harmful, and it also changes the population your estimate describes without anyone writing that down.
• Matching on post-treatment variables. Including usage measured after adoption as a matching covariate does not control for confounding; it conditions on a consequence, which can manufacture the association outright. That is collider bias, and matching is a common way to walk into it.

A note on vocabulary, since the word is overloaded. Propensity weighting in surveys, covered in our survey weighting guide, models the probability that someone responds and corrects a skewed sample. Propensity score matching models the probability that someone was treated and tries to remove confounding. Same word, different estimand, different failure modes. The paradox described here is about the second.

Detecting it in your own analysis

The diagnostic is a plot, and it takes minutes.

Compute a multivariate imbalance measure across your full covariate profile -- not the propensity score. Then compute it repeatedly as you prune: at zero units pruned, ten, twenty, and so on. Plot imbalance on the vertical axis against units pruned on the horizontal.

A method that is working produces a curve that falls and keeps falling over a long stretch. A method in the paradox produces a curve that falls, reaches a minimum, and turns upward. Everything to the right of that minimum is damage.

The recommended alternatives target blocking rather than randomisation: coarsened exact matching, which bins each covariate and matches within bins, and Mahalanobis distance matching. King and Nielsen show these reduce imbalance more effectively for the same number of units pruned and do not generate a comparable paradox until much later, when pruning is obviously unnecessary anyway.

Their conclusion is narrower than the title suggests, and worth stating fairly: they are arguing against propensity scores for matching, not against propensity scores. The abstract closes by noting that "propensity scores have other productive uses," and their broader position is that matching in general "should remain a highly recommended method of causal inference."

The limit no matching method passes

Suppose you do everything right -- coarsened exact matching, balance verified on every covariate, pruning stopped at the minimum. Your groups are now genuinely comparable on everything you measured.

They may still differ completely on why they adopted.

This is the assumption underneath every matching method: that treatment assignment depends only on observed covariates. In product data that assumption is close to never true. People adopt a feature because a colleague recommended it, because they hit a specific problem that week, because a competitor's outage pushed them, because a new manager mandated a workflow. None of that is in your warehouse. Two accounts identical on every logged field can have arrived at adoption for opposite reasons, and those reasons predict retention directly.

This is the same structural problem as the healthy adherer effect, where users who complete onboarding retain better largely because of who they already were, and it is closely related to immortal time bias, where the adopter group is defined in a way that guarantees a survival advantage. Matching on observables cannot touch any of it. LaLonde's 1986 comparison of non-experimental estimators against an experimental benchmark made this point decades ago, and it was a major reason economists turned toward design-based methods rather than statistical adjustment.

You have two honest routes. Change the design so comparability is not required -- randomise the invitation and use an encouragement design, or build a counterfactual from untreated units with synthetic control. Or measure the selection directly.

The modern approach: measuring the reason for selection

The second route sounds harder than it is, and it is the one teams neglect because it was traditionally slow. The unobserved confounder in nearly every adoption analysis is a reason that exists in a person's memory and nowhere else. It is not hidden in principle. It is simply not in the database, and it can be asked for.

Koji makes this a practical addition rather than a separate research project. Run a short AI-moderated interview across a sample of adopters and matched non-adopters and ask what was happening when they did or did not adopt. The AI moderator follows up on the answer it actually receives, which matters here because the first answer -- "it looked useful" -- is never the real one, and a static form has no way to push past it. Traditional survey tools return the first answer and stop.

Structured questions make the results comparable across the two arms. Koji supports six types, and a selection study uses them in a specific way:

• yes_no to establish the fact that splits the sample: was there a specific triggering event?
• single_choice to classify that trigger into a small set of categories
• scale to measure prior intent, the confounder you most need and least often have
• multiple_choice to capture which alternatives were evaluated at the same time
• ranking to order the drivers when several applied
• open_ended, with follow-up, to catch the trigger category you did not think to list

The analytical payoff is direct. If adopters and non-adopters report the same distribution of triggers and prior intent, your matched comparison is more credible than the covariates alone could establish. If they differ sharply -- and they usually do -- you have measured the confounder, and you can either include it as a covariate or, more honestly, report that the adoption effect is not identified from this data and say what would identify it.

That second outcome feels like a failure and is not. Reporting a bounded, honest null is more valuable than a matched estimate that quietly encodes selection, because the matched estimate will be used to fund a roadmap.

A working checklist

1. Write down the treatment, the outcome, and every covariate you believe drives both.
2. Confirm no covariate is measured after treatment.
3. Prefer coarsened exact matching or Mahalanobis distance matching over propensity score matching.
4. Measure imbalance on the full covariate profile, never on the score alone.
5. Plot imbalance against units pruned and find the turning point.
6. Stop pruning at the minimum; report how many units you dropped and who they were.
7. Name the plausible unobserved confounders explicitly in the writeup.
8. Collect the reason for selection from a sample of both arms.
9. State clearly whether the estimate is identified, or only suggestive.

Frequently asked questions

What is a propensity score?

It is the estimated probability that a unit received the treatment, given its observed characteristics. Rosenbaum and Rubin showed in 1983 that if treatment assignment depends only on observed covariates, comparing units with the same propensity score removes the confounding from those covariates. The score compresses many variables into one number.

What exactly is the PSM paradox?

As you prune observations to improve balance, propensity score matching eventually starts matching units almost at random, because many different covariate profiles share the same score. Past that point, further pruning increases imbalance rather than reducing it. King and Nielsen describe this as PSM often accomplishing the opposite of its intended goal.

How do I tell whether my analysis has hit the paradox?

Plot a multivariate imbalance measure against the number of units pruned, and watch the direction of travel. Well-behaved matching drives imbalance down monotonically for a long stretch. If your curve reaches a minimum and then turns upward, every unit you prune after that point is making the estimate worse.

Should I stop using propensity scores entirely?

No. King and Nielsen are specific that the problem is using propensity scores for matching, and they note that propensity scores have other productive uses. Weighting, stratification, and diagnostics remain reasonable. It is the one-to-one greedy matching workflow that carries the paradox.

Is this the same as propensity weighting in surveys?

No, and the shared word causes confusion. Survey propensity weighting models the probability of responding and corrects a skewed sample. Propensity score matching models the probability of being treated and tries to remove confounding. Different quantity, different problem, different failure modes.

If matching cannot fix selection, what does?

Nothing statistical, if the selection runs on something you never measured. You either need a design that does not require comparability, such as an encouragement design with instrumental variables, or you need to collect the reason for selection directly by asking the people who selected in.

Related Resources

• Synthetic Control -- building a counterfactual when no comparable group exists at all
• Instrumental Variables and Encouragement Designs -- the design that abandons comparability on purpose
• The Healthy Adherer Effect -- why users who complete onboarding always retain better
• Immortal Time Bias -- how the adopter group can be defined into a survival advantage
• Collider Bias -- when adding a control variable creates the correlation
• Structured Questions in AI Interviews -- the six question types and how to use them to measure selection