Case-Control Research: How to Study Churn and Lost Deals Without Fooling Yourself (2026)

Answer first: when you interview thirty customers who churned and look for the reason, you are running a case-control study - a design that starts from an outcome and works backwards to find its causes. Epidemiology has been running these since 1950 and has documented exactly how they fail. The single biggest failure is the one product teams make by default: having no control group. Without a comparison group of customers who did not churn, every reason your churned customers give is unfalsifiable, because the customers who stayed would very likely have given you the same reasons.

This guide covers the design properly - what it can establish, what it structurally cannot, and how to run one that survives scrutiny.

What a case-control study actually is

Most research designs are prospective: you take a group, do something or observe them, and wait to see what happens. A case-control study runs the other way. You start with people who already have the outcome (the cases), assemble a comparable group who do not (the controls), and compare their histories to find exposures that differ.

This is the only feasible design when the outcome is rare, expensive, or slow. You cannot randomly assign customers to churn. You cannot wait three years to see which enterprise deals are lost. So you sample on the outcome and reason backwards.

Product research does this constantly and almost never names it:

In every row, the right-hand column is the study. The left-hand column on its own is a collection of anecdotes about people who share an outcome.

The 1950 study that invented the design, and what it got right

The founding modern case-control study is Doll and Hill, "Smoking and carcinoma of the lung: preliminary report", British Medical Journal 2(4682):739-748, published 30 September 1950. It is worth reading not for the conclusion, which everyone knows, but for the protocol, which is a masterclass in the thing product teams skip.

Doll and Hill interviewed 649 men and 60 women with carcinoma of the lung. Among the men, 0.3 percent were non-smokers; among the women, 31.7 percent. Those are the cases. The design work was all in the controls.

Four almoners engaged wholly on research conducted every interview using, in the paper's own words, "a set questionary" - one standardised instrument, applied identically to both arms. For each lung-carcinoma patient, the almoners were instructed to interview a patient of the same sex, within the same five-year age group, and in the same hospital at or about the same time. Where more than one suitable patient was available, the choice fell on the first in the ward list the ward sister considered fit for interview.

Read that matching rule again, because it is doing four separate jobs. Same sex and age group removes two confounders by construction. Same hospital means the controls are drawn from the same source population as the cases - the same catchment, the same referral patterns, the same social composition. Same time removes seasonal and period effects. And a fixed selection procedure removes the interviewer's discretion about who becomes a control.

The authors were also explicit about the threat they could not fully close. On the risk that patients who escaped notification were systematically different, they reasoned that this was unlikely to bias the inquiry because the points of interest were either unknown or known only in broad outline to the staff doing the notifying. And where the matching rule had to be relaxed - at two specialist hospitals where a same-hospital control was not always available - those records were analysed separately.

That is the standard. A 1950 study, run with paper forms and four staff, matched its controls on three variables, standardised its instrument across both arms, reasoned explicitly about selection, and segregated the data where its own protocol broke down. A 2026 churn study typically does none of these things.

Why your churn study is unfalsifiable without controls

Here is the failure in its simplest form. You interview thirty churned customers. Twenty-two mention price. You report that price is the leading driver of churn.

Now ask the question the design was supposed to answer: what fraction of customers who renewed would also have mentioned price? If it is also around 70 percent, price explains nothing about churn. It is a fact about your market, not a cause of departure. You have measured the prevalence of a complaint, not its association with an outcome.

This is not a hypothetical failure mode. Complaints about price, missing features, and confusing navigation are close to universal in every customer population ever sampled, including populations with excellent retention. Any explanation that is equally consistent with both outcomes has no diagnostic value - a point developed at length in our guide to analysis of competing hypotheses, where the ordering principle is how much a piece of evidence separates explanations rather than how strongly it supports the favoured one.

The remedy is structural, not analytical. You cannot fix a missing control group in the analysis. You have to go and interview the people who stayed, with the same instrument, at the same time.

How to choose controls, which is the entire design

Almost every serious criticism of a case-control study is a criticism of its control group.

The tenure match deserves emphasis because it is the one product teams get wrong most often. If your churned customers averaged 8 months of tenure and your comparison group averaged 30 months, then any difference you find may simply be the difference between new and established customers. You have matched on nothing and confounded everything.

Berkson bias: when your controls are not a fair comparison

In 1946 Joseph Berkson published "Limitations of the application of fourfold table analysis to hospital data" (Biometrics Bulletin 2(3):47-53), showing that when both cases and controls are drawn from a population that has already been filtered - in his case, hospital admission - a spurious association can appear between two conditions that are unrelated in the general population. The filter itself manufactures the correlation.

This is the sharpest single criticism of Doll and Hill's design: their controls were hospital patients, who are by definition ill. It happens not to have overturned their conclusion, but the objection was structurally valid.

The product version is everywhere and almost never noticed. Your controls are usually people who responded to a research invitation. That is a filter, and it selects for engagement, goodwill, and available time - exactly the traits associated with not churning. Comparing churned customers against a control group of enthusiastic respondents will find differences that are artifacts of the recruitment channel rather than causes of churn.

This is the mirror image of the problem covered in survivorship bias, which is about the cases you never reach. Berkson bias is about the controls you reach too easily. A study can suffer from both simultaneously.

Two threats the design carries by construction

Recall bias. Because both arms are being asked to remember, and only one arm has experienced a memorable outcome, the two arms remember differently. Someone who cancelled has narrativised the decision; someone who renewed has not thought about it. This is the "effort after meaning" effect: people who experienced a notable outcome search their memory harder for causes, so cases can appear to report more exposures than controls purely as an artifact of motivated recall. Our guide to recall bias covers the mitigations, of which shortening the gap between event and report is by far the most effective.

Confounding by indication. In medicine this describes the situation where the reason a treatment was given is itself associated with the outcome, so treated patients look worse. The product analogue: customers who were given a discount, assigned a customer success manager, or enrolled in a save programme were selected for those interventions because they were already at risk. Comparing them naively makes the intervention look harmful. If your analysis concludes that customers who spoke to support churned more, check whether support contact is a marker of trouble rather than a cause of it.

Base rates: why your at-risk flag is usually wrong

Case-control logic gives you an association. Turning that into a prediction runs into a separate problem that is arithmetic rather than methodological, and it is the one that most often embarrasses a research team in front of an executive.

Suppose your annual churn rate is 5 percent and you build a churn-risk model that is 90 percent accurate in both directions - it correctly flags 90 percent of customers who will churn, and correctly clears 90 percent of those who will not. Out of 1,000 customers:

Of the 140 customers the model flags, only 45 actually churn. The positive predictive value is 32 percent - the flag is wrong about two times out of three, despite 90 percent accuracy on both arms. Nothing is broken. The base rate is low, so the far larger healthy population contributes more false positives than the small at-risk population contributes true ones.

Two consequences follow. First, do not evaluate a screener or a risk model on accuracy; evaluate it on positive predictive value at your actual base rate. Second, when you recruit research participants using such a flag, remember that most of them are not the population you think you sampled - which contaminates the case group of your next case-control study.

What a case-control study cannot tell you

Being honest about the design's limits is what makes its conclusions credible.

• It cannot give you incidence. Because you chose how many cases and how many controls to sample, the ratio between them is an artifact of your recruiting. You can estimate the odds ratio - how much more common an exposure is among cases - but not the churn rate itself.
• It cannot establish temporal order on its own. Did the reduced usage cause the churn, or did the decision to leave cause the reduced usage? Only careful questioning about sequence can separate these, and memory is an unreliable narrator about sequence.
• It cannot rule out unmeasured confounders. Matching handles the confounders you thought of. This limitation is shared with all non-randomised designs; see quasi-experimental design for the prospective counterpart and the conditions under which it earns a causal reading.
• It is not a substitute for an experiment. It is what you use when an experiment is impossible, which for churn it usually is.

The modern approach: how Koji helps

The reason product teams skip the control group is almost never that they think it is unnecessary. It is that it doubles the recruiting and scheduling burden on a study that was already hard to staff. Halving the cost per interview is therefore not a convenience; it is what makes the correct design affordable.

Both arms, same week, same instrument. Koji runs AI-moderated interviews in parallel, so fielding a matched control arm alongside the case arm is a scheduling change rather than a second project. That directly restores Doll and Hill's "at or about the same time" condition, which is the one most often lost when the comparison group is added later as an afterthought.

The same questionary, enforced rather than intended. The 1950 study depended on four trained almoners applying one instrument consistently. An AI moderator applies the identical protocol to interview 3 and interview 300, in both arms, which removes the interviewer-variation threat that makes case and control data non-comparable. It also removes a subtler problem: a human moderator who knows they are talking to a churned customer probes for reasons to leave.

Structured questions make the two arms directly comparable. Koji supports six question types - open_ended, scale, single_choice, multiple_choice, ranking, and yes_no - each carrying a stable question ID from plan through analysis into the report. That stability is what lets you put the case arm and the control arm side by side on the same item: the proportion of leavers who selected a reason in a multiple_choice item against the proportion of stayers who selected it, or a scale distribution compared across arms. ranking is particularly useful here, because forcing both arms to order the same set of factors surfaces differences that open-ended complaint counts hide. See the structured questions guide for how each type is aggregated.

Reaching the leavers at all. Churned customers are the hardest group to recruit and the fastest to become unreachable. Asynchronous AI-moderated interviews that a former customer can complete in ten minutes, at any hour, without booking a call with the vendor they just left, materially change response rates on the arm that matters most - see churned customer interviews for the recruiting approach.

Automatic thematic analysis across both arms at once. The comparison you need - which themes are differentially present in cases versus controls - is a cross-tabulation, not a reading exercise. Producing it automatically is the difference between a comparison you actually run and one you intended to run.

What Koji does not do is choose your controls. Matching on tenure, plan, and cohort is a design judgement, and it is the judgement the whole study rests on.

Frequently asked questions

Do I really need to interview customers who stayed?

Yes, and it is the single highest-value change you can make to a churn programme. Without a comparison group you cannot tell a reason for leaving from a universal complaint. If 70 percent of leavers mention price and 70 percent of stayers would too, price explains nothing about churn - and you will not know that until you ask them.

How many controls should I recruit per case?

One-to-one matching is the simplest and is what Doll and Hill used. Recruiting two or three controls per case increases statistical precision, with diminishing returns past about four. If controls are much easier to reach than cases, which is usual in churn research, take the extra ratio.

What should I match controls on?

Match on the variables you already believe drive the outcome and can measure: tenure, plan tier, company size, acquisition channel, and onboarding cohort. Tenure is the one teams most often forget, and forgetting it means any difference you find may just be the difference between newer and older customers.

Is win-loss analysis a case-control study?

Structurally, yes - lost deals are the cases and won deals are the controls. Win-loss practice is somewhat ahead of churn practice here because interviewing both winners and losers is a common convention. The gap that remains is matching: comparing lost enterprise deals against won small-business deals confounds deal size with outcome.

Why is my 90 percent accurate churn model wrong most of the time it fires?

Because churn is rare. At a 5 percent base rate, a model that is 90 percent accurate on both arms produces roughly 45 true positives and 95 false positives per 1,000 customers, giving a positive predictive value near 32 percent. Judge screeners and risk models on positive predictive value at your real base rate, never on accuracy.

Can a case-control study prove that something causes churn?

No design that samples on the outcome can prove causation on its own. A well-matched case-control study with a standardised instrument and a plausible temporal sequence gives you a strong, testable association - which is the right input to an experiment, not a replacement for one.

Related Resources

• Churned Customer Interviews - reaching and interviewing the case arm, which is the hard half of the recruiting
• Survivorship Bias in Customer Research - the mirror problem: the cases you never reach at all
• Recall Bias - why cases and controls remember differently, and how to shorten the gap
• Quasi-Experimental Design - the prospective counterpart, for when you start from an intervention instead of an outcome
• Analysis of Competing Hypotheses - sorting evidence by how much it separates explanations rather than how much it supports one
• Structured Questions in AI Interviews - the six question types that make two arms directly comparable