Market basket analysis finds products that appear in the same transaction more often than chance would predict. It is genuinely useful for merchandising, bundling, and recommendations. What it cannot do is tell you why those products travelled together - and that gap is where most of the money is lost.
This guide covers what the three core metrics actually measure, the two well-documented ways they mislead, why the most famous result in the field never happened the way you were told, and what to do about it.
The short answer
Market basket analysis is a descriptive technique. It measures co-occurrence in a transaction log. Co-occurrence is not preference, not complementarity, and not causation. Two products in one basket can mean the shopper bought them for a single purpose, for two unrelated purposes on the same trip, or for no connected reason at all. The transaction log cannot distinguish these cases, because the distinguishing information - the shopper's intent - was never recorded.
The three metrics, and what each one hides
Association rule mining rests on three numbers for a rule of the form "if a basket contains X, it also contains Y".
| Metric | Formula | What it measures | The trap |
|---|---|---|---|
| Support | P(X and Y) | How often the pair appears at all | Popular items dominate; rare but valuable pairs fall below any sensible threshold |
| Confidence | P(Y given X) | How often Y shows up when X does | Ignores how common Y is on its own |
| Lift | P(X and Y) / (P(X) x P(Y)) | Co-occurrence versus statistical independence | Explodes on rare items that coincided a handful of times |
Confidence is the metric most often quoted to executives, and it is the one that breaks first. As the arules reference documentation puts it, confidence "is sensitive to the frequency of the consequent Y in the database. Caused by the way confidence is calculated, consequents with higher support will automatically produce higher confidence values even if there exists no association between the items."
The classic illustration from the association rule literature makes this concrete. Suppose coffee and tea appear together in 15% of baskets, and 75% of baskets containing coffee also contain tea. A 75% confidence score looks like a strong rule. But if 80% of all baskets contain tea regardless, then buying coffee makes a shopper less likely to buy tea, not more. The lift is 75/80 = 0.94, which is below 1 - a negative association wearing the costume of a strong rule.
Lift, introduced as "interest" by Brin, Motwani, Ullman and Tsur in their 1997 paper Dynamic itemset counting and implication rules for market basket data, was designed to fix exactly this. It has its own failure mode, and the arules documentation is blunt about it: "lift is susceptible to noise in small databases. Rare itemsets with low counts (low probability), which by chance occur a few times (or only once) together, can produce enormous lift values."
So the practical rule is: a high lift on a low-support pair is usually noise, which is why practitioners set minimum support somewhere around 0.01 to 0.05 on retail data and discard everything below it. That threshold is a confession. It means the technique is reliable only where you already have volume, and silent exactly where a new product, a small segment, or an emerging behaviour lives.
The most famous result in the field is a cautionary tale, not a success story
Every introduction to market basket analysis mentions beer and diapers: an analysis supposedly found that men buying diapers on Friday evenings also bought beer, the retailer moved the products together, and sales jumped.
The story traces back to a real 1992 analysis. Karen Heath and her team ran SQL queries against a retailer's Teradata data warehouse and found the correlation. Almost everything bolted on afterwards is decoration. Writing for TDWI, Mark Madsen - who traced the story to its origin - found that attempts to replicate the correlation across other retailers produced inconsistent results, sometimes finding it and sometimes not, and that the finding never had a rigorous statistical basis.
The detail worth keeping is the one about what happened next. In 1997, after reading about the correlation in Chain Store Age, a drug store chain put beer next to diapers in all of its stores. Madsen's observation about this is the single most useful sentence in the whole saga: the retailer actions created the signal they used to validate the signal.
That is a feedback loop, and it is not a historical curiosity. Every recommendation engine, every planogram change, and every "customers also bought" module does the same thing. Once you act on an affinity, you manufacture more of it. The next quarter's data confirms the rule because you built the confirmation into the shelf. This is why co-purchase data cannot audit itself, and why teams end up defending merchandising decisions with evidence they generated by making the decision. For the general form of this problem, see our guide to correlation versus causation in research.
Four shopper realities, one affinity score
Here is the core limitation, stated plainly. These four situations produce the same row in your association rules table.
| What actually happened | What the shopper would say | What the data shows | What you should do |
|---|---|---|---|
| One job. Both items serve a single purpose - pasta and sauce for tonight's dinner | "I was making dinner" | High lift | Bundle, co-locate, cross-promote. The rule is real |
| Two jobs, one trip. A stock-up run that happens to include a birthday card | "Those are unrelated, I just needed both" | High lift | Do nothing. Co-locating them helps nobody |
| Substitute under constraint. The preferred item was unavailable, so a second choice entered the basket | "They were out of my usual" | High lift | Fix availability. Promoting the substitute entrenches the failure |
| Shared driver. Both items are on promotion, in the same aisle, or bought by the same demographic for unconnected reasons | "It was on offer" | High lift | The association belongs to the promotion, not the products |
Row three is the expensive one. A forced substitution looks in scanner data exactly like a preference, and acting on it optimises for the version of your store where the thing the shopper wanted is missing. Our guide to private label versus national brands works through how forced substitution, deliberate value switching, and plain category indifference all collapse into the same scanner event.
Row four is the most common. Two products bought by overlapping populations for unconnected reasons will show elevated lift forever, and no amount of additional transaction data will separate them, because the separating variable was never in the transaction.
A note on the word "lift"
Association rule lift is not the same thing as experiment lift. In an A/B test or an incrementality study, lift is a causal estimate produced by a design with a control group. In market basket analysis, lift is a descriptive ratio against a statistical independence baseline, with no control group anywhere. The two travel under one word and mean genuinely different things. If you want the causal version, see marketing mix modeling versus attribution versus incrementality.
How to recover the missing why
The information market basket analysis needs is not in the transaction log and never will be. It is in the shopper's account of the trip. The practical method is to use the affinity data to select who to talk to, then ask.
- Rank your rules by commercial exposure, not by lift. Which pairs are you already spending money on - shelf position, bundles, recommendation slots?
- Recruit shoppers whose baskets actually contain the pair. This is where transaction data is genuinely irreplaceable: it is an excellent sampling frame and a poor explanation.
- Anchor every question to one specific trip. Not "do you usually buy these together" but "walk me through the trip where you bought both". Memory for a specific episode is far more reliable than a self-report of a habit. The critical incident technique is the formal version of this move.
- Ask the counterfactual. "If the second item had not been there, what would you have done?" This is the single question that separates a real complement from a coincidence, and it does not exist in any dataset.
- Close the loop with structured measurement so you can size what you heard.
Koji is built for exactly this shape of work. You can point an AI-moderated voice interview at a list of shoppers pulled from your own basket data and have every one of them walked through the same trip reconstruction, with the same follow-up logic, without a moderator getting more curious about the interesting-sounding baskets. That consistency matters here more than usual: unequal probing across basket types would manufacture exactly the pattern you are trying to test.
Koji's six structured question types let you attach hard measurement to the open narrative in the same session. Use open_ended for the trip reconstruction, single_choice to classify what the second item was actually for, multiple_choice to capture every driver that applied, scale to rate how deliberate the pairing was, ranking to order the reasons the basket came together, and yes_no for the clean counterfactual - would you have bought it anyway. The closed types give you a detection-independent baseline: if the open-ended themes diverge across basket types but the closed answers do not, you are looking at a probing artifact rather than a real difference.
That is the difference between a rules table with 4,000 rows nobody trusts and a short list of pairs you know the reason for. Ten interviews will not tell you which pairs are frequent. They will tell you which of the four realities above you are actually looking at - and that is the part the algorithm structurally cannot supply.
Where market basket analysis is still the right tool
None of this makes the technique bad. It makes it a hypothesis generator rather than an answer. Use it to find candidate pairs at scale, to prioritise where to look, and to size opportunities once you understand them. Do not use it to explain behaviour, to justify a range decision on its own, or to infer that one product drives another. For the broader question of when to buy the numbers and when to ask the question, see syndicated data versus custom research.
Frequently Asked Questions
What is market basket analysis?
Market basket analysis is a data mining technique that identifies products purchased together in the same transaction more often than statistical independence would predict. It produces association rules scored by support (how often the pair occurs), confidence (how often the second item appears when the first does), and lift (how much the co-occurrence exceeds chance). It is descriptive, not causal.
What is a good lift value in market basket analysis?
A lift above 1 indicates the items co-occur more than chance, and a lift below 1 indicates they co-occur less. But the number alone is not enough. High lift on a pair with very low support is usually noise, because rare itemsets that coincide a few times by chance can produce enormous lift values. Always read lift alongside support, and treat pairs below roughly 0.01 support with suspicion.
Why is confidence a misleading metric?
Confidence ignores how common the consequent is on its own. If 80% of all baskets contain tea, a rule with 75% confidence that predicts tea is actually identifying shoppers who are less likely than average to buy tea. Confidence rises automatically with the popularity of the predicted item, even when no genuine association exists.
Is the beer and diapers story true?
Partly. A real 1992 analysis using SQL queries against a Teradata warehouse did surface the correlation. But the finding lacked a rigorous statistical basis, replication across other retailers was inconsistent, and the widely repeated claims about the sales increase are embellishment. A drug store chain did place beer next to diapers in 1997 after reading about it, which means subsequent data confirming the pattern was partly created by the action taken on it.
Can market basket analysis tell me why customers buy products together?
No. The transaction log records what was bought, not the intent behind it. One job, two unrelated jobs on the same trip, a forced substitution when the preferred item was unavailable, and a shared external driver like a promotion all produce identical association scores. Recovering the reason requires asking shoppers about specific trips.
How do I combine market basket analysis with customer interviews?
Use the transaction data as a sampling frame rather than an explanation. Rank rules by how much money you already have riding on them, recruit shoppers whose real baskets contain those pairs, and interview them about one specific trip rather than their general habits. Ask the counterfactual - what would you have done if the second item was not there - because that question separates genuine complements from coincidence and appears in no dataset.
Related reading
- Private Label vs National Brands: How to Research Why Shoppers Actually Switch
- Shopper Mission Research: Why the Same Customer Is a Different Shopper on Every Trip
- Cross-Shop Analysis: Why No Dataset Contains a Whole Shopping Trip
- NielsenIQ vs Circana vs Numerator: Retail Measurement Data Compared
- Syndicated Data vs Custom Research: When to Buy the Numbers and When to Ask
