{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-08-14T17:22:18.018Z"},"content":[{"type":"blog","id":"85f646f4-5686-4b37-a23a-1e8f5a542e91","slug":"market-basket-analysis-guide-2026","title":"Market Basket Analysis in 2026: What Co-Purchase Data Proves, and What It Cannot","url":"https://www.koji.so/blog/market-basket-analysis-guide-2026","summary":"Market basket analysis measures co-occurrence in transaction logs using support, confidence and lift. Confidence is misleading because it ignores the base rate of the predicted item, and lift explodes on rare items. Four different shopper realities - one job, two unrelated jobs, forced substitution, and a shared driver like a promotion - produce identical affinity scores, so the technique is a hypothesis generator rather than an explanation. Recovering intent requires trip-anchored interviews with counterfactual questions.","content":"Market basket analysis finds products that appear in the same transaction more often than chance would predict. It is genuinely useful for merchandising, bundling, and recommendations. What it cannot do is tell you **why** those products travelled together - and that gap is where most of the money is lost.\n\nThis guide covers what the three core metrics actually measure, the two well-documented ways they mislead, why the most famous result in the field never happened the way you were told, and what to do about it.\n\n## The short answer\n\nMarket basket analysis is a **descriptive** technique. It measures co-occurrence in a transaction log. Co-occurrence is not preference, not complementarity, and not causation. Two products in one basket can mean the shopper bought them for a single purpose, for two unrelated purposes on the same trip, or for no connected reason at all. The transaction log cannot distinguish these cases, because the distinguishing information - the shopper's intent - was never recorded.\n\n## The three metrics, and what each one hides\n\nAssociation rule mining rests on three numbers for a rule of the form \"if a basket contains X, it also contains Y\".\n\n| Metric | Formula | What it measures | The trap |\n|---|---|---|---|\n| Support | P(X and Y) | How often the pair appears at all | Popular items dominate; rare but valuable pairs fall below any sensible threshold |\n| Confidence | P(Y given X) | How often Y shows up when X does | Ignores how common Y is on its own |\n| Lift | P(X and Y) / (P(X) x P(Y)) | Co-occurrence versus statistical independence | Explodes on rare items that coincided a handful of times |\n\n**Confidence is the metric most often quoted to executives, and it is the one that breaks first.** As the `arules` reference documentation puts it, confidence \"is sensitive to the frequency of the consequent Y in the database. Caused by the way confidence is calculated, consequents with higher support will automatically produce higher confidence values even if there exists no association between the items.\"\n\nThe classic illustration from the association rule literature makes this concrete. Suppose coffee and tea appear together in 15% of baskets, and 75% of baskets containing coffee also contain tea. A 75% confidence score looks like a strong rule. But if 80% of all baskets contain tea regardless, then buying coffee makes a shopper **less** likely to buy tea, not more. The lift is 75/80 = 0.94, which is below 1 - a negative association wearing the costume of a strong rule.\n\nLift, introduced as \"interest\" by Brin, Motwani, Ullman and Tsur in their 1997 paper *Dynamic itemset counting and implication rules for market basket data*, was designed to fix exactly this. It has its own failure mode, and the `arules` documentation is blunt about it: \"lift is susceptible to noise in small databases. Rare itemsets with low counts (low probability), which by chance occur a few times (or only once) together, can produce enormous lift values.\"\n\nSo the practical rule is: **a high lift on a low-support pair is usually noise**, which is why practitioners set minimum support somewhere around 0.01 to 0.05 on retail data and discard everything below it. That threshold is a confession. It means the technique is reliable only where you already have volume, and silent exactly where a new product, a small segment, or an emerging behaviour lives.\n\n## The most famous result in the field is a cautionary tale, not a success story\n\nEvery introduction to market basket analysis mentions beer and diapers: an analysis supposedly found that men buying diapers on Friday evenings also bought beer, the retailer moved the products together, and sales jumped.\n\nThe story traces back to a real 1992 analysis. Karen Heath and her team ran SQL queries against a retailer's Teradata data warehouse and found the correlation. Almost everything bolted on afterwards is decoration. Writing for TDWI, Mark Madsen - who traced the story to its origin - found that attempts to replicate the correlation across other retailers produced inconsistent results, sometimes finding it and sometimes not, and that the finding never had a rigorous statistical basis.\n\nThe detail worth keeping is the one about what happened next. In 1997, after reading about the correlation in *Chain Store Age*, a drug store chain put beer next to diapers in all of its stores. Madsen's observation about this is the single most useful sentence in the whole saga: the retailer actions **created the signal they used to validate the signal**.\n\nThat is a feedback loop, and it is not a historical curiosity. Every recommendation engine, every planogram change, and every \"customers also bought\" module does the same thing. Once you act on an affinity, you manufacture more of it. The next quarter's data confirms the rule because you built the confirmation into the shelf. This is why co-purchase data cannot audit itself, and why teams end up defending merchandising decisions with evidence they generated by making the decision. For the general form of this problem, see our guide to [correlation versus causation in research](/docs/correlation-vs-causation-research).\n\n## Four shopper realities, one affinity score\n\nHere is the core limitation, stated plainly. These four situations produce the same row in your association rules table.\n\n| What actually happened | What the shopper would say | What the data shows | What you should do |\n|---|---|---|---|\n| **One job.** Both items serve a single purpose - pasta and sauce for tonight's dinner | \"I was making dinner\" | High lift | Bundle, co-locate, cross-promote. The rule is real |\n| **Two jobs, one trip.** A stock-up run that happens to include a birthday card | \"Those are unrelated, I just needed both\" | High lift | Do nothing. Co-locating them helps nobody |\n| **Substitute under constraint.** The preferred item was unavailable, so a second choice entered the basket | \"They were out of my usual\" | High lift | Fix availability. Promoting the substitute entrenches the failure |\n| **Shared driver.** Both items are on promotion, in the same aisle, or bought by the same demographic for unconnected reasons | \"It was on offer\" | High lift | The association belongs to the promotion, not the products |\n\nRow three is the expensive one. A forced substitution looks in scanner data exactly like a preference, and acting on it optimises for the version of your store where the thing the shopper wanted is missing. Our guide to [private label versus national brands](/blog/private-label-vs-national-brands-research-2026) works through how forced substitution, deliberate value switching, and plain category indifference all collapse into the same scanner event.\n\nRow four is the most common. Two products bought by overlapping populations for unconnected reasons will show elevated lift forever, and no amount of additional transaction data will separate them, because the separating variable was never in the transaction.\n\n## A note on the word \"lift\"\n\nAssociation rule lift is not the same thing as experiment lift. In an A/B test or an incrementality study, lift is a causal estimate produced by a design with a control group. In market basket analysis, lift is a descriptive ratio against a statistical independence baseline, with no control group anywhere. The two travel under one word and mean genuinely different things. If you want the causal version, see [marketing mix modeling versus attribution versus incrementality](/blog/marketing-mix-modeling-vs-attribution-vs-incrementality-2026).\n\n## How to recover the missing why\n\nThe information market basket analysis needs is not in the transaction log and never will be. It is in the shopper's account of the trip. The practical method is to use the affinity data to **select** who to talk to, then ask.\n\n1. **Rank your rules by commercial exposure, not by lift.** Which pairs are you already spending money on - shelf position, bundles, recommendation slots?\n2. **Recruit shoppers whose baskets actually contain the pair.** This is where transaction data is genuinely irreplaceable: it is an excellent sampling frame and a poor explanation.\n3. **Anchor every question to one specific trip.** Not \"do you usually buy these together\" but \"walk me through the trip where you bought both\". Memory for a specific episode is far more reliable than a self-report of a habit. The [critical incident technique](/docs/critical-incident-technique) is the formal version of this move.\n4. **Ask the counterfactual.** \"If the second item had not been there, what would you have done?\" This is the single question that separates a real complement from a coincidence, and it does not exist in any dataset.\n5. **Close the loop with structured measurement** so you can size what you heard.\n\nKoji is built for exactly this shape of work. You can point an AI-moderated voice interview at a list of shoppers pulled from your own basket data and have every one of them walked through the same trip reconstruction, with the same follow-up logic, without a moderator getting more curious about the interesting-sounding baskets. That consistency matters here more than usual: unequal probing across basket types would manufacture exactly the pattern you are trying to test.\n\nKoji's six [structured question types](/docs/structured-questions-guide) let you attach hard measurement to the open narrative in the same session. Use `open_ended` for the trip reconstruction, `single_choice` to classify what the second item was actually for, `multiple_choice` to capture every driver that applied, `scale` to rate how deliberate the pairing was, `ranking` to order the reasons the basket came together, and `yes_no` for the clean counterfactual - would you have bought it anyway. The closed types give you a detection-independent baseline: if the open-ended themes diverge across basket types but the closed answers do not, you are looking at a probing artifact rather than a real difference.\n\nThat is the difference between a rules table with 4,000 rows nobody trusts and a short list of pairs you know the reason for. Ten interviews will not tell you which pairs are frequent. They will tell you which of the four realities above you are actually looking at - and that is the part the algorithm structurally cannot supply.\n\n## Where market basket analysis is still the right tool\n\nNone of this makes the technique bad. It makes it a **hypothesis generator** rather than an answer. Use it to find candidate pairs at scale, to prioritise where to look, and to size opportunities once you understand them. Do not use it to explain behaviour, to justify a range decision on its own, or to infer that one product drives another. For the broader question of when to buy the numbers and when to ask the question, see [syndicated data versus custom research](/blog/syndicated-data-vs-custom-research-2026).\n\n## Frequently Asked Questions\n\n### What is market basket analysis?\n\nMarket basket analysis is a data mining technique that identifies products purchased together in the same transaction more often than statistical independence would predict. It produces association rules scored by support (how often the pair occurs), confidence (how often the second item appears when the first does), and lift (how much the co-occurrence exceeds chance). It is descriptive, not causal.\n\n### What is a good lift value in market basket analysis?\n\nA lift above 1 indicates the items co-occur more than chance, and a lift below 1 indicates they co-occur less. But the number alone is not enough. High lift on a pair with very low support is usually noise, because rare itemsets that coincide a few times by chance can produce enormous lift values. Always read lift alongside support, and treat pairs below roughly 0.01 support with suspicion.\n\n### Why is confidence a misleading metric?\n\nConfidence ignores how common the consequent is on its own. If 80% of all baskets contain tea, a rule with 75% confidence that predicts tea is actually identifying shoppers who are less likely than average to buy tea. Confidence rises automatically with the popularity of the predicted item, even when no genuine association exists.\n\n### Is the beer and diapers story true?\n\nPartly. A real 1992 analysis using SQL queries against a Teradata warehouse did surface the correlation. But the finding lacked a rigorous statistical basis, replication across other retailers was inconsistent, and the widely repeated claims about the sales increase are embellishment. A drug store chain did place beer next to diapers in 1997 after reading about it, which means subsequent data confirming the pattern was partly created by the action taken on it.\n\n### Can market basket analysis tell me why customers buy products together?\n\nNo. The transaction log records what was bought, not the intent behind it. One job, two unrelated jobs on the same trip, a forced substitution when the preferred item was unavailable, and a shared external driver like a promotion all produce identical association scores. Recovering the reason requires asking shoppers about specific trips.\n\n### How do I combine market basket analysis with customer interviews?\n\nUse the transaction data as a sampling frame rather than an explanation. Rank rules by how much money you already have riding on them, recruit shoppers whose real baskets contain those pairs, and interview them about one specific trip rather than their general habits. Ask the counterfactual - what would you have done if the second item was not there - because that question separates genuine complements from coincidence and appears in no dataset.\n\n## Related reading\n\n- [Private Label vs National Brands: How to Research Why Shoppers Actually Switch](/blog/private-label-vs-national-brands-research-2026)\n- [Shopper Mission Research: Why the Same Customer Is a Different Shopper on Every Trip](/blog/shopper-mission-research-2026)\n- [Cross-Shop Analysis: Why No Dataset Contains a Whole Shopping Trip](/blog/cross-shop-analysis-research-2026)\n- [NielsenIQ vs Circana vs Numerator: Retail Measurement Data Compared](/blog/nielseniq-vs-circana-vs-numerator-2026)\n- [Syndicated Data vs Custom Research: When to Buy the Numbers and When to Ask](/blog/syndicated-data-vs-custom-research-2026)\n","category":"Research","lastModified":"2026-08-14T03:21:47.436547+00:00","metaTitle":"Market Basket Analysis in 2026: What Co-Purchase Data Proves and What It Cannot","metaDescription":"Market basket analysis finds products bought together but never why. Learn what support, confidence and lift really measure, why the beer-and-diapers story is a cautionary tale, and how to recover the missing intent.","keywords":["market basket analysis","association rules","lift metric","co-purchase analysis","affinity analysis","apriori algorithm","retail analytics","shopper research"],"aiSummary":"Market basket analysis measures co-occurrence in transaction logs using support, confidence and lift. Confidence is misleading because it ignores the base rate of the predicted item, and lift explodes on rare items. Four different shopper realities - one job, two unrelated jobs, forced substitution, and a shared driver like a promotion - produce identical affinity scores, so the technique is a hypothesis generator rather than an explanation. Recovering intent requires trip-anchored interviews with counterfactual questions.","aiKeywords":["market basket analysis","association rules","lift","support","confidence","co-purchase","shopper intent","retail analytics"],"aiContentType":"guide","faqItems":[{"answer":"Market basket analysis is a data mining technique that identifies products purchased together in the same transaction more often than statistical independence would predict. It produces association rules scored by support, confidence and lift. It is descriptive, not causal.","question":"What is market basket analysis?"},{"answer":"Lift above 1 means items co-occur more than chance and below 1 means less. But high lift on a very low support pair is usually noise, because rare itemsets that coincide a few times by chance produce enormous lift values. Always read lift alongside support.","question":"What is a good lift value in market basket analysis?"},{"answer":"Confidence ignores how common the predicted item is on its own. If 80 percent of baskets contain tea, a rule with 75 percent confidence predicting tea actually identifies shoppers less likely than average to buy it. Confidence rises with the popularity of the consequent even when no association exists.","question":"Why is confidence a misleading metric?"},{"answer":"Partly. A real 1992 analysis surfaced the correlation using SQL queries against a Teradata warehouse, but it lacked a rigorous statistical basis and replication was inconsistent. A drug store chain placed beer next to diapers in 1997 after reading about it, so later confirming data was partly created by the action taken on it.","question":"Is the beer and diapers story true?"},{"answer":"No. One job, two unrelated jobs on the same trip, a forced substitution when the preferred item was out of stock, and a shared driver like a promotion all produce identical association scores. Recovering the reason requires asking shoppers about specific trips.","question":"Can market basket analysis tell me why customers buy products together?"},{"answer":"Use transaction data as a sampling frame rather than an explanation. Rank rules by commercial exposure, recruit shoppers whose actual baskets contain the pair, and interview them about one specific trip. Ask the counterfactual question about what they would have done without the second item.","question":"How do I combine market basket analysis with customer interviews?"}],"relatedTopics":["market basket analysis","association rules","retail analytics","shopper research","co-purchase","customer interviews"]}],"pagination":{"total":1,"returned":1,"offset":0}}