All frameworks and cheat sheets

Key figure

eBay's search ads: regression said 4,100% ROI, the experiment said -63%

Stat card: eBay's randomized experiment on non-brand search ads found an ROI of minus 63%, where regression estimates had put it above 4,100%.
The sign was wrong, not just the size. That's the gap between attribution and a real control group. Source: Blake, Nosko and Tadelis, Econometrica, 2015
Download PNGRead the full guide

A key figure from Blake, Nosko and Tadelis's study of eBay's paid search, published in Econometrica in 2015. Conventional regression put the return on non-brand search ads above 4,100% with no controls and above 1,400% with time and geographic controls. A randomized experiment on the same campaigns found an ROI of minus 63%, with a 95% confidence interval of minus 124% to minus 3%.

Where this appears

What's in this key figure

-63% ROI of eBay's non-brand search ads in a randomized experimentRegression estimates of the same spend came out above 4,100% with no controls and above 1,400% with time and geographic controls.

Use this image

Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:

html
<a href="https://www.koji.so/visuals/ebay-search-ads-experiment-roi-stat"><img src="https://www.koji.so/visuals/ebay-search-ads-experiment-roi-stat/ebay-search-ads-experiment-roi-stat.png" alt="Stat card: eBay's randomized experiment on non-brand search ads found an ROI of minus 63%, where regression estimates had put it above 4,100%." width="1200" height="630" /></a>
<p>Source: <a href="https://www.koji.so/visuals/ebay-search-ads-experiment-roi-stat">eBay's search ads: regression said 4,100% ROI, the experiment said -63%</a>, Koji</p>

Related

Table of seven marketing questions and which of attribution, MMM, incrementality testing and customer research can answer each, from budgets to why.

What attribution, MMM, incrementality and research can prove

Table comparing multi-touch attribution, marketing mix modeling and incrementality testing on data needs, offline media, identity loss, speed and causality.

Multi-touch attribution vs marketing mix modeling vs incrementality

Stat card: shoppers were about 2.1 times more responsive to price changes than to size changes, a price elasticity of -1.19 against a size elasticity of 0.56.

Shoppers react 2.1 times more to price than to pack size