Guide
What margin of error does not tell you
Four kinds of survey error that a margin of error leaves out: who the sample covers, who chose not to answer, how questions were worded and ordered, and what happens to precision when you slice results by segment. Margin of error measures random sampling noise and nothing else. Check these four before you trust a tight interval, and work out the margin again for each segment you report.
Where this appears
What's in this guide
- Coverage and selection bias: If the sample over-represents power users, no sample size fixes it.
- Non-response bias: The people who ignore your survey may differ from the people who answer it.
- Question wording and order: A leading or badly ordered question spoils the data before the margin applies.
- Sub-group slicing: Filter to one segment and the real sample shrinks, so the margin grows.
Use this image
Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:
html
<a href="https://www.koji.so/visuals/what-margin-of-error-does-not-tell-you"><img src="https://www.koji.so/visuals/what-margin-of-error-does-not-tell-you/what-margin-of-error-does-not-tell-you.png" alt="List of four survey errors margin of error does not cover: selection bias, non-response bias, question wording and order, and sub-group slicing." width="1200" height="645" /></a>
<p>Source: <a href="https://www.koji.so/visuals/what-margin-of-error-does-not-tell-you">What margin of error does not tell you</a>, Koji</p>
