Reference table
A survey error budget: the cheapest fix for each component
A one-page error budget for product research, built on the total survey error framework. For each of the seven components it gives the cheapest meaningful action, its rough cost and how large the error usually is. Coverage is often the largest and almost never measured, while sampling is usually the smallest and the only one you can compute. Fill it in before fielding and name the components you chose not to spend on.
Where this appears
What's in this reference table
| Component | Cheapest meaningful action | Rough cost | Typical size |
|---|---|---|---|
| Coverage | Add one absent group to the frame: churned, trial-abandoned or never-converted | Hours of list work | Often the largest single error and almost never measured |
| Sampling | Increase n | Linear in budget | Usually the smallest, and the only one you can compute |
| Nonresponse | Vary invite channel and timing, and compare early and late responders | Low | Large when incentives attract one segment |
| Adjustment | Weight on a variable that predicts the outcome, not whatever you happen to have | Low | Silent, and can add error rather than remove it |
| Validity | Write the definition of the construct before the question | Free | Large and invisible, and nothing downstream can fix it |
| Measurement | Pretest wording and let the interviewer probe vague answers | Low with AI moderation | Large on sensitive and effortful questions |
| Processing | Fix the coding scheme and check inter-coder agreement | Low | Small but concentrated in open-ended data |
Use this image
Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:
html
<a href="https://www.koji.so/visuals/survey-error-budget-actions-by-component"><img src="https://www.koji.so/visuals/survey-error-budget-actions-by-component/survey-error-budget-actions-by-component.png" alt="Table of a survey error budget listing the cheapest action, rough cost and typical size of each of the seven total survey error components in product research." width="1200" height="1201" /></a>
<p>Source: <a href="https://www.koji.so/visuals/survey-error-budget-actions-by-component">A survey error budget: the cheapest fix for each component</a>, Koji</p>
