{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-10-22T07:57:22.949Z"},"content":[{"type":"documentation","id":"c9804f3e-61d8-44ca-8b62-d9a59b05b380","slug":"self-selection-bias","title":"Self-Selection Bias: Why the People Who Answer Aren't the People You Need","url":"https://www.koji.so/docs/self-selection-bias","summary":"Self-selection bias occurs when people choose whether to take part in research and those who opt in differ from those who don't. The 1936 Literary Digest poll mailed about 10 million ballots, got about 2.4 million back and predicted a Landon win; Roosevelt won with about 61%, and Squire (1988) found Landon supporters were more likely to respond. Hu, Pavlou and Zhang (2017) traced the J-shaped distribution of online reviews to acquisition and under-reporting biases. Pew Research Center (2024) found 12% of under-30 opt-in respondents claimed a nuclear submarine licence. Heckman (1979) developed a two-step correction. The guide separates self-selection from sampling and nonresponse bias, lists six places it hides in product research, gives a six-step process to reduce it and shows how Koji's personal invite links, outbound calls, voice or text answering, screening, panel targeting and quality scores help.","content":"Self-selection bias happens when people decide for themselves whether to take part in your research, and the ones who opt in differ from the ones who don't in ways that matter to your question. The feature-request board is full of power users. The NPS survey is answered by fans and people who are furious. The open call for interviews attracts the curious, the chatty and the people who want the gift card. Then the results get read as if they speak for everyone.\n\nYou can't remove self-selection completely, because nobody can be made to take part. You can shrink it by inviting deliberately, making it easy for quiet people to answer and checking who is missing before you report. This guide covers what self-selection bias is, the best-known examples, where it hides in everyday product research and a six-step process to reduce it.\n\n## What self-selection bias is\n\nIn a self-selected sample, membership is decided by the participant, not the researcher. The bias appears when the reasons people choose to join are linked to the thing you're measuring.\n\nIf you ask \"how satisfied are you with our onboarding?\" in an optional in-app survey, the people most likely to answer are those with strong feelings, the very engaged and those who happen to have a spare minute. If satisfaction is linked to any of those, and it usually is, the average you get is wrong in a predictable direction.\n\nIt is closely related to two other problems, and the differences help when choosing a fix:\n\n- **Sampling bias** is about who you invite. If your invite list leaves out a group, they can't be in the data however keen they are. See [sampling bias](/docs/sampling-bias-research).\n- **[Nonresponse bias](/docs/nonresponse-bias)** is about who, among the people you invited, doesn't answer. It is a form of self-selection that starts from a list you chose.\n- **Self-selection bias** is the broader case where people put themselves forward, often through an open link, a widget or a public post, with no invite list at all.\n\nThe more open the door, the stronger the self-selection. An open link shared on social media sits at one end. A random sample of customers who are each personally invited and followed up sits at the other.\n\n## The classic example: the 1936 Literary Digest poll\n\n_The Literary Digest_ mailed about 10 million ballots before the 1936 US presidential election and got roughly 2.4 million back. That's an enormous sample. It predicted a clear win for Alf Landon. Franklin Roosevelt won with about 61% of the vote.\n\nFor decades the standard explanation was a skewed list, built partly from telephone directories and car registrations. Peverill Squire's 1988 analysis of a 1937 Gallup survey pointed at something else as well: Landon supporters were more likely to send their ballots back than Roosevelt supporters. The people who chose to answer were not like the people who didn't, and 2.4 million responses couldn't fix that.\n\nThe lesson for product teams is that volume does not cure self-selection. A thousand self-selected survey responses can be more misleading than twenty interviews with people you chose on purpose.\n\n## A modern example: online reviews\n\nStar ratings are a self-selected sample by design. Nan Hu, Paul Pavlou and Jie Zhang studied why so many products on Amazon show a J-shaped rating pattern: lots of five-star reviews, some one-star reviews and few in the middle. Their work, published in _MIS Quarterly_ in 2017, describes two self-selection effects:\n\n- **Acquisition bias.** Only people who expected to like the product bought it, so only they can review it.\n- **Under-reporting bias.** Among buyers, people with very strong experiences, good or bad, are much more likely to write a review than people who thought it was fine.\n\nWhen they asked every participant in an experiment to rate a product, the ratings came out roughly normally distributed. The J shape came from who chose to write, not from the product. The same pattern shows up in app store ratings, support satisfaction scores and feature-voting boards.\n\n## Opt-in panels and bogus respondents\n\nPaid opt-in panels add a different kind of self-selection: people who join to earn rewards. Most are honest, but some answer as fast as possible and say whatever gets them through the screener.\n\nPew Research Center tested this in a 2022 survey experiment, published in 2024. They asked respondents whether they were licensed to operate a class SSGN nuclear submarine. The true share of Americans with that licence rounds to 0%. In the opt-in samples, 12% of adults under 30 said yes. Among opt-in respondents who said they were Hispanic, 24% claimed the licence, compared with 2% of non-Hispanic opt-in respondents. In Pew's probability-based panels, very few people in any group gave answers like that.\n\nThe point isn't that panels are useless. It's that the reasons people sign up, and the reasons they answer a given study, shape who you hear from. [Screener questions](/docs/research-screener-questions) and quality checks matter more the more open the recruitment.\n\n## Where self-selection hides in product research\n\nDaniel Kahneman described a habit of fast thinking with the phrase \"What you see is all there is.\" We judge from the evidence in front of us and rarely ask what's missing. Self-selected research feeds that habit, because the missing people leave no trace.\n\n![List of six common places self-selection bias hides in product research, from feature request boards to churn surveys](/visuals/self-selection-bias-product-research-sources/self-selection-bias-product-research-sources.png \"If people chose to be in the data, ask who chose not to before you act on it\")\n\n1. **Feature-request boards.** Dominated by power users and the most vocal accounts. The silent majority's needs never get a post.\n2. **Optional in-app surveys.** Answered by the engaged and the annoyed. People in a hurry, often your most efficient users, skip them.\n3. **NPS and CSAT.** Response rates are often low, and people with strong feelings answer most. See [NPS follow-up interviews](/docs/nps-follow-up-interviews) for ways to dig into the scores.\n4. **Open calls for interviews.** A post asking for volunteers attracts the confident, the talkative and people with time on their hands.\n5. **Beta programmes.** Early adopters sign up because they love new things, so they forgive rough edges your mainstream users won't.\n6. **Churn and exit surveys.** People who leave angry fill them in. People who quietly drifted away rarely do, and they are often the larger group.\n\nReviews, support tickets and social mentions have the same problem. They are [spontaneous rather than solicited feedback](/docs/solicited-vs-spontaneous-feedback): useful for spotting issues, risky for estimating how common they are.\n\n## How to reduce self-selection bias\n\nYou can't force participation, so the work is in who you invite, how easy you make it and how honestly you report.\n\n![Six-step process to reduce self-selection bias, from defining the population to reporting who is missing](/visuals/reduce-self-selection-bias-process/reduce-self-selection-bias-process.png \"Most of the fix happens before the first answer comes in\")\n\n1. **Define who you need to hear from.** Write the population down: \"everyone who cancelled in the last 30 days\", not \"some churned users\".\n2. **Invite from the full list.** Send a personal invitation to everyone in that population, or a random sample of it, instead of posting an open link.\n3. **Lower the effort to answer.** Short, mobile-friendly, self-paced formats, and a choice of voice or text, bring in people who would never book a call.\n4. **Follow up with the quiet ones.** One or two reminders, or a different channel such as a phone call, reach people who ignored the first ask.\n5. **Compare respondents with the list.** Check plan, tenure, usage and segment against the full population to see who is over- or under-represented.\n6. **Weight or report the gap.** If a group is under-represented, [weight the results](/docs/survey-weighting-guide) or say plainly that the findings don't cover them.\n\nStatistical corrections exist. James Heckman's 1979 paper in _Econometrica_ set out a two-step method for correcting sample selection bias, part of the work that earned him a share of the 2000 Nobel Memorial Prize in Economic Sciences. But corrections need information about the people who didn't take part, which is why inviting from a known list matters so much.\n\n## Reaching the people who stay quiet\n\nThe hardest part of self-selection bias is human, not statistical. The people you most need, the busy, the disengaged, the ones who left without a word, are exactly the ones least likely to raise their hand.\n\n![A customer reading an invitation on their phone in a quiet moment, the kind of person who rarely volunteers feedback](https://images.unsplash.com/photo-1516210938721-0703012a297a?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3w5NDUzMDN8MHwxfHNlYXJjaHwyfHxwZXJzb24lMjByZWFkaW5nJTIwcGhvbmUlMjBtZXNzYWdlJTIwY2FmZXxlbnwxfDB8fHwxNzkyNjQ4MjExfDA&ixlib=rb-4.1.0&q=80&w=1080)\n\n_Photo by [Andrej Lišakov](https://unsplash.com/@lishakov?utm_source=koji&utm_medium=referral) on [Unsplash](https://unsplash.com/?utm_source=koji&utm_medium=referral)_\n\nA few habits help. Make the invitation personal and specific about why their view matters. Say how long it takes and keep that promise. Offer an [incentive](/docs/research-participant-incentives) that is fair for the time, but not so large that it becomes the main reason to join. Offer more than one way to answer: some people will type at midnight but never join a video call, and some would rather talk on the phone than write.\n\nThen look at who replied. If every respondent is a heavy user from your biggest plan, you've learned about that group, not your customers.\n\n## How Koji helps\n\nKoji runs AI-moderated interviews by voice or text. It doesn't stop people choosing whether to answer, but it makes the invite-from-a-list approach practical and makes answering easier for people who would skip a survey.\n\n- **Invite everyone on a list, not an open link.** Import a CSV of contacts and each person gets their own interview link with their details filled in, so you can see who answered and who didn't.\n- **Reach people who ignore email.** Outbound calls let Koji phone each person on a list to run the interview by voice, and the call link lets people type their number and get a call back.\n- **Make answering easy.** Participants pick voice or text on the welcome screen and answer at their own pace on their own device, with no call to book.\n- **Screen on what matters.** Screening questions run before every interview, whatever route someone came in by, and people who don't qualify are turned away.\n- **Recruit with targeting when you need new people.** Panel recruitment lets you describe the audience, add a screening question and see the margin of error your sample size buys before launch.\n- **Keep low-effort answers out of the report.** Every interview gets a quality score from 1 to 5, and only interviews scoring 3 or more enter the report.\n\n![Koji CSV import dialog creating personalised interview links for three respondents](/email/shots/shot-csv-import.jpg \"Importing the full list means you can see who hasn't answered yet, not just who has\")\n\n## Related Resources\n\n- [Nonresponse bias](/docs/nonresponse-bias)\n- [Sampling bias in research](/docs/sampling-bias-research)\n- [Survivorship bias in customer research](/docs/survivorship-bias-customer-research)\n- [Survey weighting guide](/docs/survey-weighting-guide)\n- [Solicited vs spontaneous feedback](/docs/solicited-vs-spontaneous-feedback)\n- [Structured questions guide](/docs/structured-questions-guide)\n\n## Sources\n\n- Heckman, J. J. (1979). Sample selection bias as a specification error. _Econometrica_, 47(1), 153–162.\n- Hu, N., Pavlou, P. A., & Zhang, J. (2017). On self-selection biases in online product reviews. _MIS Quarterly_, 41(2), 449–471.\n- Kahneman, D. (2011). _Thinking, Fast and Slow_. Farrar, Straus and Giroux.\n- Pew Research Center (2024, March 5). Online opt-in polls can produce misleading results, especially for young people and Hispanic adults.\n- Squire, P. (1988). Why the 1936 Literary Digest poll failed. _Public Opinion Quarterly_, 52(1), 125–133.\n","category":"Research Methods","lastModified":"2026-10-22T05:50:11.432836+00:00","metaTitle":"Self-Selection Bias: Examples and How to Reduce It (2026)","metaDescription":"What self-selection bias is, how it differs from nonresponse bias, the Literary Digest and online review examples, and six steps to reduce it in research.","keywords":["self-selection bias","self-selection bias examples","voluntary response bias","self-selection bias in surveys","self-selection vs nonresponse bias","opt-in sample bias","literary digest poll"],"aiSummary":"Self-selection bias occurs when people choose whether to take part in research and those who opt in differ from those who don't. The 1936 Literary Digest poll mailed about 10 million ballots, got about 2.4 million back and predicted a Landon win; Roosevelt won with about 61%, and Squire (1988) found Landon supporters were more likely to respond. Hu, Pavlou and Zhang (2017) traced the J-shaped distribution of online reviews to acquisition and under-reporting biases. Pew Research Center (2024) found 12% of under-30 opt-in respondents claimed a nuclear submarine licence. Heckman (1979) developed a two-step correction. The guide separates self-selection from sampling and nonresponse bias, lists six places it hides in product research, gives a six-step process to reduce it and shows how Koji's personal invite links, outbound calls, voice or text answering, screening, panel targeting and quality scores help.","aiPrerequisites":["Basic familiarity with surveys or customer interviews"],"aiLearningOutcomes":["Define self-selection bias and tell it apart from sampling and nonresponse bias","Explain the Literary Digest and online review examples","Recognise where self-selection hides in product research","Apply a six-step process to reduce self-selection bias","Report findings honestly when some groups are missing"],"aiDifficulty":"beginner","aiEstimatedTime":"9 min read"}],"pagination":{"total":1,"returned":1,"offset":0}}