{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-11-12T07:03:47.691Z"},"content":[{"type":"documentation","id":"8ba47a89-244c-46cb-b931-43b192fde346","slug":"why-are-my-survey-responses-low-quality","title":"Why Are My Survey Responses Low Quality? A Diagnostic for Thin Answers","url":"https://www.koji.so/docs/why-are-my-survey-responses-low-quality","summary":"Low-quality survey and interview responses usually come from one of six causes: the wrong people, an incentive that rewards speed, a study that is too long, vague or leading questions, a format that cannot ask follow-ups, or short answers that are actually fine. Diagnose by where the thin answers sit: in a few people (recruiting), at the same questions (wording), later in the study (length, per Galesic and Bosnjak 2009), or everywhere (format). Pew found 12% of opt-in respondents under 30 claimed a nuclear submarine licence, and an AI agent passed 99.8% of attention checks (Westwood 2025), though a Prolific-funded study found under 1% of about 4,800 responses likely AI-written. Koji scores each interview 1 to 5, leaves Shallow ones out of the report and uncharged, and asks follow-up questions on thin answers.","content":"Survey and interview responses are usually low quality for one of six reasons: you asked the wrong people, the incentive attracted the wrong effort, the study is too long, the questions are vague or leading, the format can't draw out detail, or nobody asked a follow-up. You find which one by looking at _where_ the thin answers sit: in a few people, at a few questions, late in the study, or everywhere.\n\n**Try it now:** export your responses, sort by answer length, and read the shortest twenty side by side. Then use the decision rule below. It takes about fifteen minutes and usually names the cause.\n\nThis guide treats thin answers as a signal about your study design rather than a verdict on your participants. Detection (attention checks, speeders, fraud screening) is covered in [Survey Data Quality](/docs/survey-data-quality-guide) and [Survey Fraud and Respondent Quality](/docs/survey-fraud-respondent-quality). This page is about the step after that: working out _why_ real people gave you so little, and what to change before the next round.\n\n## The decision rule: where are the thin answers?\n\nLow-quality responses leave a pattern, and the pattern points to the cause. Before you rewrite anything, sort your thin answers into one of these shapes.\n\n| If the thin answers are...                           | The likely cause is...               | Change this first                 |\n| ---------------------------------------------------- | ------------------------------------ | --------------------------------- |\n| Concentrated in a few people, across every question  | Who you asked, or why they took part | Source, screener, incentive       |\n| Spread across most people, at the same few questions | Those questions                      | Wording                           |\n| Getting shorter the further into the study people go | Length                               | Cut and reorder                   |\n| Short everywhere but on topic                        | Possibly nothing                     | Read them before you discard them |\n| Thin everywhere and off topic                        | The format                           | Add follow-ups or change the mode |\n\nThe logic is the same one survey methodologists use for drop-off: a problem concentrated in one place indicts that place, and a problem spread evenly indicts the whole design. (The same reasoning applies to people who leave halfway, covered in [Partial Interviews and Breakoff](/docs/partial-interviews-breakoff-analysis).)\n\n![Flowchart for diagnosing thin survey answers: four yes or no questions on who, which questions, how late and relevance, ending in follow-ups.](/visuals/why-survey-answers-are-thin-flowchart/why-survey-answers-are-thin-flowchart.png \"Work through these four questions before you rewrite your survey or blame your participants.\")\n\n### A worked example\n\nA founder runs a 14-question churn survey and gets 200 responses. About 40 of them are one or two words per open question (\"too expensive\", \"ok\", \"n/a\").\n\n- **Step 1, who.** She sorts by respondent. Eleven people account for most of the \"n/a\" answers, and all eleven came from a paid panel rather than her own customer list. That is a source problem: those eleven probably never used the product.\n- **Step 2, which questions.** Across everyone else, question 6 (\"How do you feel about our pricing and support?\") gets the shortest answers. It is a double-barrelled question, and people answer whichever half is easier, briefly.\n- **Step 3, how late.** Answers to questions 11 to 14 average a third of the length of questions 1 to 4. The survey is too long for what she is asking.\n- **Step 4, on topic.** Many of the remaining short answers (\"switched to a spreadsheet\", \"boss cancelled the budget\") are short but useful. She keeps them.\n\nThree fixes follow: drop the panel and invite only real churned customers, split question 6 in two, and cut the survey to eight questions with the most important ones first. None of these is \"find better participants\" in the abstract.\n\n## Cause 1: You asked the wrong people\n\nIf a handful of people give thin or nonsense answers to everything, start with your source. People who are not in your audience have nothing to say about your problem, and in opt-in panels some respondents are not who they claim to be.\n\nPew Research Center tested this directly by asking opt-in respondents whether they held a licence to operate a class SSGN nuclear submarine, a credential that rounds to 0% of the population. Among opt-in respondents under 30, **12% said yes** ([Pew Research Center, 2024](https://www.pewresearch.org/short-reads/2024/03/05/online-opt-in-polls-can-produce-misleading-results-especially-for-young-people-and-hispanic-adults/)). Those people will also give you confident, short, useless answers about your onboarding flow.\n\n**What to change:**\n\n- Recruit from where your real users are (your customer list, your community, people who did the thing recently) before you buy sample.\n- Screen on behaviour, not identity: \"Have you cancelled a software subscription in the last 60 days?\" beats \"Are you a SaaS buyer?\". See [Research Screener Questions](/docs/research-screener-questions).\n- Treat a screener as a filter, not a lie detector. It narrows the pool. It does not verify anyone.\n\n### What about bots and AI-written answers?\n\nThey are real, but probably not your main problem yet. Sean Westwood at Dartmouth built an AI agent that passed **99.8% of attention checks** across 6,000 trials ([Westwood, PNAS, 2025](https://www.eurekalert.org/news-releases/1106172)). \"We can no longer trust that survey responses are coming from real people,\" he said when the study came out.\n\n![An AI respondent passed 99.8% of attention checks](/visuals/ai-respondent-passes-99-8-percent-attention-checks/ai-respondent-passes-99-8-percent-attention-checks.png \"Attention checks alone no longer prove a human answered. · Source: Sean Westwood, PNAS, 2025\")\n\nAt the same time, a Prolific-funded study of around 4,800 responses collected by 12 different companies found **fewer than 1%** contained text that was likely not written by a human ([Retraction Watch, April 2026](https://retractionwatch.com/2026/04/30/are-ai-chatbots-infiltrating-online-survey-data-not-yet-says-new-study/)). The two findings fit together: AI can pass your checks, but most thin answers today still come from real people who weren't the right people, weren't motivated, or weren't asked well. More on this in [AI-Generated Survey Responses](/blog/ai-generated-survey-responses-2026).\n\n## Cause 2: The incentive bought the wrong kind of effort\n\nIncentives change who shows up and how hard they try. Too little, and busy people rush. Too much, or paid with no link to effort, and you attract people who take surveys for the money and answer as fast as the form allows.\n\nJon Krosnick's theory of satisficing explains why. When the effort a question needs is higher than a person's motivation, they settle for a \"good enough\" answer. His 1991 paper lists the shortcuts: \"choosing the first response alternative that seems to constitute a reasonable answer, agreeing with an assertion made by a question, endorsing the status quo\" ([Krosnick, Applied Cognitive Psychology, 1991](https://doi.org/10.1002/acp.2350050305)). How often people do this depends on three things: how hard the task is, how able the respondent is, and how motivated they are. See [Satisficing](/docs/satisficing-survey-responses) for the full model.\n\n**What to change:**\n\n- Pay for time, at a rate that respects the person's job. See [Research Participant Incentives](/docs/research-participant-incentives) for typical rates.\n- Review responses before you pay, and say up front that you will.\n- For your own customers, a reason to care (\"we're deciding what to fix next and you'll see the result\") often beats a bigger gift card.\n\n## Cause 3: The study is too long\n\nIf answers get shorter the further people go, length is the cause. This is the most predictable pattern in survey research.\n\nMirta Galesic and Michael Bosnjak varied the stated length of a web survey between 10, 20 and 30 minutes and randomised where question blocks appeared. Fewer people started the longer versions: **75% started the 10-minute survey against 65% for the 20-minute one**. And for the people who did take part, answers to questions placed later were faster, shorter and more uniform than the same questions placed near the start ([Galesic and Bosnjak, Public Opinion Quarterly, 2009](https://doi.org/10.1093/poq/nfp031)).\n\nThat second finding matters more than the first. It means the same question gets a worse answer simply by being question 12 instead of question 2.\n\n**What to change:**\n\n- Put your most important open question in the first third.\n- Cut every question that doesn't map to a decision. See [How Long Should a Survey Be?](/docs/ideal-survey-length-guide).\n- If you need depth on many topics, run two short studies instead of one long one.\n\n## Cause 4: The questions are vague, leading or two-in-one\n\nIf most people go thin at the _same_ questions, the questions are the problem. Three wording faults produce short answers most reliably:\n\n- **Vague questions** (\"Any feedback?\") give people no starting point. Ask about one recent event: \"Tell me about the last time you exported a report. What happened?\"\n- **Leading questions** (\"How much did you enjoy the new dashboard?\") signal the answer you want. People agree briefly and move on. See [Avoiding Leading Questions](/docs/avoiding-leading-questions).\n- **Double-barrelled questions** (\"How do you feel about pricing and support?\") make people pick one half. See [Double-Barreled Questions](/docs/double-barreled-questions).\n\nA useful test: read the question aloud to a colleague. If they ask \"what do you mean?\", your participants are thinking it too, and they won't ask.\n\n## Cause 5: The format can't draw out detail\n\nSometimes the questions are fine and the box is the problem. In web surveys the size of the answer box itself shapes the answer. Jolene Smyth, Don Dillman and colleagues found that a larger answer box drew more words than a small one, especially from people who answered later ([Smyth et al., Public Opinion Quarterly, 2009](https://digitalcommons.unl.edu/sociologyfacpub/668)). On a phone, any text box is small.\n\nA form also asks once. When someone types \"it was confusing\", a form records that and moves on. An interviewer would ask \"confusing how? what were you trying to do?\". That second question is where most of the useful detail comes from.\n\n**What to change:**\n\n- For anything that needs a story, use an interview, or a survey that can ask a follow-up.\n- Let people answer by voice when they're on a phone. Speaking is faster than typing for most people.\n- Keep closed questions (scales, choices) for things that really are a choice.\n\n## Cause 6: Short answers that are actually fine\n\nNot every short answer is a bad one. \"Switched to a spreadsheet\" is four words and tells you who your competitor is. Before you throw anything out, read it.\n\nDiscarding short answers by length alone is risky. You end up keeping the people who write a lot, who aren't a fair sample of your customers. A better test is relevance: did the person answer the question that was asked? Short and on topic stays. Long and off topic goes.\n\n![Table of six causes of low-quality survey responses, what each looks like in your data and the first fix for each.](/visuals/low-quality-survey-responses-causes-fixes-table/low-quality-survey-responses-causes-fixes-table.png \"Match the pattern you see to a cause, then change that one thing first.\")\n\n## How to run the diagnosis on your next study\n\n1. **Read the thin answers side by side.** Export, sort by length, and read the bottom 10 to 20% together. Patterns you'd miss one at a time become obvious in a list.\n2. **Tag each one** with the respondent, the question number and whether it's on topic.\n3. **Count by shape.** Mostly a few people? Mostly a few questions? Mostly late? Mostly everywhere?\n4. **Change one thing**, the one the shape points to, and run a small pilot (five to ten people) before the full launch.\n5. **Write down your exclusion rule** before you look at results, and report how many responses you excluded and why. Stakeholders trust findings more when they can see what was left out.\n\n![A small team reading through printed survey answers together at a table to spot where the thin responses come from](https://images.unsplash.com/photo-1550623627-5494e1e93ed6?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3w5NDUzMDN8MHwxfHNlYXJjaHw0fHx0ZWFtJTIwcmV2aWV3aW5nJTIwbm90ZXMlMjBsYXB0b3AlMjB0YWJsZXxlbnwxfDB8fHwxNzk0NDY2MjEwfDA&ixlib=rb-4.1.0&q=80&w=1080)\n\n_Photo by [Mad Fish Digital](https://unsplash.com/@madfishdigital?utm_source=koji&utm_medium=referral) on [Unsplash](https://unsplash.com/?utm_source=koji&utm_medium=referral)_\n\n## How Koji helps\n\nKoji runs AI-moderated interviews in text or voice, and it is built so that several of these causes are handled while the interview is running.\n\n- **It asks the follow-up.** When someone gives a thin answer to an open question, Koji's interviewer asks what they meant or what happened next, the way a human interviewer would. That covers cause 5 directly. See [How Koji's AI Probing Works](/docs/ai-probing-guide).\n- **Every interview gets a quality score from 1 to 5**, shown as Shallow (1 to 2), Fair (3), Good (4) or Excellent (5). Thin or filler answers to open questions and follow-ups bring the score down, so clicking through the choice questions doesn't earn a high score. Short, on-topic answers still count. See [Understanding Quality Scores](/docs/understanding-quality-scores).\n- **Shallow interviews stay out of the report and cost nothing.** Interviews scoring 3 or more go into the report and use a credit. Ones below 3 are left out and not charged, so you don't pay for the thin answers this guide is about.\n- **You can read the thin ones side by side.** In your study's responses, filter the Evaluation column to \"Shallow only\" to see every low-scoring interview together, which is step 1 of the diagnosis above. The CSV export includes each interview's score for your own analysis.\n- **Screening Questions** turn away people who don't fit before the interview starts, which is the fix for cause 1. See [Adding Screening Questions to Your Study](/docs/adding-screening-questions-to-your-study).\n- **Voice interviews** let people talk instead of type, which helps on phones.\n\n![Koji quality rating card scoring an interview Excellent, with relevance, depth, coverage and completion scores.](/email/shots/shot-rating-details.jpg \"Each interview's quality rating, with what it was judged on.\")\n\nKoji doesn't replace good recruiting. A quality score catches low-effort and off-topic interviews. It isn't identity verification, so pair it with a real source and a screener. And if your questions are leading, an AI interviewer will still ask leading questions: fix the wording first.\n\nCompared with cleaning a survey by hand (sorting by time, flagging straight-liners, reading every open answer), the scoring happens per interview as responses arrive, and the thin ones are already separated when you open the results. Compared with most survey tools, the difference is the follow-up: a form records \"it was confusing\", while an interview asks why.\n\n## Frequently asked questions\n\n### Why are my survey responses low quality?\n\nUsually one of six reasons: the wrong people, an incentive that rewards speed, a survey that's too long, vague or leading questions, a format that can't ask follow-ups, or answers that are short but actually fine. Look at where the thin answers sit (in a few people, at a few questions, late in the survey, or everywhere) to tell which one.\n\n### Should I delete short survey answers?\n\nNot by length alone. Short answers that address the question are often useful, and deleting them biases your sample toward people who write a lot. Delete answers that are off topic, nonsense or from people outside your audience, and write your exclusion rule down before you look at results.\n\n### How do I know if low-quality responses are from bots?\n\nAttention checks are no longer enough: an AI agent built by Sean Westwood passed 99.8% of them in his 2025 PNAS study. But a Prolific-funded study of about 4,800 responses found fewer than 1% contained text likely not written by a human, so most thin answers still come from real people. Check your source and screener first.\n\n### Does a longer survey give me more data?\n\nOften less. Galesic and Bosnjak found fewer people start longer surveys, and that answers to questions placed later are faster, shorter and more uniform than the same questions placed early. Put your most important question first and cut anything that doesn't feed a decision.\n\n### How does Koji handle low-quality interviews?\n\nEach interview gets a 1 to 5 quality score. Interviews scoring 3 or more go into the report and use a credit; Shallow ones (1 to 2) are left out and not charged. You can filter to \"Shallow only\" to read them together, and Koji's interviewer asks follow-up questions when an answer is thin.\n\n## Related Resources\n\n- [Structured Questions Guide](/docs/structured-questions-guide): every question type and when to use each\n- [Survey Data Quality](/docs/survey-data-quality-guide): detecting and preventing bad responses\n- [Satisficing](/docs/satisficing-survey-responses): why people give good-enough answers\n- [Survey Fraud and Respondent Quality](/docs/survey-fraud-respondent-quality): fake and low-effort respondents\n- [Understanding Quality Scores](/docs/understanding-quality-scores): how Koji rates each interview\n- [Attention Check Questions](/docs/attention-check-questions): what they catch and what they miss\n","category":"Research Methods","lastModified":"2026-11-12T06:50:10.714395+00:00","metaTitle":"Why Are My Survey Responses Low Quality? 6 Causes and Fixes","metaDescription":"Low-quality survey responses usually come from the wrong people, the wrong incentive, length, wording or format. Find which one with a 15-minute diagnostic.","keywords":["why are my survey responses low quality","low quality survey responses","thin survey answers","survey response quality","short open-ended answers","bad survey data","satisficing","survey design mistakes"],"aiSummary":"Low-quality survey and interview responses usually come from one of six causes: the wrong people, an incentive that rewards speed, a study that is too long, vague or leading questions, a format that cannot ask follow-ups, or short answers that are actually fine. Diagnose by where the thin answers sit: in a few people (recruiting), at the same questions (wording), later in the study (length, per Galesic and Bosnjak 2009), or everywhere (format). Pew found 12% of opt-in respondents under 30 claimed a nuclear submarine licence, and an AI agent passed 99.8% of attention checks (Westwood 2025), though a Prolific-funded study found under 1% of about 4,800 responses likely AI-written. Koji scores each interview 1 to 5, leaves Shallow ones out of the report and uncharged, and asks follow-up questions on thin answers.","aiPrerequisites":["A survey or interview study with responses to review"],"aiLearningOutcomes":["Diagnose the cause of thin answers from where they appear","Tell recruiting, incentive, length, wording and format problems apart","Decide which short answers to keep and which to exclude","Write an exclusion rule before reading results"],"aiDifficulty":"beginner","aiEstimatedTime":"12 minutes"}],"pagination":{"total":1,"returned":1,"offset":0}}