{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-09-28T17:50:18.235Z"},"content":[{"type":"documentation","id":"157eb7c4-870e-44b5-b663-9d4d33584cd5","slug":"solicited-vs-spontaneous-feedback","title":"Solicited vs Spontaneous Feedback: Why Pooling Them Corrupts Both","url":"https://www.koji.so/docs/solicited-vs-spontaneous-feedback","summary":"Feedback that arrives unasked and feedback you solicited are two data-generating processes with different populations and different denominators, and merging them into one ranked list is an arithmetic error. Spontaneous reports are a numerator with no recoverable denominator but are enriched for severe issues, because reporting costs effort. Solicited responses carry a denominator you chose but are enriched for mild issues, because answering is free. Pooling compounds the two biases in opposite directions and produces something close to an inverted severity ranking. The ICH E2D guideline makes this a regulatory rule: reports from organised data collection systems are explicitly not counted as spontaneous. A worked example shows a cosmetic issue beating data loss 180 to 60 in a pooled list, and the distortion widening from 3x to 6x when the survey response rate improves - a metric that gets worse as your research gets better. The correct combination is sequential: the inbox generates hypotheses, a Koji study estimates prevalence.","content":"## The short answer\n\nFeedback arrives two ways. Someone writes in without being asked, or you ask and they answer. These are not two sources of the same thing. They are two different data-generating processes with different populations, different selection mechanisms, and - critically - different denominators. One has a denominator you chose. The other has none at all.\n\nMerging them into a single ranked list of \"top customer issues\" is the most common analytical error in product feedback work. It is not a rough approximation that loses a little precision. It systematically promotes mild, widely-sampled issues over severe, rarely-reported ones, and it gets worse the better your survey program performs. Keeping the two apart costs one column in a spreadsheet, and a platform like Koji makes the solicited half cheap enough that the separation is practical rather than aspirational.\n\n## Two processes, two denominators\n\n### Spontaneous: a numerator with no denominator\n\nAn unprompted complaint exists because someone hit a problem, cared enough to act, found your channel, and wrote something legible. You see the people for whom all of that was true. You cannot see or count the people for whom any link in that chain failed, which means you have a numerator and no way to compute a rate from it. That limitation is severe enough to deserve its own treatment, and it has one.\n\nWhat spontaneous feedback carries instead is a signal about intensity. Nobody files a ticket about a minor annoyance they can route around. The act of reporting is itself weak evidence that the problem crossed a threshold worth someone's time.\n\n### Solicited: a denominator you chose\n\nWhen you invite 2,000 customers and 500 answer, you know both numbers. You can compute a real rate, state its base, and defend it. You also inherit a different problem: you decided what to ask about, so the results are bounded by your imagination. A solicited channel cannot surface a problem you did not think to ask about, and it will happily return a distribution of opinions on an issue nobody actually cares about, because you asked and people are obliging.\n\nThere is a second inherited problem. The 1,500 who did not answer are not a random subset of the 2,000, so even your clean rate carries nonresponse bias. That is a smaller and better-understood problem than having no denominator, but it does not vanish.\n\n## How safety regulation draws the line\n\nPharmacovigilance treats this distinction as a matter of regulation rather than preference. Under the ICH E2D guideline on post-approval safety data management, a spontaneous report is an unsolicited communication describing a suspected adverse reaction, while a solicited report is one derived from an organised data collection system - a clinical trial, a registry, a patient support or disease management programme, or a survey of patients or providers.\n\nThe rule that follows is the one product teams should borrow: reports obtained from those organised systems are explicitly not to be counted as spontaneous. They are classified as study reports and carry a causality assessment made by a qualified person.\n\n### Why the rule exists\n\nTwo reasons, and both apply directly to product work.\n\nThe first is that pooling destroys the interpretability of the spontaneous stream. Disproportionality methods work by comparing one slice of a report database against the rest of that database. Inject a few thousand solicited responses about a topic you chose and you have changed the baseline that every comparison is measured against. Signals in unrelated areas will appear to weaken, because the denominator of the comparison group grew.\n\nThe second is that solicited data comes with a causal judgement attached and spontaneous data does not. Someone answering \"yes\" to \"did export drop rows for you?\" is responding to your framing. Someone who writes in unprompted supplied both the symptom and their own attribution. Pooling them merges answers to a leading question with independent observations.\n\n## What goes wrong when you pool\n\n### The dilution error\n\nSolicited campaigns produce responses in bulk on whatever you asked about. Spontaneous reports trickle in across every topic. Pool them and any surveyed topic outranks any unsurveyed one almost automatically, because you manufactured volume for the first and not the second. The ranking measures what you asked about recently.\n\n### The severity inversion\n\nThis is the damaging one. Because spontaneous reporting requires the problem to clear a motivation threshold, the spontaneous stream is enriched for severe issues. Because solicited responses cost the respondent nothing but a click, the solicited stream is enriched for mild ones.\n\nPool them and the two biases compound in opposite directions: mild issues gain volume they could never have earned unprompted, and severe issues keep their small unprompted counts. The pooled list is close to an inverted severity ranking. A team following it in good faith will work on cosmetic problems while data loss sits below the fold.\n\n### The double count\n\nThe same person can appear in both streams - they filed a ticket in March and answered your survey in April about the same frustration. There is no key that reliably joins them, and the overlap is invisible. Even regulators with mandatory identifiers acknowledge this class of problem: FDA notes of its own database that \"There are also duplicate reports where the same report was submitted by a consumer and by the sponsor.\"\n\n### A worked example of the pooling error\n\nTake a quarter with both channels running.\n\n- **Spontaneous:** 4,000 inbound reports. 60 describe rows going missing during CSV export. Severity: data loss.\n- **Solicited:** 2,000 customers invited, 500 responded. 180 of them agreed that export column ordering is annoying. Severity: cosmetic.\n\n| | Count | Denominator | Rate | Severity |\n| --- | --- | --- | --- | --- |\n| Export data loss | 60 | unknown | not computable | Critical |\n| Export column order | 180 | 500 respondents | 36% | Cosmetic |\n\nPooled into one list sorted by count, column ordering beats data loss 180 to 60 and wins the roadmap slot. Unpooled, the two are not even comparable: one is a rate on a known base, the other a count from an unbounded population. The number 180 is larger than 60 for reasons that have nothing to do with importance.\n\nNow the part that makes this worse rather than better over time. Suppose you improve your survey program and lift the response rate from 25% to 50%. The same 36% now yields 360 responses instead of 180. The pooled gap widens from 3x to 6x. **Improving the quality of your solicited research makes a pooled ranking more wrong, not less.** Any metric with that property is not a metric.\n\n## Keep them separate, then use each for what it can do\n\n### What only spontaneous feedback can tell you\n\n- That a problem exists which nobody on your team had considered\n- That something crossed a threshold worth a customer's unpaid effort\n- The customer's own words and their own attribution, unprompted by your framing\n- Which issues are severe enough to generate action rather than opinion\n\n### What only solicited feedback can tell you\n\n- How common something is, with a stated denominator\n- How a defined population divides on a question, including the people who would never write in\n- A comparable number across two time periods, because the instrument was held fixed\n- The view of the silent majority, who are the bulk of your users and are absent from the inbox\n\n### The one legitimate way to combine them\n\nSequentially, never pooled. The inbox is a hypothesis generator; a study is the estimator.\n\n1. A cluster of unprompted reports flags a candidate issue.\n2. Disproportionality analysis on the spontaneous stream confirms it is over-represented rather than merely present.\n3. You grade one or two cases properly to check the feature is actually implicated.\n4. A solicited study, such as a Koji study sent to a defined slice of the affected population, measures prevalence and severity against a known base.\n5. The roadmap argument cites step 5 for the size and step 1 for the discovery.\n\nEach channel does the job it can do. Nothing is added to anything. This is the entire discipline, and it is a reporting convention rather than a technique - which is why it is so easy to skip and so cheap to adopt.\n\n## How Koji handles this\n\nKoji is a solicited channel by design, and the value of saying so plainly is that you know exactly what its output means.\n\n- **A denominator by construction.** Every Koji study has an invited count and a completed count, so every finding is a rate with a stated base rather than a count of unknown completeness.\n- **Six structured question types.** open_ended, scale, single_choice, multiple_choice, ranking and yes_no. A yes_no gives you prevalence, a scale gives you severity, and a ranking forces the tradeoff your pooled list was pretending to make. These are countable without a tagging pass.\n- **AI follow-ups recover what surveys normally lose.** The usual objection to solicited data is that it only returns what you asked. Koji's AI interviewer probes each answer with unscripted follow-ups, so an open_ended question can surface the problem you did not anticipate - closing much of the gap that makes teams reach for the inbox.\n- **The silent majority answers.** Invitation rather than motivation decides who responds, which is precisely the population the spontaneous stream cannot reach.\n- **Voice or text, no moderator.** A confirmatory study is an afternoon, so step 4 above stops being the step everyone skips.\n- **Reports build live**, with distributions for the structured questions and themes plus quotes for the open-ended ones, already separated from your ticket data.\n\nKeep the two in separate tables, label every chart with which process produced it, and never sort them together. If your tickets live in Zendesk, Koji can trigger a study straight off a ticket - which is the sequential pattern above, wired up: the spontaneous report stays a signal, and the study produces the number.\n\n## Frequently asked questions\n\n### Is solicited feedback lower quality than unprompted complaints?\n\nNeither is lower quality; they answer different questions. Unprompted complaints are better evidence that something matters enough to act on, because reporting costs effort. Solicited responses are better evidence of how common something is, because they come with a denominator. The error is not preferring one - it is adding them together and reading the total as though it measured a single thing.\n\n### Can I just weight the two streams instead of separating them?\n\nTo weight them you would need the spontaneous reporting rate, which is the one quantity a spontaneous channel cannot give you. Any weight you pick is a guess, and it silently determines the ranking your roadmap follows. Separate reporting is honest and free; weighting buys a single tidy list at the cost of burying an unfalsifiable assumption inside it.\n\n### Does this mean our top issues list is wrong?\n\nIf it merges ticket counts with survey counts and sorts by total, then yes, and probably in a specific direction: mild surveyed issues sitting above severe unsurveyed ones. The fix takes an hour. Split the list in two, label each with its process and denominator, and sort each on its own terms. Most teams find the top of the two lists barely overlap, which is the finding.\n\n### Where do NPS verbatims and in-app prompts belong?\n\nBoth are solicited, because you chose the moment and the prompt. An in-app \"how was this?\" widget feels passive but is an organised collection system, and it carries a denominator you can recover from impression counts. Only feedback that arrives with no prompt from you - a ticket, an unsolicited email, a public post - counts as spontaneous.\n\n### How large does the solicited sample need to be?\n\nLarge enough for the precision the decision needs, which is usually far smaller than teams expect - a few dozen responses will separate a 10% problem from a 50% one. The sample size question is answerable precisely because you have a denominator, and it is a strong argument for running the study rather than accumulating tickets: with a study the uncertainty is a number, and with an inbox it is unknown.\n\n### What is the single change with the biggest payoff?\n\nAdd one column to your feedback tracker recording whether each item was solicited or spontaneous, then stop sorting across it. That one field prevents the severity inversion, makes your rates interpretable, and costs nothing. Everything else in this article is refinement on top of it.\n\n## Related Resources\n\n- [Structured Questions in AI Interviews](/docs/structured-questions-guide) - the six question types that give solicited feedback a countable shape\n- [Why Complaint Counts Cannot Become Rates](/docs/complaint-counts-cannot-be-rates) - the missing-denominator problem in the spontaneous stream\n- [Did the Feature Cause the Complaint?](/docs/complaint-causality-dechallenge-rechallenge) - step 3 of the sequential pattern above\n- [Nonresponse Bias](/docs/nonresponse-bias) - the bias that solicited channels inherit in exchange for a denominator\n- [Sampling Bias](/docs/sampling-bias-research) - how each channel selects who you hear from\n- [Support Ticket Analysis](/docs/support-ticket-research-analysis) - getting the most from the spontaneous stream on its own terms","category":"Research Methods","lastModified":"2026-09-28T03:43:27.780753+00:00","metaTitle":"Solicited vs Spontaneous Feedback: Never Pool Them","metaDescription":"Asked-for and unasked-for feedback are different processes with different denominators. Pooling them inverts your severity ranking.","keywords":["solicited vs spontaneous feedback","prompted vs unprompted feedback","pooling feedback sources","feedback denominator","top issues list error","survey vs support tickets"],"aiSummary":"Feedback that arrives unasked and feedback you solicited are two data-generating processes with different populations and different denominators, and merging them into one ranked list is an arithmetic error. Spontaneous reports are a numerator with no recoverable denominator but are enriched for severe issues, because reporting costs effort. Solicited responses carry a denominator you chose but are enriched for mild issues, because answering is free. Pooling compounds the two biases in opposite directions and produces something close to an inverted severity ranking. The ICH E2D guideline makes this a regulatory rule: reports from organised data collection systems are explicitly not counted as spontaneous. A worked example shows a cosmetic issue beating data loss 180 to 60 in a pooled list, and the distortion widening from 3x to 6x when the survey response rate improves - a metric that gets worse as your research gets better. The correct combination is sequential: the inbox generates hypotheses, a Koji study estimates prevalence.","aiDifficulty":"intermediate","aiEstimatedTime":"11 min"}],"pagination":{"total":1,"returned":1,"offset":0}}