Thumbs Up, Thumbs Down: What In-Product AI Feedback Signals Actually Tell You (2026)

Your AI feature has a thumbs-up and thumbs-down button. The dashboard says 87% positive. That number is close to meaningless, and this guide explains precisely why, what it is actually measuring, and what to put in its place.

The short version: a voluntary feedback widget does not sample opinion. It samples the intersection of three much narrower things - outputs that were bad enough to notice, users engaged enough to react, and moments where clicking felt worth the effort. Decades of research on exactly this problem exist, mostly in the recommender-systems and online-review literature, and almost none of it has made it into how product teams read their AI feedback dashboards.

The number you are looking at is a product of three probabilities

Here is the decomposition that should be printed above every thumbs dashboard. Your observed thumbs-down rate is not the rate at which your AI is wrong. It is:

P(the output was bad) x P(the user noticed it was bad) x P(the user bothered to click)

Every one of those three terms moves independently, and none of them is visible in the number you get. Which produces the trap that catches most teams:

If your thumbs-down rate drops 30% after a release, you cannot tell whether the model improved, whether users got worse at spotting errors, or whether someone moved the button. All three produce an identical, celebratory chart. Teams have shipped "quality improvements" that were pure measurement artefacts, and teams have missed real regressions because a UI change quietly suppressed the second and third terms.

The second term deserves particular attention because it moves in the wrong direction over time. As users grow more reliant on an AI feature, they check its output less carefully, so P(noticed) falls. A maturing feature can therefore show falling complaint rates while its real error rate is flat - which is the same dynamic covered in AI over-reliance and automation bias research.

Who actually clicks: participation inequality

Jakob Nielsen documented the base rate two decades ago and it has not moved. His 90-9-1 rule holds that "90% of users are lurkers (i.e., read or observe, but do not contribute). 9% of users contribute from time to time, but other priorities dominate their time. 1% of users participate a lot and account for most contributions."

The skew is worse in some contexts than others. Nielsen notes that "with blogs, the rule is more like 95-5-0.1," that Wikipedia follows "a 99.8-0.2-0.003 rule," and that fewer than 1% of Amazon book customers write reviews.

His conclusion is the sentence every product team should read before their next feedback review:

"If your company looks to Web postings for customer feedback on its products and services, you are getting an unrepresentative sample."

And on why: "On any given user-participation site, you almost always hear from the same 1% of users, who almost certainly differ from the 90% you never hear from."

This is not a reason to remove the buttons. It is a reason to stop treating their output as a measurement of your user base.

The hard evidence: the same product, rated two ways

The most convincing demonstration of what voluntary feedback does to your data comes from a controlled experiment by Nan Hu, Paul Pavlou and Jennifer Zhang, whose work on self-selection in online reviews appeared in MIS Quarterly (41(2), 449-475).

Their setup is the one every product team wishes they could run. They took a single product - a music CD - and collected ratings two ways: the voluntary ratings Amazon had accumulated, and ratings from an experiment where every respondent had to report a score whether they felt strongly or not.

The two distributions were not variations on a theme. They were different shapes.

Two numbers there are worth committing to memory.

Voluntary feedback overstated the mean by 0.42 points on a five-point scale - roughly 13% - for the identical product and the identical question. That is your satisfaction inflation, measured.

Voluntary feedback made the population look more than six times as polarised as it was. Over 60% extreme versus under 10% extreme. The "our users either love it or hate it" conclusion that teams draw from feedback widgets is, in this experiment, almost entirely an artefact of who chose to speak.

The authors attribute this to two distinct self-selection biases that stack:

• Acquisition (purchasing) bias - the people who chose the product were positively predisposed to it before they used it. In AI product terms: the people using your feature are the ones who expected it to help.
• Under-reporting bias - people with polarised opinions are far more likely to report them, to "brag-or-moan," than people with moderate views.

Their bottom line generalises cleanly to any voluntary channel: when the distribution is J-shaped, the mean is a biased estimator. Averaging your thumbs data does not recover the truth, because the problem is not noise. It is the sampling frame.

Explicit signals versus implicit signals

Explicit feedback - thumbs, stars, report buttons - is scarce and biased but unambiguous in intent. Implicit feedback - what users actually did - is abundant and unbiased in the self-selection sense, but ambiguous in meaning. You need both, and you need to know what each one confounds.

Edit distance is the most underrated signal on that list. If your product lets users modify AI output before using it, the amount they change is a continuous, unsolicited, low-bias measure of how close the model got. It costs the user nothing to produce and it is generated by everyone, not just the vocal 1%. Most teams instrument the thumbs and ignore the edits.

Making the explicit channel less bad

You cannot fix self-selection with a bigger sample - a bigger biased sample is just a more confidently wrong one. But four changes materially improve what the channel gives you.

1. Log the denominator, always. A thumbs-down count with no exposure count is uninterpretable. Report the rate per eligible response and the participation rate itself, so you can see when the third probability moved.
2. Never change the widget and the model in the same release. If you must, hold one segment on the old widget so you have a control. This single discipline prevents most false quality wins.
3. Ask for a reason, and make it structured. An unlabelled thumbs-down tells you almost nothing. A thumbs-down with "wrong facts / did not follow instructions / too generic / unsafe / too slow" attached turns an unusable signal into a triage queue.
4. Sample the silent majority on purpose. Periodically prompt a random slice of users for a rating they did not volunteer. That involuntary sample is the only thing that lets you estimate how far your voluntary channel is off - which is exactly what the Hu, Pavlou and Zhang experiment did.

Point four is the one that changes the epistemics. Everything else improves a biased instrument. That one gives you a yardstick to calibrate it against.

The step the dashboard cannot do for you

Even a well-instrumented feedback widget answers "how many" and "what category." It never answers "why," and "why" is the only thing that tells you what to build.

A structured thumbs-down reason of "wrong facts" covers at least four different product problems: the retrieval missed, the model hallucinated, the source data was stale, or the user asked about something outside the scope you built for. These need completely different fixes, and no dropdown will ever distinguish them. You have to ask a person.

This is where most teams stall, because the traditional path from a feedback signal to a conversation is slow and expensive. You export the thumbs-down cohort, hand it to a researcher, schedule sessions, run them, transcribe them, code them. By the time you know why, you have shipped two more releases.

Closing the loop with Koji

Koji is designed to make the signal-to-conversation step nearly instant. Rather than exporting a cohort and booking calls, you trigger an AI-moderated interview from the feedback event itself - a PostHog cohort of users who left a thumbs-down, a Segment audience, or a Zendesk ticket. The interview runs at whatever hour the user is free, in text or voice, and follows up on every answer rather than accepting the first sentence.

The difference in what you get back is not subtle. A traditional survey tool collects "the answer was wrong." Koji AI interviewer asks which part was wrong, what the user expected, what they did next, and whether they went back to the old way of doing the task - the reasoning chain that a dropdown structurally cannot capture.

A study design that pairs cleanly with feedback telemetry, using Koji six structured question types:

Because the numbers live in structured fields rather than in free-form speech, they survive voice transcription intact and aggregate automatically - and the open-ended answers carry the reasoning. Thematic analysis happens as responses arrive, so a study that would have meant a week of manual coding becomes a live report.

The wider point is that Koji lets you interview the silent 90% as easily as the vocal 1%. AI moderation removes the scheduling cost that made "go and ask a random sample" impractical, which is precisely the constraint that forced teams to over-rely on voluntary feedback in the first place. If your instrument only hears from people motivated enough to raise their hand, the fix is not a better instrument - it is a cheaper way to ask everyone else.

The summary you can act on

• Your thumbs rate is P(bad) x P(noticed) x P(bothered). Never attribute a change to the model alone.
• Voluntary feedback overstated a mean by 0.42 points and inflated apparent polarisation more than sixfold in a controlled test on the same product.
• Around 1% of users generate nearly all explicit feedback, and they differ systematically from the rest.
• Implicit signals - especially edit distance - are more representative and badly under-instrumented.
• Periodically collect involuntary ratings so you can calibrate the voluntary ones.
• Categories tell you what; only conversations tell you why.

---

Try it yourself. Koji gives you 10 free credits at signup - enough to interview a real thumbs-down cohort and find out what your dashboard has been hiding. No seat licences, no sales call, no PhD in research methods required.

Frequently asked questions

Why is my thumbs-up percentage misleading?

Because it is not a measurement of your users. An observed rate is the product of three independent probabilities: that the output was genuinely bad, that the user noticed, and that the user bothered to click. All three move on their own. If the rate drops 30% after a release you cannot tell whether the model improved, users became less attentive, or someone repositioned the button - each produces an identical chart. Always report the participation rate and the denominator alongside the ratio.

What proportion of users actually leave feedback?

Roughly 1%, and they are not a random 1%. Jakob Nielsen 90-9-1 rule holds that 90% of users are lurkers, 9% contribute occasionally, and 1% account for most contributions; blogs skew to about 95-5-0.1 and Wikipedia to 99.8-0.2-0.003, and fewer than 1% of Amazon book customers write reviews. Nielsen conclusion applies directly to product feedback: you almost always hear from the same 1% of users, who almost certainly differ from the 90% you never hear from.

How far off is voluntary feedback, in real numbers?

In a controlled experiment reported by Hu, Pavlou and Zhang in MIS Quarterly (41(2), 449-475), the same music CD was rated voluntarily on Amazon and involuntarily by experiment respondents. The voluntary distribution was bimodal and J-shaped with over 60% of ratings at 1 or 5 stars and a mean of 3.54; the involuntary distribution was roughly normal with over 90% of ratings moderate and a mean of 3.12. Voluntary feedback therefore overstated the mean by 0.42 points on a five-point scale and inflated apparent polarisation more than sixfold.

What are acquisition bias and under-reporting bias?

They are the two self-selection effects that stack to produce the J-shaped distribution. Acquisition or purchasing bias means the people who chose to use something were positively predisposed to it beforehand - in product terms, your AI feature users expected it to help. Under-reporting bias means people with polarised opinions are far more likely to report them, to brag or moan, than people with moderate views. Because the result is J-shaped rather than merely noisy, averaging the data does not recover the truth: the mean is a biased estimator.

Which implicit signals are worth instrumenting?

Regeneration and retry, copy-to-clipboard, edit-before-use, mid-stream abandonment, immediate rephrasing, and escalation to a human. Edit distance is the most underrated: if users can modify AI output before using it, how much they change is a continuous, unsolicited, low-bias quality measure produced by everyone rather than by the vocal 1%. Each implicit signal has its own confound - abandonment cannot distinguish poor quality from slow response - so read them as a set rather than individually.

How do I collect feedback that is not self-selected?

Prompt a random slice of users for a rating they did not volunteer, on a regular cycle. That involuntary sample is the only way to estimate how far your voluntary channel is skewed, which is precisely the design that made the Hu, Pavlou and Zhang comparison possible. Everything else - better copy, better placement, reason codes - improves a biased instrument; only an involuntary sample gives you a yardstick to calibrate it against. Koji makes this practical because AI-moderated interviews remove the scheduling cost that made asking a random sample impractical.

Do reason codes on a thumbs-down solve the problem?

They help with triage but not with diagnosis. A structured reason such as "wrong facts" covers at least four distinct product problems - retrieval missed, model hallucinated, source data was stale, or the request was outside the scope you built for - and these need completely different fixes. No dropdown can separate them. In Koji you can pair a single_choice reason code with an open_ended question that probes automatically for what the user expected and what they did next, so the categories and the causes arrive in the same conversation.

Related Resources

• Structured Questions in AI Interviews - the six question types and when each one earns its place
• Human Evaluation of AI Outputs - building a rigorous evaluation programme
• AI Over-Reliance and Automation Bias - why complaint rates fall as trust rises
• Graceful Degradation for AI Features - the failure states your telemetry misses
• AI-to-Human Handoff and Escalation - what escalation signals really mean
• Trust and Safety Research - reporting flows and moderation
• How to Analyze Open-Ended Survey Responses with AI - turning free text into themes