Did Users Actually Notice? Sensitivity vs Criterion in Did-You-Notice Questions (2026)

The percentage of users who say they noticed your change is not a measurement of whether they noticed your change. It is the sum of two independent things: how well they can actually tell the new version from the old one, and how willing they are to answer yes. Signal detection theory separates those two, and the separation is not a refinement — it is the difference between a number you can act on and a number that reverses when you reword the question.

This matters most for teams who have already done the hard part. You sized the change against a perceptual threshold, shipped something genuinely bigger than the just-noticeable difference, and now you want to confirm that it landed. The obvious next step — ask people whether they noticed — is the step that quietly destroys the result.

The inversion, stated plainly

The threshold guide gives you a clean instruction: make the change large enough to perceive, then ask. This article is the correction to the second half of that sentence. Asking is itself an intervention. Every choice you make in how you ask — the wording, whether you named the feature, whether you told them something changed, whether there is a reward for engaging — moves the answer without moving perception at all.

Worse, it moves it in a direction you will like. Announce the improvement in a release note, then survey; announce it in the survey preamble; offer an incentive for detailed feedback. All three raise the yes rate. None of them raises anyone's ability to tell the versions apart. You will read the result as confirmation and ship the next change with an inflated prior.

The 2x2 that fixes it

Signal detection theory starts by refusing to look at one number. Any yes-or-no detection question has four possible outcomes, and you need all four:

The right-hand column is the one product teams never collect. You cannot fill it in from users who received the change, by definition — you need a matched group who received no change and were asked the identical question. That group gives you the false-alarm rate: the proportion of people who report noticing an improvement that does not exist.

As Stanislaw and Todorov put it in their standard reference on the calculations (Behavior Research Methods, Instruments, and Computers, 1999, volume 31, pages 137-149), "the major contribution of SDT to psychology is the separation of response bias and sensitivity." They are blunt about what happens without it: "Sensitivity and response bias are confounded by most performance measures, including the hit rate, the false-alarm rate, the hit rate 'corrected' by subtracting the false-alarm rate, and the proportion of correct responses in a yes/no task." And therefore, "if (for example) the hit rate varies between two different conditions, it is not clear whether the conditions differ in sensitivity, response bias, or both."

That last sentence is your entire A/B perception dashboard, described as uninterpretable.

Two numbers instead of one

From the hit rate H and the false-alarm rate F, signal detection theory computes two separate quantities using the inverse normal distribution — z() below is the z-score corresponding to a proportion, available as NORM.S.INV in a spreadsheet and qnorm in R.

Sensitivity, d-prime: how far apart the two experiences actually are, in standard deviation units.

d' = z(H) - z(F)

Stanislaw and Todorov state it directly: "d' is found by subtracting the z score that corresponds to the false-alarm rate from the z score that corresponds to the hit rate." A value of 0 means no discrimination at all; larger values mean the change is easier to detect.

Criterion, c: how much evidence a person demands before saying yes.

c = -0.5 * [ z(H) + z(F) ]

Negative values mean a liberal criterion (says yes readily). Positive values mean a conservative one. Zero is neutral. The criterion is a decision policy, not a perceptual fact, and it is the thing your question wording controls.

A worked example where the two answers disagree

Two cohorts get the same redesign. Both are asked whether the app feels faster. Cohort A is asked after a preamble that mentions a performance release; cohort B is asked cold. Each is paired with a control group that received no change at all.

The headline numbers differ by 21 percentage points. The underlying ability to detect the change is identical to four decimal places. Every point of that 21-point gap is criterion — cohort A was told to expect an improvement and lowered the bar for saying yes. Their false-alarm rate rose in exact lockstep, which is the tell, and which is precisely what you never see if you only survey the group that got the change.

Now a second pair, to show that subtracting the false-alarm rate does not rescue you either:

On the difference score, D wins. On sensitivity, C is 45% better at telling the versions apart. The difference score ranks them backwards, because it treats a percentage point near the ceiling as equal to a percentage point in the middle, and it is not. This is the confound Stanislaw and Todorov name, demonstrated in the units a product team would actually report.

One practical correction

If a cell hits 0% or 100% — nobody in the control group reported a change, everybody in the test group did — the z-score is infinite and d-prime blows up. The standard remedy, described in the same paper and attributed to Hautus, is the log-linear correction: add 0.5 to the number of hits and 0.5 to the number of false alarms, and add 1 to each of the trial totals, before computing the rates. It "seems to work reasonably well", and the authors note that some researchers advocate using it regardless of whether extreme rates occurred. Applied to 20 hits out of 20 signal trials and 0 false alarms out of 20 noise trials, it gives rates of 0.9762 and 0.0238 and a finite d-prime of 3.96 instead of an error.

What moves the criterion in a real study

Everything cheap, and nothing expensive:

• Preamble. "We have been working on performance" raises yes rates without touching perception.
• Feature naming. Asking about "the new instant search" presupposes both newness and instantness.
• Incentives and completion pressure. A participant who wants to seem useful reports more.
• Question polarity. "Did you notice any improvement?" and "Did you notice any change in speed?" are not the same question; the first supplies the direction.
• Who is asking. A prompt inside the product from the team that built it invites agreement in a way an independent interviewer does not.
• Stakes. Stanislaw and Todorov cite jury research showing that instructions about reasonable doubt "affect response bias (the willingness to convict) rather than sensitivity (the ability to distinguish guilty from innocent defendants)", and quality-control research where inspectors detect fewer faults as a shift wears on — a declining hit rate that comes from changing bias, not failing eyes. Remedies aimed at the wrong one fail.

The design implication is uncomfortable but simple: you can never remove criterion effects, so you must measure them. A control group is not a nicety here. It is the only thing that converts your hit rate into a fact.

The rating-scale upgrade

A yes-or-no question gives you exactly one point on the curve relating hits to false alarms. A graded confidence rating gives you the whole curve. Stanislaw and Todorov describe how a rating task yields multiple criteria at once: each boundary between adjacent rating categories acts as its own criterion, so a single 1-to-6 confidence scale traces out a receiver operating characteristic across several points instead of one.

In practice this means: do not ask "did it feel faster, yes or no". Ask for a rating from "definitely slower" through "definitely faster", collect the same scale from the control group, and you get a far more robust sensitivity estimate from the same number of participants. It also degrades gracefully — if the control group turns out to be unusable, you still have an interpretable distribution.

How to run this in Koji

The design has three requirements that a conventional survey tool handles badly: a matched control arm asked an identical question, a confidence scale rather than a binary, and a probe into why the person answered as they did without leading them further.

Koji covers all three in one study, using its structured question types:

• yes_no for the raw detection question, in both the change arm and the control arm.
• scale for the graded confidence rating that produces the full ROC rather than a single point.
• single_choice for forced-choice comparisons where you show two variants directly.
• ranking when there are three or more variants and you want an ordering per participant.
• multiple_choice for what the participant attributed the change to, which surfaces false alarms that come with a confident but fabricated explanation.
• open_ended for the reasoning, where Koji's AI interviewer asks its own follow-ups.

That last capability is what makes the control arm worth running. A participant in the no-change group who says the app got faster will, under AI follow-up, often explain exactly why — they upgraded their phone, they used it at a quieter time of day, they read the changelog. Those transcripts are the most valuable output of the whole study, and a static survey form collects none of them because nobody types a paragraph into an optional box.

Because the interviewer is AI-moderated, the wording is identical in every session across both arms, which is exactly what criterion control requires. A human moderator running fifty sessions will drift; a survey tool cannot probe; Koji does both. Reports aggregate the two arms side by side, so the hit rate and the false-alarm rate arrive together rather than in two studies run a month apart.

Where this sits next to what you already have

This is not anchoring bias, which is about a number you supply contaminating a number they give back. It is not observer bias, which is about the researcher's expectations shaping what the researcher records. And it is not acquiescence bias, which is a general tendency to agree regardless of content — acquiescence is one contributor to criterion, but criterion also moves with stakes, framing and evidence in ways acquiescence does not describe.

Nor is it the slower problem of the reference point itself drifting between waves, which is covered in the zero point moving — criterion is a decision policy inside a single measurement, while a moving reference changes what the scale means between measurements.

Signal detection theory is the framework that makes those biases measurable rather than merely nameable. It says: the tendency to say yes is a parameter, it has a value, and you can compute it from a control group in one line of spreadsheet arithmetic.

Frequently asked questions

What is d-prime in signal detection theory?

d-prime is a measure of sensitivity: how far apart two experiences are in the mind of the observer, expressed in standard deviation units. It is computed as the z-score of the hit rate minus the z-score of the false-alarm rate. Zero means the person cannot tell the two conditions apart at all; larger values mean the difference is easier to detect. Its key property is that it does not change when the person becomes more or less willing to say yes.

Why do I need a control group that got no change?

Because without one you cannot separate detection from willingness to answer yes. A group that received a change gives you only a hit rate, and a hit rate is jointly determined by sensitivity and criterion. The control group supplies the false-alarm rate — the proportion of people who report noticing an improvement that never shipped — and only with both can you compute either quantity.

Is subtracting the false-alarm rate from the hit rate good enough?

No. It is one of the measures explicitly identified as confounding sensitivity and bias. In the worked example above, cohort C (95% hits, 70% false alarms) has a difference score of 0.25 and cohort D (65% hits, 35% false alarms) has 0.30, so the difference score ranks D higher — while d-prime shows C is substantially more sensitive, at 1.12 against 0.77. The correction ranks them backwards.

What if nobody in my control group reported noticing anything?

A false-alarm rate of exactly zero makes d-prime infinite, so apply the log-linear correction: add 0.5 to the hit count and 0.5 to the false-alarm count, and add 1 to each trial total, before computing rates. With 20 of 20 hits and 0 of 20 false alarms this yields rates of 0.9762 and 0.0238, giving a finite d-prime of about 3.96. Some researchers apply the correction routinely rather than only in extreme cases.

Does this apply to qualitative interviews or only to surveys?

It applies to any question with a yes-or-no structure, including one an interviewer asks aloud. The advantage of an AI-moderated interview is that the wording — and therefore the criterion pressure — is identical across every session and both arms, which is very hard to achieve with a human moderator running dozens of conversations. The follow-up probing then tells you what the participant based the judgement on, which is where false alarms reveal themselves.

How large do the two arms need to be?

Sensitivity estimates stabilise faster than most attitude measures because each participant can contribute several trials. As a working rule, 40 to 60 people per arm with several detection judgements each gives a usable d-prime, and the control arm should be the same size as the change arm rather than a small token group — the false-alarm rate carries as much of the estimate as the hit rate does.

Related Resources

• Structured Questions in AI Interviews — the six question types used to build detection, confidence and attribution questions.
• Just-Noticeable Difference: The Smallest Change Users Can Perceive — sizing the change before you ask about it.
• Threshold Testing: Finding the Level Where Users Start to Notice — converging on a threshold efficiently.
• How to Avoid Leading Questions — the wording choices that move the criterion.
• Acquiescence Bias — one specific contributor to a liberal criterion.
• Anchoring Bias in Research and Surveys — the related but distinct problem of a supplied number contaminating an answer.