{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-08-23T11:20:07.020Z"},"content":[{"type":"documentation","id":"7e7b959e-b51f-469f-bbd3-6c74b565a2bc","slug":"threshold-testing-adaptive-staircase-research","title":"Threshold Testing: How to Find the Level Where Users Start to Notice (2026)","url":"https://www.koji.so/docs/threshold-testing-adaptive-staircase-research","summary":"An adaptive procedure chooses each trial from the answers to previous trials, so nearly every trial lands near the threshold instead of at levels nobody was going to get wrong. The transformed up-down rule described by Levitt targets 70.7% correct with a two-down one-up staircase and 79.4% with three-down one-up. A worked 20-trial track starting at 400 ms with a step drop from 100 ms to 50 ms after two reversals converges to an estimate of 225 ms from the last six turnarounds. Discard early reversals, average an even number, interleave ascending and descending tracks, and insert catch trials. For product work, distribute the staircase across participants at eight to twelve trials each rather than running 100-plus trials with one person.","content":"**The efficient way to find the level at which users start to notice something is not to test every level — it is to let each answer choose the next question.** A fixed ladder of stimulus levels spends most of its participants on rungs that are obviously too easy or obviously too hard, and those trials tell you almost nothing about where the threshold sits. An adaptive procedure walks toward the threshold and then oscillates around it, spending nearly every trial in the only region that carries information.\n\nThis is the method article for perception work. If you have already established that a change needs to clear a [just-noticeable difference](/docs/just-noticeable-difference-product-changes) and that a raw \"did you notice\" rate needs a [control arm](/docs/signal-detection-theory-did-users-notice), this is how you actually run the study that produces the number.\n\n## The three classical methods, and why only one scales\n\nPsychophysics has been solving this problem since Fechner in 1860. Marjorie Leek's review (*Perception and Psychophysics*, 2001, volume 63, pages 1279-1292) sets out the trade-offs, and they map cleanly onto product research.\n\n**Method of constant stimuli.** Choose a fixed set of levels spanning the range \"from imperceptible to consistently perceptible\", present each one many times in random order, and fit a curve — the psychometric function — to the resulting accuracy at each level. This gives you the most complete picture available, including the slope of the function, which tells you how sharply performance changes as the stimulus changes.\n\nIt is also, as Leek puts it, \"very expensive in terms of experiment time\", because \"of necessity, many trials are placed at stimulus levels of the underlying psychometric function that are not informative about threshold\". If you only need the threshold, most of your budget goes to levels nobody was ever going to get wrong.\n\n**Method of limits.** Start well above or well below the threshold and step toward it until the answer flips. Cheap, and biased: participants anticipate the flip, and a purely descending series and a purely ascending series give different answers.\n\n**Adaptive procedures.** Leek's definition is the one to hold onto: \"An experimental procedure is said to be adaptive if the physical characteristics of the stimuli on each trial are determined by the stimuli and responses that occurred in the previous trial or sequence of trials.\" Staircase procedures, Leek writes, \"refine the method of limits by not requiring responses to a complete set of levels and by responding with changes in direction of the staircase after a change in the subject's response.\" Knowledge of the threshold \"increases systematically as the procedure is in progress\".\n\nThe efficiency difference is not marginal. Leek reports that for a 128-trial adaptive track, Watson and Pelli found \"an efficiency of 84% for their QUEST procedure, as compared with 40%-50% efficiency for the original PEST\" — and PEST was itself designed to be as efficient as possible, generations ahead of testing every level equally.\n\n## The transformed up-down staircase\n\nThe workhorse is the transformed up-down rule described by Levitt in 1971. A simple one-up, one-down staircase converges on the level where performance is 50% — which in a two-alternative forced-choice task is chance, and therefore useless. The transformation fixes this by requiring several correct answers before making the task harder.\n\n- **Two-down, one-up:** two consecutive correct responses make the task harder; one wrong response makes it easier. This \"targets the 70.7% level on the psychometric function\", because the probability of two correct in a row equals 0.50 when the per-trial probability is the square root of 0.50.\n- **Three-down, one-up:** targets 79.4%, since the cube root of 0.50 is 0.794.\n\nBoth are widely used. Two-down, one-up is the default for product work because it converges faster and 70.7% is comfortably above chance in a two-option comparison.\n\n### A worked track\n\nHere is a full two-down, one-up run on latency. The question on each trial is a forced choice: which of two builds felt faster. The stimulus level is the difference in milliseconds between the two builds, starting at 400 ms. The step size is 100 ms and drops to 50 ms after the second reversal, which is standard practice — a coarse step to arrive quickly, a fine step to measure precisely.\n\n| Trial | Level (ms) | Correct? | Note |\n| --- | --- | --- | --- |\n| 1 | 400 | yes | |\n| 2 | 400 | yes | two correct, step down |\n| 3 | 300 | yes | |\n| 4 | 300 | yes | two correct, step down |\n| 5 | 200 | no | reversal 1 (turnaround at 200) |\n| 6 | 300 | yes | |\n| 7 | 300 | yes | reversal 2 (turnaround at 300), step size now 50 |\n| 8 | 200 | no | reversal 3 |\n| 9 | 250 | yes | |\n| 10 | 250 | yes | reversal 4 (turnaround at 250) |\n| 11 | 200 | no | reversal 5 |\n| 12 | 250 | yes | |\n| 13 | 250 | yes | reversal 6 |\n| 14 | 200 | no | reversal 7 |\n| 15 | 250 | yes | |\n| 16 | 250 | yes | reversal 8 |\n| 17 | 200 | no | reversal 9 |\n| 18 | 250 | yes | |\n| 19 | 250 | yes | reversal 10 |\n| 20 | 200 | no | reversal 11 |\n\nThe track locks into an oscillation between 200 ms and 250 ms by trial 9 and stays there. Threshold is estimated, as Leek describes, \"commonly by averaging the levels at the direction reversals in the adaptive track (i.e., the turnaround points)\". Averaging the last six turnarounds — 250, 200, 250, 200, 250, 200 — gives **225 ms**.\n\nTwenty trials. Not two hundred. And notice where they were spent: after trial 8, every single trial was within 50 ms of the answer.\n\nAs a sanity check, that 225 ms sits in the same range as the published difference limen for visual duration discrimination at a one-second standard, which Rammsayer and colleagues put at a mean of 206.4 ms. Two very different procedures landing in the same neighbourhood is what a working method looks like.\n\n## Rules that keep a staircase honest\n\n**Discard the early reversals.** The first two or three turnarounds happen while the track is still travelling down from an artificially easy starting point. Average the last six, or the last even number of reversals after the step size reaches its final value.\n\n**Use an even number of reversals.** Turnarounds alternate between peaks and troughs. Averaging an odd number biases the estimate toward whichever type you have one extra of.\n\n**Start easy, and say so.** Beginning above threshold gives participants a few obvious trials to learn the task on. Those trials are discarded, and they are worth the cost.\n\n**Interleave two tracks.** Run one staircase starting from above threshold and one from below, shuffled together so the participant cannot predict the direction. This kills the anticipation bias that ruins the method of limits.\n\n**Insert catch trials.** Occasionally present two identical versions. A participant who confidently picks a winner on a catch trial is telling you their criterion is doing the work, not their perception.\n\n**Watch for a wandering floor.** If the track keeps descending without reversing, either the participant has learned an artefact — a visual cue that gives the answer away — or your step size is too small to escape. Both are design bugs, not results.\n\n## Distributing the staircase across participants\n\nThe one place product research must depart from the laboratory: psychophysics traditionally runs 100-plus trials with a handful of trained observers. You cannot ask a customer for a hundred forced-choice judgements.\n\nThe fix is to run the staircase **across** participants rather than within one. Each participant contributes six to twelve trials, and the track carries over: participant 14 starts wherever participant 13 left off. This produces a group threshold rather than an individual one, which is usually what a product decision needs anyway. The trade-off is that between-participant variance enters the track, so plan for more reversals — twelve to sixteen rather than six — and expect a wider confidence interval.\n\nIf you need individual thresholds — for accessibility work, where the interesting question is the spread rather than the mean — keep the tracks per-participant, shorten them, and accept a noisier estimate from each.\n\nOne caution on distributed tracks: run them in days, not months. A staircase spread across a quarter is measuring a reference point that is itself moving, which is a different problem entirely and is covered in [the zero point moving](/docs/adaptation-level-response-shift-research).\n\n## What to put on the stimulus axis\n\nThe method is indifferent to what varies, which is why it transfers well beyond latency:\n\n- **Price.** How large a discount before the offer registers as a deal. How large an increase before it registers as a rise.\n- **Notification frequency.** How many messages per week before the volume becomes noticeable, then annoying. These are two different thresholds and both are findable.\n- **Copy length.** How much longer before a description reads as long-winded.\n- **Visual weight.** Contrast, size, spacing — where a change stops being a refinement and starts being a redesign.\n- **Effort.** Number of steps, fields, or clicks before a flow feels heavier than it did.\n\nIn every case the level must be manipulable in fine increments and the judgement must be forced-choice. If you can only build two versions, you do not have a staircase; you have a single comparison, and you should read the [signal detection](/docs/signal-detection-theory-did-users-notice) guide instead.\n\n## Running an adaptive procedure in Koji\n\nLeek's definition of an adaptive procedure — the stimulus on each trial determined by the responses that came before — is a precise description of what an AI-moderated interview does by construction. Koji's interviewer chooses each follow-up based on what the participant just said, which is the same control structure a staircase needs, applied to conversation rather than tones.\n\nFor a threshold study, the pieces map directly onto the six structured question types:\n\n- **single_choice** for the forced-choice trial itself: which of these two felt faster, cheaper, heavier.\n- **yes_no** for catch trials and for the detection form of the question.\n- **scale** for a confidence rating on each trial, which lets you weight trials and reconstruct more of the psychometric function from the same data.\n- **ranking** to order three or more levels in a single trial, extracting more information per participant than a pair does.\n- **multiple_choice** for what the participant thinks drove the difference, which catches artefact learning early.\n- **open_ended** for the reasoning, where the AI interviewer probes rather than accepting a one-line answer.\n\nTwo things make this practical that were not practical before. First, **no moderator is needed for any session**, so a track spanning 40 participants runs overnight instead of over three weeks of scheduling. Second, the open-ended probe runs on every trial without adding moderator cost, so you find out that participant 22 was clicking based on a loading spinner rather than actual speed — the single most valuable thing a threshold study can tell you, and the thing a survey tool structurally cannot collect. Voice and text sessions both work, so you can match the modality to how the product is used.\n\nReports aggregate the track automatically, so the reversal sequence, the estimated threshold and the supporting quotes arrive together rather than requiring a manual reconstruction from a response export.\n\n## Frequently asked questions\n\n### What is an adaptive staircase procedure?\n\nIt is a testing method in which the difficulty of the next trial depends on the answers to previous trials. Correct answers make the task harder, wrong answers make it easier, and the sequence oscillates around the level where the participant is right some target proportion of the time. Because trials cluster near that level, an adaptive procedure reaches a usable threshold estimate in a fraction of the trials a fixed ladder requires.\n\n### Why does two-down, one-up target 70.7%?\n\nBecause the rule only makes the task harder after two consecutive correct answers, and the track settles where that event is as likely as not. If the probability of two correct in a row equals 0.50, then the per-trial probability is the square root of 0.50, which is 0.707. A three-down, one-up rule targets the cube root of 0.50, which is 0.794.\n\n### How many trials and participants does a threshold study need?\n\nA single well-behaved track can produce a working estimate in 20 to 40 trials, since only the trials near threshold carry information. Distributed across participants at eight to twelve trials each, that is roughly 3 to 5 people for a rough estimate and 20 to 40 for a stable one with a reportable interval. Plan for more reversals when the track is spread across people, because between-participant variance enters the track.\n\n### When should I use the method of constant stimuli instead?\n\nWhen you need the slope of the psychometric function, not just the threshold — for example when the question is how sharply sensitivity falls off, or when you are comparing the shape of the response between two user groups. The trade-off is cost: a fully sampled function requires many trials at levels that are uninformative about the threshold itself.\n\n### How do I stop participants from gaming a staircase?\n\nInterleave an ascending and a descending track so the direction is unpredictable, randomise which side of the screen each version appears on, insert catch trials with no real difference, and probe the reasoning after trials. Artefact learning — a participant detecting a loading indicator rather than the actual speed difference — shows up first in the open-ended answers, which is why probing every trial matters.\n\n### Can this measure price thresholds as well as perceptual ones?\n\nYes, and it is one of the better uses. A staircase on discount depth finds the smallest reduction that registers as a real offer, and a staircase on increase finds the largest rise that passes unremarked. Both are proportional to the base price, so express levels as percentages rather than absolute amounts and space them geometrically.\n\n## Related Resources\n\n- [Structured Questions in AI Interviews](/docs/structured-questions-guide) — the six question types that make up a trial.\n- [Just-Noticeable Difference: The Smallest Change Users Can Perceive](/docs/just-noticeable-difference-product-changes) — what a threshold means and why it is proportional.\n- [Did Users Actually Notice? Sensitivity vs Criterion](/docs/signal-detection-theory-did-users-notice) — the control arm every detection question needs.\n- [Statistical Power and Minimum Detectable Effect](/docs/statistical-power-minimum-detectable-effect) — sample sizing for the confirmatory study that follows.\n- [The Zero Point Moved: Adaptation and Response Shift](/docs/adaptation-level-response-shift-research) — why a threshold measured once does not stay valid indefinitely.\n- [Van Westendorp Price Sensitivity Meter](/docs/van-westendorp-price-sensitivity-meter) — the standard pricing instrument a price staircase complements.\n","category":"Research Methods","lastModified":"2026-08-23T03:24:55.363749+00:00","metaTitle":"Adaptive Staircase Threshold Testing for Product Research | Koji","metaDescription":"Method of constant stimuli, method of limits, and adaptive staircases compared. A full two-down one-up worked track, the rules that keep it honest, and how to distribute a staircase across participants.","keywords":["adaptive staircase method","perception threshold testing","method of constant stimuli","psychometric function research","two-down one-up procedure","threshold estimation ux","psychophysics product research"],"aiSummary":"An adaptive procedure chooses each trial from the answers to previous trials, so nearly every trial lands near the threshold instead of at levels nobody was going to get wrong. The transformed up-down rule described by Levitt targets 70.7% correct with a two-down one-up staircase and 79.4% with three-down one-up. A worked 20-trial track starting at 400 ms with a step drop from 100 ms to 50 ms after two reversals converges to an estimate of 225 ms from the last six turnarounds. Discard early reversals, average an even number, interleave ascending and descending tracks, and insert catch trials. For product work, distribute the staircase across participants at eight to twelve trials each rather than running 100-plus trials with one person.","aiPrerequisites":["A continuum you can vary in fine increments","Familiarity with forced-choice comparison questions"],"aiLearningOutcomes":["Choose between constant stimuli, method of limits, and adaptive procedures","Run a two-down one-up staircase and estimate a threshold from reversals","Apply the design rules that prevent anticipation and artefact learning","Distribute a staircase across many participants instead of one trained observer"],"aiDifficulty":"advanced","aiEstimatedTime":"12 min"}],"pagination":{"total":1,"returned":1,"offset":0}}