AI Over-Reliance and Automation Bias: How to Research Whether Users Trust Your AI Too Much (2026)

Answer first: Acceptance rate is not a quality metric. If users accept your AI feature's output whether or not it is correct, you have not built trust — you have built automation bias, and every model error now passes straight through to a decision. The metric that matters is appropriate reliance: users accept correct output and catch incorrect output. Measuring it requires a study design where you know the ground truth and the AI is sometimes wrong on purpose, because self-report is provably unreliable here. Koji runs the human half of that measurement at scale — AI-moderated interviews with the people who used the feature, structured questions that separate what users believe from what they did, and thematic analysis that explains why verification did or did not happen.

The three reliance states

Automation research has described this shape since long before large language models. Parasuraman and Riley's 1997 Human Factors paper "Humans and Automation: Use, Misuse, Disuse, Abuse" named the two symmetric failures: misuse (over-reliance on automation) and disuse (rejecting automation that would have helped). Both are calibration failures.

Most product dashboards can only see acceptance rate, which cannot distinguish the first row from the second. A feature at 90% acceptance is either excellent or dangerous, and the dashboard is silent about which.

Why you cannot just ask users

The most important finding for anyone designing this research is that people are bad at reporting their own reliance — badly enough that a self-report study will actively mislead you.

METR's 2025 randomized controlled trial (arXiv:2507.09089) is the cleanest demonstration. Sixteen experienced open-source developers worked 246 real tasks on mature repositories, with each task randomly assigned to allow or disallow AI tooling. They completed tasks 19% slower with AI available. Afterwards, they estimated AI had made them 20% faster. That is a 39-point gap between measured and perceived outcome, in a population of expert engineers on their own codebases. (METR itself now flags the result as historical with respect to current tooling — the tooling has moved; the measurement lesson has not.)

The implication for research design is blunt: a survey asking "do you check the AI's output?" measures self-image, not behaviour. You need designs where the AI's correctness is known and the user's response is observed.

The academic base you should build on

Two CHI-lineage studies define the field, and both produce findings that are uncomfortable for the standard product playbook.

Explanations do not fix over-reliance. Bansal et al., Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (CHI 2021), ran mixed-method studies across three datasets with an AI whose accuracy was comparable to the humans'. AI assistance did improve team performance — but explanations did not increase that complementary gain. Explanations tended to increase acceptance of the AI's recommendation regardless of whether it was correct. If your plan for over-reliance is "add explainability", the evidence does not support it.

What does work is friction — and users hate it. Buçinca, Malaya and Gajos, To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-Assisted Decision-Making (Proc. ACM Hum.-Comput. Interact. 5, CSCW1, April 2021), tested three cognitive forcing interventions — designs that require an explicit analytical act before the user can accept a recommendation — against two explainable-AI approaches and a no-AI control, with N=199 participants. Cognitive forcing significantly reduced over-reliance where explanations did not. Two caveats make the paper genuinely useful rather than merely citable:

• The benefit concentrated in participants with high Need for Cognition, meaning the intervention helps least the users most at risk.
• In the authors' words, "People assigned the least favorable subjective ratings to the designs that reduced the overreliance the most."

That second finding is the reason this needs to be a research programme rather than a design decision. The intervention that protects your users will lose an A/B test scored on satisfaction. You have to measure both, and decide deliberately.

Study design 1: the seeded-error task (the gold standard)

This is the only design that measures reliance directly.

1. Build a task set with known ground truth. 12–20 realistic tasks from the actual workflow.
2. Seed errors deliberately. Have the AI produce a wrong-but-plausible answer on a known subset — 25–30% is a common ratio. Plausibility is the whole game; an obviously wrong answer measures nothing.
3. Randomise assignment so no participant can pattern-match which items are seeded.
4. Record the decision, not the opinion. Did they accept, edit, or reject? How long did they take? Did they consult the source?
5. Compute the two rates that matter: agreement on correct items and agreement on incorrect items. Over-reliance is the second number.
6. Then interview. The number tells you the rate; only the conversation tells you the reason.

Ethics note: seeded errors are deception, and they need handling. Use synthetic or sandboxed tasks rather than live consequential decisions, disclose in the debrief, and get the design reviewed. See research ethics and informed consent for the standard.

Study design 2: the retrospective reliance interview

When you cannot run a controlled task — because the workflow is consequential, regulated, or simply too long — interview users about specific recent instances instead of general habits. The Critical Incident Technique is the right frame: anchor every question to a concrete, recent, remembered event rather than a self-assessment.

Questions that produce honest data:

• "Walk me through the last time the AI gave you something you didn't use. What tipped you off?" (Users who cannot recall a single instance are almost certainly over-relying.)
• "Tell me about a time you found out afterwards that it was wrong. How did you find out?"
• "What would have to be on screen for you to double-check?"
• "When you're rushed, what do you skip?" (Time pressure is the strongest reliance moderator in practice.)
• "Who checks your work after you use this?" (Reveals whether the organisation has a real backstop or an assumed one.)

Never ask "do you verify the output?" — it has one socially acceptable answer.

The modern approach: measuring reliance with Koji

Reliance research has historically been lab work: a study room, a facilitator, twenty participants, six weeks, and a researcher with an HCI background. That cost is exactly why most product teams skip it and ship acceptance rate instead. An AI-native platform collapses the expensive parts.

• Run the debrief and the retrospective study at scale. Koji conducts AI-moderated interviews asynchronously and around the clock, so 60 participants across three user segments complete in days. The same study run by one human moderator serially is a month of calendars.
• Structured questions built for reliance measurement. All six types earn their place here: yes_no for "did you check the source before acting?", scale for confidence and perceived accuracy — which you then compare against measured accuracy to quantify the calibration gap directly — single_choice for what the user did with the last suggestion, multiple_choice for which verification behaviours they use, ranking for which signals would most prompt a second look, and open_ended with AI follow-up probing for the incident narrative. The structured questions guide covers how each type is asked and charted.
• Consistency the lab cannot match. Reliance questions are exquisitely sensitive to phrasing and moderator tone — an approving nod is enough to move an answer. An AI moderator asks all 60 participants the same probe in the same way, removing the variance described in interviewer bias.
• Automatic thematic analysis. Reliance rationales cluster into a handful of mechanisms — time pressure, perceived authority of formatting, prior success streak, absence of any checking mechanism — and the analysis surfaces them across the whole set as interviews complete rather than after two days of tagging.
• Customisable AI consultants. Load your domain context so the interviewer can tell a genuinely careful verification story from a plausible-sounding one, the way a domain-expert moderator would.
• Real-time reporting. Calibration gaps and verification rates update live, so the design decision does not wait on fieldwork closing.

Where a legacy survey tool gives you a self-reported verification rate that METR's data tells you not to believe, this gives you the observed behaviour, the stated belief, and the distance between them — which is the finding.

What to do with the finding

• Reduce the consequence, not just the error rate. Undo, staged commit, preview-before-apply and reversible defaults blunt over-reliance without asking users to be more vigilant.
• Deploy cognitive forcing where stakes justify friction. Withhold the recommendation until the user commits to an initial judgement; require a source click on high-stakes items. Expect satisfaction scores to dip and plan the trade-off explicitly.
• Show uncertainty honestly and sparingly. Uncertainty that is always high is ignored; uncertainty that is well calibrated changes behaviour.
• Stop formatting wrong answers like right ones. Confident prose and clean tables are themselves a reliance signal.
• Watch time pressure. Over-reliance is a workload phenomenon before it is an attitude. Study your busiest users, not your happiest.
• Re-measure on every model change. A more accurate model can increase absolute harm if it raises reliance faster than it raises accuracy.

Common mistakes

• Treating acceptance rate as a success metric. It rises with both quality and complacency.
• Asking users whether they verify. Self-report and behaviour diverge by a wide margin.
• Shipping explanations as an over-reliance fix. The CHI 2021 evidence says explanations raise acceptance of wrong answers too.
• Testing only with careful, motivated users. Cognitive forcing works least on the low-Need-for-Cognition users who need it most, and lab volunteers skew high.
• Running the study once. Reliance grows with familiarity; a two-week-old feature has not accumulated the trust that causes the problem.
• Optimising purely for satisfaction. The safest designs score worst on subjective rating. Decide that trade-off on purpose.

Frequently asked questions

What is automation bias? Automation bias is the tendency to over-trust automated output — accepting a system's recommendation without adequate verification, and sometimes overriding one's own correct judgement to agree with it. Parasuraman and Riley (1997) framed it as misuse of automation, one of two symmetric calibration failures alongside disuse, where users ignore automation that would have helped them.

How is over-reliance different from just trusting the AI? Trust is a belief; reliance is a behaviour. Appropriate reliance means the behaviour tracks the AI's actual correctness — accepting good output and catching bad output. Over-reliance means acceptance is disconnected from correctness. You can only tell them apart by observing what users do when the AI is wrong, which is why seeded-error designs exist.

Do explanations reduce over-reliance? Generally not. Bansal et al. (CHI 2021) found that AI explanations did not increase complementary team performance and tended to raise acceptance of recommendations regardless of correctness. Cognitive forcing functions - designs that require an explicit analytical step before accepting - did reduce over-reliance in Buçinca et al. (2021), but that benefit concentrated in users with high need for cognition, and users rated those designs least favourably.

How do I measure over-reliance without deceiving users? Two options. Use sandboxed or synthetic tasks with seeded errors and a full debrief, which is standard practice with ethics review. Or use retrospective critical-incident interviews anchored to specific recent events - "the last time you didn't use what it gave you" - which avoid deception entirely at the cost of some precision. Many teams run the interview study first and the controlled task only for high-stakes workflows.

How many participants do I need? For a seeded-error task study, 30 to 40 participants per segment gives stable acceptance rates on incorrect items. For retrospective reliance interviews, 25 to 40 typically reaches saturation on the mechanisms driving verification behaviour. Because AI-moderated interviews run in parallel and asynchronously, both are days rather than weeks of fieldwork.

Is high acceptance rate ever a good sign? Only when paired with a measured over-reliance rate. High acceptance with low acceptance-of-incorrect-output means the AI is accurate and users are calibrated - genuinely good. High acceptance with high acceptance-of-incorrect-output means users have stopped checking, and every future model regression will ship straight through to a decision.

When should we re-run this research? On every model or prompt change that alters accuracy or tone, when you add a new user segment, and at least twice a year on a stable feature. Reliance builds with familiarity, so a study run in week two of a launch will systematically understate the problem you will have in month six.

Related Resources

• Structured Questions in AI Interviews — the six question types and how each maps to a reliance metric
• User Research for AI Products — the wider programme this study sits inside
• Human Evaluation of AI Outputs — establishing the ground truth your seeded-error study needs
• AI Failure Mode Analysis (FMEA) — scoring detection when users have stopped checking
• Cognitive Biases in User Interviews — the biases that distort reliance self-report
• Critical Incident Technique — the interview frame for retrospective reliance studies
• Research Ethics and Informed Consent — handling deception in seeded-error designs