AI Explainability Testing: How to Find Out Whether Your Explanations Actually Help Users (2026)

Answer first: Do not measure whether users like your explanation. Measure whether it changes what they do. The two come apart badly: in controlled experiments, explanations that participants rate as helpful routinely fail to improve decision accuracy, and content-free "placebic" explanations produce trust ratings statistically indistinguishable from real ones. The only defensible test of an AI explanation is whether people accept correct outputs more often and reject incorrect ones more often than they do without it. That requires a study with known ground truth, not a satisfaction survey.

This is the constructive counterpart to AI over-reliance and automation bias. That guide covers the diagnosis — users trusting AI output more than it deserves. This one covers the intervention most teams reach for first, and the evidence on whether it works.

The finding that should change your test plan

Poursabzi-Sangdeh, Goldstein, Hofman, Wortman Vaughan and Wallach, "Manipulating and Measuring Model Interpretability," CHI 2021, ran a sequence of pre-registered experiments with N = 3,800 participants. Participants saw functionally identical models that differed only in two factors widely assumed to make a model more interpretable: the number of features and whether the model's internals were shown (clear) or hidden (black box).

The results, in the authors' own words:

• Participants who saw a clear model with few features could better simulate the model's predictions — the transparency worked, in the narrow sense that people could predict what it would say.
• But "we did not find that participants more closely followed its predictions" — better understanding did not translate into appropriate reliance.
• And the finding that matters most: "showing participants a clear model meant that they were less able to detect and correct for the model's sizable mistakes, seemingly due to information overload."

The transparent model made people worse at catching its errors. Not neutral — worse.

The authors' conclusion is the one to write on the wall: the results demonstrate "the importance of testing over intuition when developing interpretable models."

The second finding: your trust metric is measuring the wrong thing

Eiband, Buschek, Kremer and Hussmann, "The Impact of Placebic Explanations on Trust in Intelligent Systems," CHI 2019 Extended Abstracts, ran a lab study with 30 participants across three conditions on a nutrition-recommender prototype: no explanation, a placebic explanation, and a real explanation.

A placebic explanation is one with the form of an explanation and none of the content — the AI-interface equivalent of Langer's classic photocopier study, where "may I use the copier, because I need to make copies" secured nearly as much compliance as a real reason. The finding: placebic explanations invoked perceived levels of trust similar to real explanations.

The implication is uncomfortable and specific. If your explainability evaluation consists of asking users how much they trust the system and how helpful the explanation was, a placebo would pass your test. Any measurement instrument that cannot distinguish a real explanation from an empty one is not measuring explanation quality.

What to measure instead: the four outcomes

Replace "was the explanation helpful?" with four measures that have ground truth behind them.

Appropriate reliance is two numbers, never one. An explanation that raises acceptance of everything has made things worse, not better — it has increased compliance, not discrimination. Report acceptance-of-correct and rejection-of-incorrect separately, always. A design that improves the first while flattening the second is the automation-bias failure wearing a helpful mask.

Calibration is the cheapest high-value measure. Ask for a confidence rating on every item, then compare it against actual correctness. High confidence on wrong answers is the state you are trying to eliminate; a good explanation should lower confidence specifically on the wrong ones. If confidence rises uniformly, the explanation is functioning as a trust signal rather than as information.

What social science says an explanation is

The reason so many AI explanations fail is that they are built to describe the model rather than to answer a person's question.

Tim Miller, "Explanation in Artificial Intelligence: Insights from the Social Sciences," Artificial Intelligence 267 (2019), pp. 1–38, surveys decades of philosophy, cognitive psychology and social psychology, and draws three conclusions that map directly onto design decisions:

1. Explanations are contrastive. People do not ask "why P?" They ask "why P rather than Q?" A feature-importance list answers the first question. Users asked the second.
2. Explanations are selective. People expect one or two causes, not the complete causal chain. This is precisely the information-overload mechanism the CHI 2021 experiments observed: a full feature list is a more complete explanation and a worse one.
3. Explanations are social. They are a conversation aimed at transferring knowledge to a particular person with particular prior beliefs — not a static readout.

Those three findings explain most of the field's disappointing results. SHAP plots, attention maps and full feature lists are non-contrastive, non-selective, and non-social. They are honest descriptions of a model that answer a question nobody asked.

The design move: state the contrast explicitly. "Flagged as high-risk rather than medium-risk because the transaction was 8× this account's typical amount" is contrastive, selective, and social. "Feature importances: amount 0.42, location 0.31, time 0.11" is none of the three, and is more complete.

A study design that actually tests an explanation

Run this as a between-subjects comparison. Within-subjects contaminates the second condition because participants carry over what they learned.

1. Pick the decision, not the screen. The unit of analysis is a task with a right answer — approve/reject, act/escalate, use/edit. If there is no ground truth, you cannot measure appropriate reliance, and you are back to satisfaction.

2. Seed known errors — including plausible ones. Roughly 20–30% incorrect outputs is a workable mix. Critically, seed plausible errors, not absurd ones. Absurd errors are caught without any explanation and will make any condition look good.

3. Run at least three conditions. No explanation; your real explanation; and — if you want to know whether your explanation contains information or only signals effort — a placebic control with the same length and visual form but no diagnostic content. Most teams skip the placebic arm. It is the arm that tells you whether you built anything.

4. Capture per-item confidence before revealing correctness. This is your calibration data and it is worthless if collected retrospectively.

5. Probe the reasoning, not the preference. After the task, ask what the participant thought the system was doing and why they accepted or rejected each item. This is where you find out whether they read the explanation, misread it, or ignored it entirely — a distinction no rating scale captures.

6. Analyse the two reliance rates separately. Then look at whether confidence separated correct from incorrect items. That separation, not the average trust score, is your result.

How Koji helps

The reason most teams ship explanations they have never tested is that the study above — multiple conditions, dozens of participants each, per-item ratings plus qualitative probing — is expensive with traditional tooling. Three arms at n=50 is 150 sessions. Nobody schedules that for a UI copy decision.

Parallel AI-moderated sessions collapse the timeline. Koji runs every session concurrently, so a three-arm, 150-participant explainability study runs in an afternoon rather than over six weeks. That is the difference between testing your explanation and assuming it works.

Structured questions give you the quantitative arm; conversational probing gives you the qualitative one — from the same session. Koji supports six question types — open_ended, scale, single_choice, multiple_choice, ranking, and yes_no. A single_choice per item captures accept/reject. A scale captures per-item confidence, which is your calibration data. A yes_no captures whether they read the explanation at all. Then open_ended follow-ups, with the AI probing until the answer is concrete, capture why. Traditional survey platforms like SurveyMonkey or Qualtrics can collect the ratings but cannot ask "you said the explanation was clear — what did you think it was telling you?" and keep digging when the first answer is vague. See Structured Questions in AI Interviews.

The calibration cross-tab is automatic. Because the confidence scale answers and the accept/reject single_choice answers carry stable question IDs through to the report, Koji aggregates confidence against correctness without anyone building a spreadsheet. That cross-tab is the single most informative output of an explainability study, and it is usually the one that never gets built.

Automatic thematic analysis across conditions. Koji codes the open-ended responses and surfaces where the explanation arm's reasoning differs from the control's. If participants in both arms describe the same mental model, your explanation is not transmitting information — which is the placebo result, detected without you having to run the placebo arm blind.

No research background required. A PM can define the three conditions, launch, and read the calibration result themselves. That is what makes it realistic to test explanation copy the way you would test onboarding copy — before shipping, and again after the next model change.

Common mistakes

Measuring satisfaction. Placebic explanations pass satisfaction tests. If your metric cannot distinguish a real explanation from an empty one, it is not a metric.

Reporting one reliance number. Overall accuracy hides the case where acceptance rose for everything. Always split accept-correct from reject-incorrect.

Showing everything. More complete is not more useful. The CHI 2021 information-overload result is the direct consequence of maximal transparency.

Non-contrastive explanations. Feature importances answer "why P" when users asked "why P rather than Q." Name the alternative.

Seeding only obvious errors. Everyone catches those. Plausible errors are where explanations either earn their place or don't.

Testing once. Explanation quality is coupled to model behaviour. After a model swap, the explanation may be describing something that no longer happens — see research refresh cadence.

Assuming an explanation is the right intervention at all. Sometimes the answer is a cheaper undo, a visible confidence indicator, or a forcing step that makes the user commit before seeing the AI's answer. Explanations are one option among several, and the evidence for them is weaker than the industry assumes.

Frequently asked questions

Do AI explanations actually improve user decisions? Often not. In Poursabzi-Sangdeh et al. (CHI 2021, N=3,800), participants shown a clear model with few features were less able to detect and correct the model's sizable mistakes, apparently from information overload — and did not follow its predictions more closely despite understanding it better. Explanations can help, but that is an empirical question about your specific design, not a property of explanations in general.

Why can't I just ask users whether the explanation was helpful? Because a placebo passes that test. Eiband et al. (CHI 2019, N=30) found placebic explanations — the form of an explanation with no real content — produced trust levels similar to real explanations. Perceived helpfulness measures whether the interface signalled effort, not whether it transmitted information.

What should I measure instead? Four outcomes with ground truth behind them: appropriate reliance (reported as two separate rates — accepting correct outputs and rejecting incorrect ones), error detection on seeded plausible errors, calibration (per-item confidence cross-tabbed against actual correctness), and actionability (can the user state something specific and correct they would do differently).

What makes an explanation good, according to the research? Miller (Artificial Intelligence, 2019) synthesises the social-science literature into three properties: explanations are contrastive (they answer "why P rather than Q"), selective (one or two causes, not the full causal chain), and social (a conversation with a particular person, not a static readout). Feature-importance plots and attention maps satisfy none of the three.

How many participants do I need for an explainability study? Enough per condition that a difference in rates is meaningful — dozens per arm, not the 5 to 8 of discovery work. A three-arm design (no explanation, real explanation, placebic control) at 50 per arm is a reasonable target. Running the arms in parallel is what makes that affordable.

Should I include a placebic control condition? If you want to know whether your explanation contains information rather than merely signalling effort, yes. Match its length and visual form to the real explanation but strip the diagnostic content. If the two arms produce the same reliance and calibration results, your explanation is decorative — and that is worth discovering before launch rather than after.

Is this the same as testing for over-reliance? They are two halves of one question. Over-reliance testing asks whether users trust the output too much; explainability testing asks whether a specific intervention fixes that. Run them together: the same seeded-error task supports both, and the accept-correct versus reject-incorrect split is the shared measure.

Related Resources

• Structured Questions in AI Interviews — the six question types behind the confidence-versus-correctness cross-tab
• AI Over-Reliance and Automation Bias — the diagnosis this intervention is meant to treat
• Human Evaluation of AI Outputs — rating protocols and inter-rater reliability
• Acceptance Criteria for AI Features — turning detectability into a ship threshold
• AI Model Cards and User Disclosure — documenting what the explanation can and cannot convey
• AI Failure Mode Analysis — choosing which errors to seed
• Can You Trust AI Interviewers? — how Koji handles hallucination and bias in the research instrument itself
• User Research for AI Products — the wider practice this sits inside

Want to know whether your explanation does anything? Start free with 10 credits and run a three-arm study this week.