AI Guardrail Testing: How to Measure False Refusals and Over-Blocking with Real Users (2026)

Answer first: Every AI guardrail is a classifier, and every classifier has a false positive rate. When it fires on a legitimate request, the user does not file a bug — they leave. That makes over-blocking the most under-measured failure in AI products: invisible in your incident data, invisible in your support queue, and directly visible in your churn. The fix is a two-sided measurement discipline. Track false refusal rate (FRR) on benign traffic alongside your harm-catch rate, then run user research on the refusals themselves to learn what people were actually trying to do. Koji makes the second half practical — AI-moderated interviews with the users who got blocked, structured questions that quantify how often it happens and what they did next, and thematic analysis that turns a pile of refusals into a policy change.

Guardrails have two error types, and teams only measure one

A guardrail is any layer that inspects input or output and decides whether to allow it: a moderation classifier, a policy prompt, a keyword filter, a refusal-trained model, a tool-permission check. Whatever the implementation, it produces four outcomes:

Almost every AI safety programme instruments the top-right-to-bottom-left diagonal poorly. False negatives get red teams, incident reviews and dashboards. False positives get nothing, because the signal is a user quietly deciding your product cannot help them. The safety literature has a name for this: exaggerated safety.

The asymmetry is not small. Consider the arithmetic on a consumer-scale product: a guardrail with a 4% false positive rate blocks 40,000 benign requests per million; at 15% it blocks 150,000. Those are not edge cases. That is a product experience.

How bad is over-refusal in practice? The benchmark evidence

Three benchmarks are worth knowing, because they establish that this is a measurable engineering property, not a matter of taste.

• XSTest (Röttger et al., 2024) introduced the term exaggerated safety with a deliberately small, hand-crafted suite: 250 safe prompts that superficially resemble unsafe ones, paired with 200 contrast unsafe prompts. The pairing is the clever part — it stops a model from scoring well by simply refusing less.
• OR-Bench (Cui et al., ICML 2025) scaled the same idea to 80,000 over-refusal prompts across ten rejection categories, plus a ~1,000-prompt hard subset and 600 genuinely toxic prompts to prevent gaming, and evaluated 32 models across 8 families.
• RefusalBench (arXiv:2605.21545) held task framing constant while varying only risk tier across 141 prompts in 47 bundles. Across 19 frontier models in a May 2026 snapshot, strict refusal rates on identical prompts spanned 0.1% to 94.6%.

Read that last figure again. On the same prompts, models disagree about whether to answer by nearly the entire range. That means your refusal behaviour is not a property of the request — it is a property of the model and policy you happened to choose, and switching models silently rewrites your product's boundaries.

The mechanism is equally unflattering. Over-refusal is frequently mediated by early-layer attention to sensitive keywords rather than robust contextual analysis — the model sees "exploit" or "kill" and refuses before it has meaningfully processed the sentence. That is why a nurse asking about medication thresholds, a security engineer asking about a CVE, and a novelist asking about a fight scene all get refused, and why the refusal feels stupid to the person receiving it.

Domain context makes it worse, not better: research on small on-premises models in criminal legal contexts found that prompting for legal context increased objections (arXiv:2606.24585) — exactly the professional users who most need the tool.

Step 1: Instrument refusals before you research them

You cannot interview users about refusals you never logged. Minimum instrumentation:

• A refusal label on every response. Not just hard blocks — capture soft refusals and partial hedges, which are the ones users experience as unhelpfulness rather than policy.
• Which layer fired. Input classifier, policy prompt, output filter, tool permission. Different owners, different fixes.
• What the user did next. Rephrased, escalated, abandoned the session, cancelled the subscription. Abandonment after refusal is your single best over-blocking proxy metric.
• A per-segment refusal rate. Refusal rates are almost never uniform. Healthcare, legal, security and safeguarding users get refused far more than average, and they are frequently your highest-value accounts.

Step 2: Build a benign-but-borderline test set

Your eval set needs a matched design or the numbers will lie. Follow the XSTest pattern: for every category of genuinely harmful request you block, write a benign twin that shares its surface features.

Score both sides. A model that improves on the benign side while degrading on the harmful side has not improved. This is the same discipline as building an evaluation golden set, applied to the safety layer specifically.

Step 3: Research the refusals — the part evals cannot do

Benchmarks tell you your FRR on prompts you wrote. They cannot tell you what your users were actually trying to accomplish, what refusal cost them, or whether the wording of the refusal made things better or worse. That requires talking to blocked users.

A well-formed over-blocking study asks five things:

1. What were you trying to do? The task, not the prompt. Refusals are usually policy failures on legitimate professional tasks.
2. What did you do after? Rephrased, went elsewhere, gave up, used a personal account with a different tool. Workarounds are the highest-severity finding here — they mean the request happened anyway, outside your logging.
3. How did the refusal make you feel about the product? Over-refusal reads as accusation. Users describe being treated as suspects.
4. Was the refusal understandable? A refusal that explains the boundary and offers a path forward costs a fraction of one that says "I can't help with that."
5. How often does this happen to you? Frequency per user, not aggregate rate, is what predicts churn.

The modern approach: running the study with Koji

The old way to answer these questions is a support-ticket audit plus a survey, which reaches the users who complained — a sample biased toward people who stayed. The users who matter most are the ones who left silently.

Koji closes that gap:

• Trigger interviews from the refusal event itself. Because the study runs asynchronously and 24/7, you can invite a user into an AI-moderated interview shortly after a refusal, while the task is still fresh, instead of surveying a month later when they have rationalised it.
• Six structured question types that quantify the qualitative. Use yes_no for "did you complete the task another way?", scale for frequency and frustration, single_choice for what they did next, ranking for which blocked task types matter most, multiple_choice for the workaround inventory, and open_ended — with AI follow-up probing — for the actual task narrative. Each structured answer is charted automatically in the report; see the structured questions guide for the full behaviour of each type.
• AI-moderated depth without moderator bias. A human moderator asking "was the block reasonable?" invites courtesy bias; a consistent AI interviewer asks every participant the same probe the same way, which is exactly the variance problem interviewer bias describes.
• Automatic thematic analysis. Forty refusal narratives cluster into five or six policy problems in minutes rather than the two days a manual affinity map takes. Traditional survey tools like SurveyMonkey give you a bar chart of refusal categories; they cannot tell you that eleven of your enterprise users have moved the task to a personal ChatGPT account.
• Real-time reporting. Themes, quotes and quality scores appear as interviews complete, so a policy decision does not wait for fieldwork to close.
• Voice interviews. For emotionally loaded refusals — safeguarding, health, finance — voice gets markedly richer answers than a text box.

Teams that pair benchmark FRR with a live refusal-interview loop typically find that a third or more of their "harmful" blocks are professional users doing their jobs. You do not need a research team to find that out; you need forty conversations.

Step 4: Tune with an explicit trade-off, and write it down

Decide and document the operating point. Two things make this decision defensible:

• Segment your policy, do not average it. A verified clinical account and an anonymous free-tier session should not share a refusal threshold. Most over-blocking pain is a segmentation failure wearing a safety costume.
• Fix the refusal message, not just the threshold. A refusal that names the boundary, explains why, and offers an alternative path converts a churn event into a friction event. This is the cheapest intervention available and almost nobody tests it. Test the wording as its own variable.

Then keep both numbers on the same dashboard. A safety review that reports catch rate without FRR is reporting half of a classifier.

Where guardrail testing sits in your AI research programme

• Red teaming measures the false-negative side: what gets through that should not.
• Guardrail testing measures the false-positive side: what gets blocked that should not.
• FMEA puts both on one worksheet and forces you to score them against user harm.
• Trust and safety research covers the reporting and moderation flows that sit downstream of both.

Run only the first and you will ship a product that is safe and unusable. That is a real failure mode, and it has a churn number attached.

Common mistakes

• Reporting catch rate without false refusal rate. Half a confusion matrix is not a safety metric.
• Testing only prompts your team wrote. Your users' benign-but-borderline requests are stranger and more professional than anything you will imagine.
• Treating a model swap as behaviour-neutral. Refusal rates on identical prompts span nearly the full range across frontier models. Re-run the suite on every model change.
• Surveying only users who complained. The severe cases are silent. Recruit from the refusal log, not the support queue.
• Averaging refusal rate across segments. It hides the professional cohorts getting refused several times a week.
• Leaving the refusal copy untested. The message does more work than the threshold.

Frequently asked questions

What is a false refusal rate (FRR)? FRR is the percentage of benign, policy-compliant requests that your AI system refuses or blocks. It is the false positive rate of your safety layer. You measure it on a curated set of benign prompts — ideally matched against harmful twins so a model cannot score well by simply refusing less often — and you report it alongside your harm-catch rate, never instead of it.

Why is over-refusal harder to detect than harmful output? Harmful output generates complaints, press coverage and incident tickets. Over-refusal generates silence: the user rephrases once, gives up, and either uses a competitor or moves the task to a personal account outside your logging. Nothing in your telemetry distinguishes "user was refused and left" from "user finished early" unless you explicitly instrument it.

How do I know if my guardrails are too strict? Three signals. First, benchmark FRR on a benign-but-borderline suite — XSTest-style matched pairs are the cheapest starting point. Second, session abandonment rate immediately following a refusal. Third, interviews with refused users: if a meaningful share describe legitimate professional tasks and report completing them elsewhere, your guardrails are too strict for that segment.

Should refusal thresholds be the same for all users? Usually not. A verified clinician asking about drug interactions and an anonymous free-tier user asking the same question carry different risk profiles. Segment-aware policy, with the segmentation logic documented and auditable, resolves most over-blocking complaints without weakening protection where it matters. Document the reasoning — this is the kind of decision the EU AI Act expects you to be able to explain.

How many users do I need for an over-blocking study? Thirty to fifty refused users is usually enough to identify the recurring policy failures, because refusal categories concentrate quickly. If you need per-segment numbers — clinical, legal, security — budget 25–30 per segment. Because interviews run asynchronously and in parallel, that is days of fieldwork, not weeks.

Does a better refusal message actually help? Materially, yes — and it is the intervention teams most often skip. A refusal that names the boundary, explains the reason, and offers a supported alternative changes the user's read from "this product thinks I am a criminal" to "this product has a rule." Treat the copy as a testable variable with its own metric, not as a string constant.

Can Koji help us run refusal research? Yes. Recruit directly from your refusal log, run AI-moderated interviews asynchronously so you reach users while the blocked task is fresh, use structured questions to quantify frequency, workaround behaviour and severity, and let automatic thematic analysis cluster the narratives into policy changes. What would take a researcher three weeks of scheduling and synthesis takes a few days.

Related Resources

• Structured Questions in AI Interviews — the six question types and when to use each
• AI Red Teaming with Real Users — measuring the other error type
• AI Failure Mode Analysis (FMEA) — scoring over-refusal against every other failure
• Evaluation Datasets for AI Products — building the matched benign/harmful suite
• Trust and Safety Research — reporting and moderation flows downstream of guardrails
• Interviewer Bias — why refusal interviews need consistent moderation
• The EU AI Act and User Research — documenting safety-layer decisions