Accents, Dialects and AI Transcription: How to Check Your Voice Research Works for Everyone

Automatic speech recognition does not fail at random. It fails more often for some speaker groups than others, and if you run voice research without measuring that gap you will publish findings that are systematically thinner for the participants the model handles worst. Measure word error rate by speaker group before you trust voice-only data, and offer text as an equal-quality alternative rather than a fallback.

This is not an argument against voice research. Voice interviews produce richer, longer, more spontaneous answers than typed ones, and for many populations they are dramatically easier to complete. It is an argument for knowing where your instrument is weak, in the same way you would want to know that a survey question performs differently across segments.

The evidence, and what it actually says

The landmark study is Koenecke and colleagues in PNAS (2020), which tested five commercial ASR systems — from Amazon, Apple, Google, IBM and Microsoft — against structured interviews with 42 white speakers and 73 Black speakers across five US cities, 19.8 hours of audio, matched on the age and gender of the speaker.

Average word error rate: 0.35 for Black speakers against 0.19 for white speakers. Every one of the five systems showed the disparity. Error rates were highest for African American men.

The methodological detail that makes the finding hard to dismiss: the gap was equally large on a subset of identical phrases spoken by both groups. That rules out the comfortable explanation that the models simply encountered unfamiliar vocabulary. The disparity traces to the acoustic models — to how the systems hear, not to what they were asked to understand.

Models have improved substantially since 2020, so it is fair to ask whether this is a historical problem. The more recent work says the shape of the problem persists even as absolute error falls:

• Graham and Roll (2024), JASA Express Letters, evaluated Whisper across native and non-native English accents. Accuracy was higher for American and Canadian speakers than for British and Australian ones, and native accents outperformed non-native accents overall. The authors linked errors to L1 prosody, vowel inventory, and — importantly — to speech type, with worse performance on spontaneous speech than read speech.
• McGuire (2025) assessed five current ASR systems on the L2-ARCTIC corpus, covering speakers with Arabic, Chinese, Hindi, Korean, Spanish and Vietnamese first languages. On read speech, Whisper and AssemblyAI achieved mean match error rates of 0.054 and 0.056 — genuinely close to human transcription.
• Clinical speech research published in npj Digital Medicine (2026) found error rates significantly higher for non-native speakers when testing Whisper and WhisperX on clinical conversation.

Put those together and the practical conclusion is specific rather than alarmist: modern ASR is close to human accuracy on read speech from speakers it was well trained on, and materially worse on spontaneous, accented, domain-heavy conversation — which is exactly what a research interview is.

Benchmark numbers you see quoted in vendor marketing are almost always measured on read corpora. Your interviews are not read corpora.

Why word error rate is the wrong headline metric for research

WER counts substitutions, deletions and insertions against a reference transcript:

WER = (S + D + I) / N

where N is the number of words in the reference. It is a useful engineering metric and a poor research metric, because it weights every word equally. Your analysis does not.

Consider two transcripts of the same 200-word answer:

Transcript B has half the word error rate and is far more damaging. Research analysis depends disproportionately on a small set of high-information tokens: product names, feature names, competitor names, numbers, negations, and clinical or technical terms. A dropped "not" inverts a finding. A misheard competitor name lands in the wrong theme.

So measure two things:

1. WER — for comparison across speaker groups, because it is the standard and it is what the literature reports.
2. Concept error rate — the proportion of domain-critical tokens transcribed incorrectly. Define the token list before you start: your product names, your competitors, your feature vocabulary, numerals, and negations. This is the metric that predicts whether your analysis will be wrong.

A voice study can be perfectly usable at 15% WER and unusable at 8%, depending entirely on where the errors land.

How to audit your own study in an afternoon

You do not need a research programme to do this. You need ten interviews and about three hours.

Step 1 — Stratify your sample. Group completed interviews by whatever speaker characteristics plausibly matter for your population: first language, region or country, and any accent groups you can identify from recruitment data. You are not making claims about individuals; you are checking whether your instrument performs evenly across groups you are trying to represent.

Step 2 — Sample within strata. Take at least three interviews per group. Fewer than that and you are measuring individual variation.

Step 3 — Take a fixed window. Human-correct a five-minute stretch from the middle of each interview — not the start, where speech is more scripted and warm-up questions are simpler.

Step 4 — Produce a reference transcript. Correct the machine transcript by ear. Keep normalisation rules consistent: decide up front how you handle contractions, numerals and filler words, and apply the same rules to every group. Inconsistent normalisation is the most common way these audits produce a fake disparity.

Step 5 — Compute both metrics per group. WER as above, and concept error rate against your predefined token list.

Step 6 — Compare and set a threshold. Two numbers matter: the absolute error rate for your worst-performing group, and the gap between the best and worst. A useful working rule is that a concept error rate above roughly 5% in any group means quotes from that group must be human-verified before they are used, and a between-group gap of more than about 1.5× warrants a design change rather than a caveat.

Record what you find in the study documentation. Six months later, when someone asks why the enterprise segment sounded less articulate in your report, you will want that number.

What to do about it

Ranked roughly by effect:

1. Offer text as an equal option, not a downgrade. This is the single largest lever, and the framing matters. If text is presented as the accessibility fallback, participants who would benefit most often do not take it. If both modes are offered as equivalent choices at the start, self-selection does the work for you. See voice vs text interviews for when each mode suits the research question.

2. Put your numbers in structured questions. Transcription error cannot corrupt a metric that was never spoken as prose. This is the most under-used mitigation in voice research: let the conversation carry the reasoning and let structured questions carry the measurement. A scale response recorded as a discrete value is immune to acoustic modelling entirely.

3. Avoid open-set proper nouns in voice. "Which competitor did you evaluate?" as an open spoken question is the worst case for ASR — unfamiliar proper nouns, no context to constrain them. As a single_choice or multiple_choice question with an "other" option, it is error-free for the closed set and only the residue needs transcription.

4. Ask the interviewer to confirm entities back. A moderator who repeats an unclear product name for confirmation creates a second, cleaner instance of the token in the transcript. AI moderation can do the same thing.

5. Keep spoken option lists short. Auditory presentation has its own limits regardless of transcription, and long lists read aloud produce recency effects. Four items is a sensible ceiling — see survey randomization for the order-effect side of this.

6. Never auto-reject low-confidence responses. If your workflow silently drops conversations that transcribed poorly, you have built a filter that removes exactly the speakers you were worried about. Route them for human review instead.

7. Verify every quote before publication. Listen to the audio for any quote that will appear in a report or a slide. This is cheap, it is fast, and it is the difference between misquoting a customer and not.

The part that is not a data-quality problem

There is a second consequence that dashboards do not show. If voice interviews are harder for non-native speakers, and non-native speakers therefore abandon more often or answer more briefly, then your sample composition shifts — and it shifts along a dimension that often correlates with geography, seniority and market segment.

That is a coverage problem, not a transcription problem, and it needs a coverage response: check completion rates by speaker group, not just error rates, and weight or re-recruit if the gap is material. Survey weighting can adjust for imbalance you measured; it cannot repair imbalance you never looked for.

It is also, for many organisations, a compliance question. If your research programme is part of how you evidence inclusive design, an instrument that works less well for some groups undermines the evidence — see accessibility compliance research for what WCAG, the ADA and the European Accessibility Act actually require you to test, and accessibility research for including participants with disabilities, where speech differences may be part of the disability itself.

How Koji is built for this

Koji runs voice and text interviews in the same study, with the same questions, the same AI moderator behaviour and the same analysis pipeline. That is the structural version of recommendation 1: text is not a lesser path through your research, it is the same research conducted differently, so a participant who chooses it is fully represented in your results rather than shunted into a separate dataset.

Several other things follow from how the platform works:

• AI follow-up questions perform repair in the moment. When an answer is ambiguous, the AI asks about it — the same conversational move a skilled human moderator makes when they did not quite catch something. Traditional survey tools have no repair mechanism at all; a garbled answer is simply a garbled answer forever.
• Structured questions protect your metrics. All six types — open_ended, scale, single_choice, multiple_choice, ranking and yes_no — are captured as discrete values. Only open_ended depends on transcription. Design your tracked metrics as structured questions and your trend lines are immune to acoustic variation across waves and speaker groups.
• Quality scores surface thin or incoherent conversations rather than hiding them, so a systematically poor transcript is visible as a flagged conversation rather than as quiet noise in your themes. The quality gate means only conversations scoring 3 or above consume credits — you are not paying for sessions that did not work.
• Multi-language studies are configured with a language code per study, so participants who would struggle in English can be interviewed in their own language instead. See multi-language user research. Within-language accent variation is a different problem from cross-language support, and both need attention — but if a meaningful share of your participants are non-native English speakers, running the study in their first language usually beats optimising the English transcription.

None of this makes ASR error disappear. What it does is make the error visible, keep it away from your quantitative measures, and give participants a route through the study that does not depend on the model hearing them well.

Frequently asked questions

Is AI transcription accurate enough for professional research? For most participants and most purposes, yes — recent systems approach human accuracy on clear speech. The caveat is that headline accuracy figures usually come from read-speech benchmarks, while research interviews are spontaneous, domain-heavy and often accented. Audit your own study rather than trusting a vendor benchmark, and always verify quotes against audio before publishing them.

What is a good word error rate for research interviews? There is no universal threshold, because it depends where the errors fall. A more useful target is a concept error rate — errors on product names, competitor names, numbers and negations — below roughly 5% for every speaker group, and a between-group gap under about 1.5×. A 12% WER concentrated in filler words is harmless; a 6% WER concentrated in product names is not.

Does this mean we should not run voice research with international participants? No. Voice research produces longer, more spontaneous answers and is easier for many people to complete. It means you should measure performance by group, offer text as an equal option, run studies in participants first language where you can, and keep your tracked metrics in structured questions where transcription cannot reach them.

Have newer models fixed the accent gap? Absolute error rates have fallen a great deal since the 2020 PNAS study, but the pattern persists in recent work: native accents outperform non-native ones, and spontaneous speech is handled worse than read speech. The 2026 clinical study in npj Digital Medicine found significantly higher error rates for non-native speakers using current Whisper models. Improvement is real; parity is not here.

How do I stop transcription errors affecting my quantitative results? Ask the quantitative questions as structured questions rather than as spoken prose. A scale, single_choice, multiple_choice, ranking or yes_no response is recorded as a discrete value and is unaffected by transcription entirely. Reserve open_ended questions for the reasoning, where an occasional error costs you nuance rather than a number.

Should I discard interviews that transcribed badly? No — that is a filter that removes precisely the speakers whose representation you were trying to protect, and it will bias your sample in a direction you cannot later detect. Route poorly transcribed conversations to human review instead, correct them, and keep them in the dataset.

Related Resources

• Structured Questions in AI Interviews — the six question types, and why your metrics belong in them
• Voice vs Text Interview: When to Use Each Mode — choosing modes, and offering both as equals
• Multi-Language User Research — interviewing participants in their own language
• Accessibility Research: Including Users with Disabilities — where speech difference is part of the population you must reach
• Accessibility Compliance Research — what WCAG, the ADA and the European Accessibility Act require you to test
• Understanding Quality Scores — how thin or incoherent conversations are surfaced
• Survey Weighting: How to Correct a Skewed Sample — repairing coverage gaps you have measured

---

Want voice and text interviews in one study, with metrics that transcription cannot corrupt? Start free with 10 credits and run your first AI-moderated interview.