{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-08-04T08:54:43.369Z"},"content":[{"type":"documentation","id":"0b1321ac-9d3a-46ad-a9f2-ce289d06814e","slug":"voice-research-accents-transcription-accuracy","title":"Accents, Dialects and AI Transcription: How to Check Your Voice Research Works for Everyone","url":"https://www.koji.so/docs/voice-research-accents-transcription-accuracy","summary":"Automatic speech recognition fails unevenly across speaker groups, so voice research needs a per-group accuracy audit. Koenecke et al (PNAS 2020) found average word error rates of 0.35 for Black speakers versus 0.19 for white speakers across five commercial ASR systems, with the gap persisting on identical phrases - implicating acoustic models rather than vocabulary. Graham and Roll (JASA 2024) found native accents outperform non-native and read speech outperforms spontaneous speech in Whisper; McGuire (2025) found near-human match error rates of 0.054 on read L2 speech. WER is the wrong headline metric for research because errors on product names, competitors, numerals and negations damage analysis far more than dropped filler words - track concept error rate instead. Mitigations ranked: offer text as an equal option, put tracked metrics in structured questions where transcription cannot reach them, avoid open-set proper nouns in voice, confirm entities back, keep spoken lists short, never auto-reject low-confidence responses, verify quotes against audio.","content":"**Automatic speech recognition does not fail at random. It fails more often for some speaker groups than others, and if you run voice research without measuring that gap you will publish findings that are systematically thinner for the participants the model handles worst. Measure word error rate by speaker group before you trust voice-only data, and offer text as an equal-quality alternative rather than a fallback.**\n\nThis is not an argument against voice research. Voice interviews produce richer, longer, more spontaneous answers than typed ones, and for many populations they are dramatically easier to complete. It is an argument for knowing where your instrument is weak, in the same way you would want to know that a survey question performs differently across segments.\n\n## The evidence, and what it actually says\n\nThe landmark study is Koenecke and colleagues in *PNAS* (2020), which tested five commercial ASR systems — from Amazon, Apple, Google, IBM and Microsoft — against structured interviews with 42 white speakers and 73 Black speakers across five US cities, 19.8 hours of audio, matched on the age and gender of the speaker.\n\n**Average word error rate: 0.35 for Black speakers against 0.19 for white speakers.** Every one of the five systems showed the disparity. Error rates were highest for African American men.\n\nThe methodological detail that makes the finding hard to dismiss: the gap was **equally large on a subset of identical phrases** spoken by both groups. That rules out the comfortable explanation that the models simply encountered unfamiliar vocabulary. The disparity traces to the acoustic models — to how the systems hear, not to what they were asked to understand.\n\nModels have improved substantially since 2020, so it is fair to ask whether this is a historical problem. The more recent work says the shape of the problem persists even as absolute error falls:\n\n- **Graham and Roll (2024)**, *JASA Express Letters*, evaluated Whisper across native and non-native English accents. Accuracy was higher for American and Canadian speakers than for British and Australian ones, and native accents outperformed non-native accents overall. The authors linked errors to L1 prosody, vowel inventory, and — importantly — to **speech type**, with worse performance on spontaneous speech than read speech.\n- **McGuire (2025)** assessed five current ASR systems on the L2-ARCTIC corpus, covering speakers with Arabic, Chinese, Hindi, Korean, Spanish and Vietnamese first languages. On **read speech**, Whisper and AssemblyAI achieved mean match error rates of 0.054 and 0.056 — genuinely close to human transcription.\n- Clinical speech research published in *npj Digital Medicine* (2026) found error rates significantly higher for non-native speakers when testing Whisper and WhisperX on clinical conversation.\n\nPut those together and the practical conclusion is specific rather than alarmist: **modern ASR is close to human accuracy on read speech from speakers it was well trained on, and materially worse on spontaneous, accented, domain-heavy conversation — which is exactly what a research interview is.**\n\nBenchmark numbers you see quoted in vendor marketing are almost always measured on read corpora. Your interviews are not read corpora.\n\n## Why word error rate is the wrong headline metric for research\n\nWER counts substitutions, deletions and insertions against a reference transcript:\n\n**WER = (S + D + I) / N**\n\nwhere N is the number of words in the reference. It is a useful engineering metric and a poor research metric, because it weights every word equally. Your analysis does not.\n\nConsider two transcripts of the same 200-word answer:\n\n| | Transcript A | Transcript B |\n|---|---|---|\n| Errors | 20 dropped filler words (\"um\", \"you know\", \"like\") | 10 errors, all on product and feature names |\n| WER | 10% | 5% |\n| Effect on your findings | Essentially none | Severe — the answer is now about the wrong thing |\n\nTranscript B has half the word error rate and is far more damaging. Research analysis depends disproportionately on a small set of high-information tokens: product names, feature names, competitor names, numbers, negations, and clinical or technical terms. A dropped \"not\" inverts a finding. A misheard competitor name lands in the wrong theme.\n\nSo measure two things:\n\n1. **WER** — for comparison across speaker groups, because it is the standard and it is what the literature reports.\n2. **Concept error rate** — the proportion of *domain-critical tokens* transcribed incorrectly. Define the token list before you start: your product names, your competitors, your feature vocabulary, numerals, and negations. This is the metric that predicts whether your analysis will be wrong.\n\nA voice study can be perfectly usable at 15% WER and unusable at 8%, depending entirely on where the errors land.\n\n## How to audit your own study in an afternoon\n\nYou do not need a research programme to do this. You need ten interviews and about three hours.\n\n**Step 1 — Stratify your sample.** Group completed interviews by whatever speaker characteristics plausibly matter for your population: first language, region or country, and any accent groups you can identify from recruitment data. You are not making claims about individuals; you are checking whether your instrument performs evenly across groups you are trying to represent.\n\n**Step 2 — Sample within strata.** Take at least three interviews per group. Fewer than that and you are measuring individual variation.\n\n**Step 3 — Take a fixed window.** Human-correct a five-minute stretch from the middle of each interview — not the start, where speech is more scripted and warm-up questions are simpler.\n\n**Step 4 — Produce a reference transcript.** Correct the machine transcript by ear. Keep normalisation rules consistent: decide up front how you handle contractions, numerals and filler words, and apply the same rules to every group. Inconsistent normalisation is the most common way these audits produce a fake disparity.\n\n**Step 5 — Compute both metrics per group.** WER as above, and concept error rate against your predefined token list.\n\n**Step 6 — Compare and set a threshold.** Two numbers matter: the absolute error rate for your worst-performing group, and the **gap** between the best and worst. A useful working rule is that a concept error rate above roughly 5% in any group means quotes from that group must be human-verified before they are used, and a between-group gap of more than about 1.5× warrants a design change rather than a caveat.\n\nRecord what you find in the study documentation. Six months later, when someone asks why the enterprise segment sounded less articulate in your report, you will want that number.\n\n## What to do about it\n\nRanked roughly by effect:\n\n**1. Offer text as an equal option, not a downgrade.** This is the single largest lever, and the framing matters. If text is presented as the accessibility fallback, participants who would benefit most often do not take it. If both modes are offered as equivalent choices at the start, self-selection does the work for you. See [voice vs text interviews](/docs/voice-vs-text-interviews) for when each mode suits the research question.\n\n**2. Put your numbers in structured questions.** Transcription error cannot corrupt a metric that was never spoken as prose. This is the most under-used mitigation in voice research: let the conversation carry the reasoning and let [structured questions](/docs/structured-questions-guide) carry the measurement. A `scale` response recorded as a discrete value is immune to acoustic modelling entirely.\n\n**3. Avoid open-set proper nouns in voice.** \"Which competitor did you evaluate?\" as an open spoken question is the worst case for ASR — unfamiliar proper nouns, no context to constrain them. As a `single_choice` or `multiple_choice` question with an \"other\" option, it is error-free for the closed set and only the residue needs transcription.\n\n**4. Ask the interviewer to confirm entities back.** A moderator who repeats an unclear product name for confirmation creates a second, cleaner instance of the token in the transcript. AI moderation can do the same thing.\n\n**5. Keep spoken option lists short.** Auditory presentation has its own limits regardless of transcription, and long lists read aloud produce recency effects. Four items is a sensible ceiling — see [survey randomization](/docs/survey-randomization-guide) for the order-effect side of this.\n\n**6. Never auto-reject low-confidence responses.** If your workflow silently drops conversations that transcribed poorly, you have built a filter that removes exactly the speakers you were worried about. Route them for human review instead.\n\n**7. Verify every quote before publication.** Listen to the audio for any quote that will appear in a report or a slide. This is cheap, it is fast, and it is the difference between misquoting a customer and not.\n\n## The part that is not a data-quality problem\n\nThere is a second consequence that dashboards do not show. If voice interviews are harder for non-native speakers, and non-native speakers therefore abandon more often or answer more briefly, then your *sample composition* shifts — and it shifts along a dimension that often correlates with geography, seniority and market segment.\n\nThat is a coverage problem, not a transcription problem, and it needs a coverage response: check completion rates by speaker group, not just error rates, and weight or re-recruit if the gap is material. [Survey weighting](/docs/survey-weighting-guide) can adjust for imbalance you measured; it cannot repair imbalance you never looked for.\n\nIt is also, for many organisations, a compliance question. If your research programme is part of how you evidence inclusive design, an instrument that works less well for some groups undermines the evidence — see [accessibility compliance research](/docs/accessibility-compliance-research) for what WCAG, the ADA and the European Accessibility Act actually require you to test, and [accessibility research](/docs/accessibility-research-guide) for including participants with disabilities, where speech differences may be part of the disability itself.\n\n## How Koji is built for this\n\nKoji runs **voice and text interviews in the same study**, with the same questions, the same AI moderator behaviour and the same analysis pipeline. That is the structural version of recommendation 1: text is not a lesser path through your research, it is the same research conducted differently, so a participant who chooses it is fully represented in your results rather than shunted into a separate dataset.\n\nSeveral other things follow from how the platform works:\n\n- **AI follow-up questions perform repair in the moment.** When an answer is ambiguous, the AI asks about it — the same conversational move a skilled human moderator makes when they did not quite catch something. Traditional survey tools have no repair mechanism at all; a garbled answer is simply a garbled answer forever.\n- **[Structured questions](/docs/structured-questions-guide) protect your metrics.** All six types — `open_ended`, `scale`, `single_choice`, `multiple_choice`, `ranking` and `yes_no` — are captured as discrete values. Only `open_ended` depends on transcription. Design your tracked metrics as structured questions and your trend lines are immune to acoustic variation across waves and speaker groups.\n- **[Quality scores](/docs/understanding-quality-scores)** surface thin or incoherent conversations rather than hiding them, so a systematically poor transcript is visible as a flagged conversation rather than as quiet noise in your themes. The [quality gate](/docs/how-the-quality-gate-works) means only conversations scoring 3 or above consume credits — you are not paying for sessions that did not work.\n- **Multi-language studies** are configured with a language code per study, so participants who would struggle in English can be interviewed in their own language instead. See [multi-language user research](/docs/multilingual-research-guide). Within-language accent variation is a different problem from cross-language support, and both need attention — but if a meaningful share of your participants are non-native English speakers, running the study in their first language usually beats optimising the English transcription.\n\nNone of this makes ASR error disappear. What it does is make the error visible, keep it away from your quantitative measures, and give participants a route through the study that does not depend on the model hearing them well.\n\n## Frequently asked questions\n\n**Is AI transcription accurate enough for professional research?**\nFor most participants and most purposes, yes — recent systems approach human accuracy on clear speech. The caveat is that headline accuracy figures usually come from read-speech benchmarks, while research interviews are spontaneous, domain-heavy and often accented. Audit your own study rather than trusting a vendor benchmark, and always verify quotes against audio before publishing them.\n\n**What is a good word error rate for research interviews?**\nThere is no universal threshold, because it depends where the errors fall. A more useful target is a concept error rate — errors on product names, competitor names, numbers and negations — below roughly 5% for every speaker group, and a between-group gap under about 1.5×. A 12% WER concentrated in filler words is harmless; a 6% WER concentrated in product names is not.\n\n**Does this mean we should not run voice research with international participants?**\nNo. Voice research produces longer, more spontaneous answers and is easier for many people to complete. It means you should measure performance by group, offer text as an equal option, run studies in participants first language where you can, and keep your tracked metrics in structured questions where transcription cannot reach them.\n\n**Have newer models fixed the accent gap?**\nAbsolute error rates have fallen a great deal since the 2020 PNAS study, but the pattern persists in recent work: native accents outperform non-native ones, and spontaneous speech is handled worse than read speech. The 2026 clinical study in npj Digital Medicine found significantly higher error rates for non-native speakers using current Whisper models. Improvement is real; parity is not here.\n\n**How do I stop transcription errors affecting my quantitative results?**\nAsk the quantitative questions as structured questions rather than as spoken prose. A scale, single_choice, multiple_choice, ranking or yes_no response is recorded as a discrete value and is unaffected by transcription entirely. Reserve open_ended questions for the reasoning, where an occasional error costs you nuance rather than a number.\n\n**Should I discard interviews that transcribed badly?**\nNo — that is a filter that removes precisely the speakers whose representation you were trying to protect, and it will bias your sample in a direction you cannot later detect. Route poorly transcribed conversations to human review instead, correct them, and keep them in the dataset.\n\n## Related Resources\n\n- [Structured Questions in AI Interviews](/docs/structured-questions-guide) — the six question types, and why your metrics belong in them\n- [Voice vs Text Interview: When to Use Each Mode](/docs/voice-vs-text-interviews) — choosing modes, and offering both as equals\n- [Multi-Language User Research](/docs/multilingual-research-guide) — interviewing participants in their own language\n- [Accessibility Research: Including Users with Disabilities](/docs/accessibility-research-guide) — where speech difference is part of the population you must reach\n- [Accessibility Compliance Research](/docs/accessibility-compliance-research) — what WCAG, the ADA and the European Accessibility Act require you to test\n- [Understanding Quality Scores](/docs/understanding-quality-scores) — how thin or incoherent conversations are surfaced\n- [Survey Weighting: How to Correct a Skewed Sample](/docs/survey-weighting-guide) — repairing coverage gaps you have measured\n\n---\n\n*Want voice and text interviews in one study, with metrics that transcription cannot corrupt? [Start free with 10 credits](https://www.koji.so) and run your first AI-moderated interview.*","category":"Research Methods","lastModified":"2026-08-04T03:22:42.027511+00:00","metaTitle":"Accents, Dialects and AI Transcription Accuracy in Voice Research (2026)","metaDescription":"ASR error is not evenly distributed. Measure word error rate and concept error rate by speaker group, offer text as an equal option, and keep tracked metrics in structured questions.","keywords":["speech recognition accuracy accents","transcription accuracy voice research","ASR bias word error rate","non-native speaker interviews","dialect transcription errors","voice research inclusion","whisper accuracy accents"],"aiSummary":"Automatic speech recognition fails unevenly across speaker groups, so voice research needs a per-group accuracy audit. Koenecke et al (PNAS 2020) found average word error rates of 0.35 for Black speakers versus 0.19 for white speakers across five commercial ASR systems, with the gap persisting on identical phrases - implicating acoustic models rather than vocabulary. Graham and Roll (JASA 2024) found native accents outperform non-native and read speech outperforms spontaneous speech in Whisper; McGuire (2025) found near-human match error rates of 0.054 on read L2 speech. WER is the wrong headline metric for research because errors on product names, competitors, numerals and negations damage analysis far more than dropped filler words - track concept error rate instead. Mitigations ranked: offer text as an equal option, put tracked metrics in structured questions where transcription cannot reach them, avoid open-set proper nouns in voice, confirm entities back, keep spoken lists short, never auto-reject low-confidence responses, verify quotes against audio.","aiPrerequisites":["Familiarity with running voice or moderated interviews","Basic understanding of transcription and sampling"],"aiLearningOutcomes":["Audit word error rate and concept error rate by speaker group in an afternoon","Explain why WER understates the research impact of transcription error","Design voice studies whose quantitative measures are immune to transcription error","Detect coverage bias caused by mode difficulty rather than transcription alone","Set publication rules for verifying quotes against audio"],"aiDifficulty":"intermediate","aiEstimatedTime":"13 min"}],"pagination":{"total":1,"returned":1,"offset":0}}