{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-09-22T10:59:51.721Z"},"content":[{"type":"documentation","id":"e0127bd0-7f4d-496c-a42f-fdbaa5b7a09f","slug":"poka-yoke-mistake-proofing-research","title":"Poka-Yoke for Research: Mistake-Proofing Studies Before Errors Become Findings (2026)","url":"https://www.koji.so/docs/poka-yoke-mistake-proofing-research","summary":"Applies Shigeo Shingo's poka-yoke (mistake-proofing) to research. Distinguishes mistakes (inevitable) from defects (mistakes that reach stakeholders), control vs warning devices, and Shingo's contact, fixed-value and motion-step methods. Uses the gene-name Excel case: Ziemann 2016 found errors in about one-fifth of papers; the 2021 follow-up found 30.9% despite the warning; HGNC renaming 27 genes in 2020 designed the error out. Includes a research error catalogue and how Koji builds controls in.","content":"**Short answer:** Poka-yoke means designing a process so that a mistake either cannot happen or shows up the moment it does. In research, that means building your study so the common errors - a skipped question, a flipped scale, a mis-typed ID, the wrong participant - are blocked or flagged when they occur, not found weeks later in analysis. The key idea is Shigeo Shingo's distinction between a **mistake** (inevitable, human) and a **defect** (a mistake that reached the customer). You cannot remove mistakes. You can stop them turning into findings.\n\nMost research quality advice is about inspection: review the guide, audit the transcripts, double-check the spreadsheet. Poka-yoke asks a different question: *why was that error possible in the first place?* This guide covers where poka-yoke came from, the two kinds of device (control and warning), the clearest evidence from science of why warnings fail and controls work, and a practical catalogue of mistake-proofing for surveys, interviews and analysis, including what AI-native platforms like Koji now do by default.\n\n## Where poka-yoke comes from\n\nShigeo Shingo, an industrial engineer closely involved with the Toyota Production System, formalised poka-yoke in the 1960s. The original term, *baka-yoke* (\"fool-proofing\"), was softened to *poka-yoke* (\"mistake-proofing\") because the point was never that workers were fools. His book *Zero Quality Control: Source Inspection and the Poka-Yoke System* (1986) set out the approach. As Bekenn and Hooper put it in a paper applying the idea to spreadsheets, *Shingo realised that there was a clear distinction to be made between a mistake and a defect.*\n\nThe classic example involves a switch with two push-buttons, each needing a small spring. Workers sometimes forgot a spring, and the switch left the line broken. Shingo's fix was not training or a warning sign. Workers first placed the two springs in a small placeholder, then assembled the switch. If a spring was still in the placeholder at the end, the mistake was obvious at once, at the workstation, before the switch went anywhere.\n\nThe construction researcher Iris Tommelein sums up the goal: *To eliminate the need for quality control, the practice of mistake proofing sets out to prevent errors or defects from occurring in the first place.*\n\n## Control versus warning: the distinction that matters\n\nPoka-yoke devices come in two strengths.\n\n### Control poka-yoke\n\nA **control** device makes the mistake physically impossible, or stops the process until it is fixed. A plug that only fits one way is a control. In research, a scale question that will only accept values from 1 to 5 is a control: an out-of-range answer cannot be recorded.\n\n### Warning poka-yoke\n\nA **warning** device lets the mistake happen but makes it immediately visible. The spring left in the placeholder is a warning. In research, a banner saying \"12 participants have not answered question 4\" is a warning. Someone still has to notice it and act.\n\nWarnings are cheaper and more flexible, and they are often the right choice. But they depend on attention, and attention is exactly what runs short under deadline. When an error is frequent and costly, prefer a control.\n\n### Shingo's three detection methods\n\nShingo also described three ways a device can detect a mistake. Each maps cleanly onto research work:\n\n| Method | Manufacturing meaning | Research equivalent |\n|---|---|---|\n| Contact | Checks a physical attribute (shape, size, colour) | Checks an answer's *type and range* - a rating must be 1-5, a date must be a date |\n| Fixed-value | Checks that a set number of actions happened | Checks that every required question was asked in every session |\n| Motion-step | Checks that steps happened in the right order | Checks sequence - consent before recording, screener before interview, deduplication before analysis |\n\n## The strongest evidence that warnings fail: gene names in Excel\n\nThe clearest case study of poka-yoke in research comes from genomics, and it is worth knowing in detail because it is a rare natural comparison between a warning and a control aimed at the same error.\n\nMicrosoft Excel, with default settings, converts some gene symbols into dates. *SEPT2* becomes \"2-Sep\", *MARCH1* becomes \"1-Mar\". The data is silently corrupted, and because supplementary spreadsheets are reused by other scientists, the corruption spreads.\n\n**The warning.** In 2016, Mark Ziemann, Yotam Eren and Assam El-Osta published \"Gene name errors are widespread in the scientific literature\" in *Genome Biology* (17:177). They scanned 35,175 supplementary Excel files from 18 journals and found errors in 704 articles: *approximately one-fifth of papers with supplementary Excel gene lists contain erroneous gene name conversions.* Errors had been rising at 15% a year. The paper was widely covered. In poka-yoke terms, it was a warning delivered to an entire field.\n\n**The result of the warning.** In 2021, Abeysooriya, Soria, Kasu and Ziemann checked whether the warning had worked (\"Gene name errors: Lessons not learned\", *PLOS Computational Biology*). It had not: *gene name errors continued to accumulate unabated in the period after 2016.* Their improved scanner found errors in **30.9% (3,436 of 11,117)** of articles with supplementary Excel gene lists.\n\n**The control.** Meanwhile the HUGO Gene Nomenclature Committee (HGNC) did something different. Its 2020 naming guidelines added symbols that *affect data handling and retrieval* as a reason to rename a gene, and 27 human gene symbols were changed. *SEPT1* became *SEPTIN1*; *MARCH1* became *MARCHF1*. The new names cannot be misread as dates. The error was not discouraged. It was designed out of the input.\n\nThat is poka-yoke in one sentence: the field was warned and kept making the mistake; changing the input removed it, for every gene that was renamed. Most research teams have their own version of *SEPT2*, and most still rely on the warning.\n\n## Mistake-proofing research: a practical catalogue\n\nThe table pairs common research errors with a warning-level and a control-level fix. Use the control where the error is frequent or expensive.\n\n| Error | Warning poka-yoke | Control poka-yoke |\n|---|---|---|\n| Key question skipped in some interviews | Checklist on the moderator's screen | Required question the session cannot end without (fixed-value) |\n| Rating recorded as free text (\"pretty good, maybe a 4\") | Coder flags unparseable ratings | Scale question type that only accepts the defined range (contact) |\n| Scale direction flipped between questions (1 = best vs 1 = worst) | Note in the analysis plan | One scale definition, reused, stored with its labels |\n| Wrong-segment participant interviewed | Researcher reviews screener answers | Screener that terminates on disqualifying answers |\n| Session recorded before consent | Reminder in the guide | Recording that cannot start until consent is captured (motion-step) |\n| Participant IDs or postcodes mangled by a spreadsheet (leading zeros dropped, IDs read as dates) | Spot-check after import | Import ID columns as text; use IDs that cannot parse as numbers or dates |\n| Duplicate respondents counted twice | Manual dedupe before analysis | Unique-link or unique-ID enforcement at entry |\n| Theme counted by mentions in one place and by participants in another | Reviewer notices inconsistent numbers | One counting rule defined in the analysis template |\n\nThe spreadsheet rows are not trivial. Bekenn and Hooper describe the default state of spreadsheet quality control in one line: *Defect reduction is usually done by time consuming manual inspection (audit).* Research synthesis still runs through spreadsheets more often than teams admit, and every hand-built pivot table is a place where the equivalent of *SEPT2* can happen.\n\n## Where not to mistake-proof\n\nPoka-yoke is a tool for the *plumbing* of research, not for the conversation. Over-constraining discovery is its own error.\n\n- **Do not force structure onto exploration.** An open-ended question that only accepts pre-set answers is not mistake-proofed; it is broken. Keep open_ended questions open, and mistake-proof the scaffolding around them: required coverage, consent, IDs, scales.\n- **Do not turn every warning into a hard stop.** A screener that terminates on any unusual answer will filter out the edge cases discovery research exists to find. Use controls for clear disqualifiers and warnings for judgment calls.\n- **Do not mistake-proof the wrong step.** Shingo's point was to stop the error *at the source*. A validation rule at the reporting stage is still inspection, just automated.\n\n## A five-step method for your next study\n\n### 1. List the errors that actually happened\n\nGo through your last three studies and write down every error that reached analysis or a stakeholder. This is your defect log. Poka-yoke starts from real mistakes, not imagined ones.\n\n### 2. Trace each one to its source step\n\nFor each defect, find where the *mistake* happened, not where the defect was found. A mis-counted theme found in review usually started in how the analysis template defined counting.\n\n### 3. Choose control or warning\n\nFrequent and costly: control. Rare, or needing judgment: warning. Be honest about whether anyone reliably reads your warnings. The genomics case suggests they often do not.\n\n### 4. Build the device into the tool, not the training\n\nA rule in a wiki is not a poka-yoke. A question type that refuses invalid input is. Where you can, put the constraint in the instrument itself.\n\n### 5. Re-check the defect log next quarter\n\nIf a class of error has not dropped, the device is not working, however sensible it looked.\n\n## How Koji mistake-proofs research by design\n\nTraditional research stacks - a survey tool, a video call, a transcription service and a spreadsheet - have a hand-off at every step, and every hand-off is a place for a mistake to become a defect. Koji puts the whole study in one system, which lets many of the controls above be built in instead of bolted on.\n\n- **Six structured question types as contact controls.** Koji supports open_ended, scale, single_choice, multiple_choice, ranking and yes_no questions inside AI-moderated interviews. A scale answer is captured as a scale value with its labels, and a ranking as a ranking, so the \"a 4, I suppose\" problem cannot reach the dataset. See the [structured questions guide](/docs/structured-questions-guide).\n- **Required coverage as a fixed-value control.** Koji's AI interviewer works through the study's questions in every session, text or voice, so question 4 is not quietly skipped in interview 14 because the conversation ran long.\n- **Consistent probing.** The AI moderator follows up the same way for every participant, removing a mistake source that manual moderation cannot fully control: fatigue after the sixth session of the day.\n- **Quality flags as warnings.** Each interview gets a 1-5 quality score, so thin or off-topic sessions are visible before they are averaged into results.\n- **No spreadsheet step for the core analysis.** Koji's automatic thematic analysis and real-time reporting work directly on the interview data, with every theme linked to its quotes, so there is no manual export, no pivot table and no chance of an ID turning into a date.\n- **Methodology in the brief.** Frameworks such as Mom Test and JTBD shape the questions from the start, catching leading or hypothetical questions at the source step rather than in review.\n\nTeams that previously spent days reconciling exports and re-checking tallies can have structured results within minutes of the last interview. Just as important, whole classes of error no longer need checking because the design has removed them.\n\n## Common mistakes when applying poka-yoke\n\n- **Calling a reminder a poka-yoke.** A note in the discussion guide is a warning at best. If it depends on someone remembering, it is not mistake-proofing.\n- **Mistake-proofing at the end of the process.** Validation in the final report is automated inspection. Move it to the step where the mistake happens.\n- **Constraining the conversation.** Use structure for measurement and coverage, not for the exploratory questions where surprises live.\n- **Assuming a published warning changes behaviour.** The gene-name story shows a field kept making the same error for years after it was documented.\n- **Blaming the person.** Poka-yoke assumes people will make mistakes. If an error keeps happening, the design is the problem.\n- **Keeping the spreadsheet step out of habit.** If Koji or another tool can analyse the data where it was collected, every manual export you keep is a new chance for a defect.\n\n## Frequently asked questions\n\n### What does poka-yoke mean?\n\nPoka-yoke is Japanese for \"mistake-proofing\". It describes any mechanism that prevents a mistake or makes it immediately visible, so it does not become a defect. Shigeo Shingo developed the idea at Toyota in the 1960s and described it in *Zero Quality Control: Source Inspection and the Poka-Yoke System* (1986).\n\n### What is the difference between a mistake and a defect?\n\nA mistake is the human error itself, such as skipping a question or typing the wrong ID. A defect is what happens when that mistake reaches the customer - in research, a stakeholder acting on a wrong number or finding. Poka-yoke accepts that mistakes will happen and aims to stop them becoming defects.\n\n### What are the two types of poka-yoke?\n\nControl poka-yoke makes the mistake impossible or stops the process until it is fixed, such as a question that only accepts valid answers. Warning poka-yoke lets the mistake happen but flags it immediately, such as an alert that some participants skipped a question. Controls are stronger; warnings are cheaper and more flexible.\n\n### How do you apply poka-yoke to surveys and interviews?\n\nUse typed questions that only accept valid answers, make key questions required, enforce consent before recording, terminate screeners on clear disqualifiers, and import identifiers as text so spreadsheets cannot reformat them. Keep open-ended questions unconstrained so exploration is not lost.\n\n### Isn't careful review enough to catch research errors?\n\nReview finds some errors, but it depends on attention and tends to happen late, after the error has spread. The gene-name studies show a documented, widely publicised error still appearing in 30.9% of affected papers years later. Designing the error out of the process is more reliable than hoping every reviewer catches it every time.\n\n### How does Koji help mistake-proof research?\n\nKoji builds controls into the study itself: six structured question types that only accept valid answers, AI-moderated interviews that cover every required question, per-interview quality scores that flag weak sessions, and thematic analysis that runs on the source data without a spreadsheet step. That removes many common errors at the source instead of catching them in review.\n\n## Related Resources\n\n- [Structured Questions in AI Interviews](/docs/structured-questions-guide) - the six question types, and how typed answers act as built-in controls\n- [Research Quality Inspection Sampling](/docs/research-quality-inspection-sampling) - the inspection side of quality, and why it cannot carry the load alone\n- [Pilot Study in User Research](/docs/pilot-study-user-research-guide) - finding the source-step mistakes before the real study starts\n- [Attention Check Questions](/docs/attention-check-questions) - a warning-level device for respondent inattention\n- [Survey Data Quality Guide](/docs/survey-data-quality-guide) - the data errors most worth designing out\n- [The Swiss Cheese Model for Research](/docs/swiss-cheese-model-research-quality) - how mistake-proofed layers fit into the wider stack of defences\n- [Slips vs Mistakes: Classifying User Errors](/docs/slips-vs-mistakes-usability-errors) - why the same wrong click needs opposite fixes depending on intention","category":"Research Methods","lastModified":"2026-09-22T03:27:34.185817+00:00","metaTitle":"Poka-Yoke for Research: Mistake-Proofing Guide (2026)","metaDescription":"Apply poka-yoke to research: control vs warning devices, why the gene-name Excel warning failed and renaming worked, and how to mistake-proof studies.","keywords":["poka-yoke","mistake-proofing","error-proofing","shigeo shingo","survey data quality","research errors","poka yoke examples"],"aiSummary":"Applies Shigeo Shingo's poka-yoke (mistake-proofing) to research. Distinguishes mistakes (inevitable) from defects (mistakes that reach stakeholders), control vs warning devices, and Shingo's contact, fixed-value and motion-step methods. Uses the gene-name Excel case: Ziemann 2016 found errors in about one-fifth of papers; the 2021 follow-up found 30.9% despite the warning; HGNC renaming 27 genes in 2020 designed the error out. Includes a research error catalogue and how Koji builds controls in.","aiPrerequisites":["Experience designing a survey or interview guide","Basic spreadsheet-based analysis"],"aiLearningOutcomes":["Distinguish a mistake from a defect and a control from a warning device","Map Shingo's three detection methods onto research work","Design control-level mistake-proofing for common research errors","Know when not to constrain exploratory research"],"aiDifficulty":"intermediate","aiEstimatedTime":"12 min"}],"pagination":{"total":1,"returned":1,"offset":0}}