Research Design Sensitivity: Would Your Study Have Caught It If You Were Wrong? (2026)

Answer first: design sensitivity is the probability that your study would have produced a different result if your belief were wrong. It is not the same thing as coverage - the list of topics your discussion guide touches. A guide can cover every topic on the roadmap and still be incapable of contradicting a single thing you already believe. The fastest way to measure sensitivity is to borrow the software engineering technique of mutation testing: seed a small number of specific falsehoods into your own assumptions, then ask, question by question, which of them your protocol would actually catch. The fraction you catch is your design mutation score, and in most research plans it is far lower than the team expects.

Koji makes this practical because a study that used to cost three weeks of calendar time now runs in an afternoon, so pressure-testing a design before fieldwork stops being a luxury and becomes a checklist step.

The question this article is built around

Software teams have a version of this problem that is older and better studied than ours. A test suite runs, it goes green, and the team concludes the code works. But a green suite proves only that nothing in the suite noticed anything. To find out whether the suite can notice anything, engineers use mutation testing: they deliberately corrupt the program - flip a comparison operator, delete a line, change a constant - and re-run the suite. If the suite still passes, it would not have caught that bug either. The fraction of deliberately introduced faults that a suite catches is its mutation score.

Richard DeMillo, Richard Lipton and Frederick Sayward introduced the idea in IEEE Computer in 1978, and their justification was empirical rather than theoretical. Programs, they observed, are usually nearly right: "Programmers do not create programs at random". Faults are therefore small perturbations of a correct program, and the paper's central claim, which they called the coupling effect, was that "tests of a program that uncover simple errors are also effective in uncovering much more complex errors" - test data sensitive enough to distinguish every simple corruption tends to catch the complicated ones too. Their supporting error survey of 1,258 errors across Fortran, Cobol, PL/1 and Basic programs found the single largest class was errors in assignment or computation, at 27 percent of all errors: small, local, easy to simulate.

Research designs have the same property. Your discussion guide is usually nearly right. The failure mode is rarely that you studied the wrong company; it is that one specific premise underneath the study is false and nothing in the protocol would have revealed it.

Coverage is not detection, and the evidence is unusually clean

The reason to import mutation testing rather than just talking about rigor is that the software world has already run the experiment on the cheaper metric - coverage - and the result is blunt.

Laura Inozemtseva and Reid Holmes, in an ICSE 2014 paper whose title says most of it, generated 31,000 test suites for five Java systems of up to 724,000 lines of source code, measured three kinds of coverage for each suite, and used mutation testing to measure whether those suites actually caught faults. Their conclusion: "there is a low to moderate correlation between coverage and effectiveness when the number of test cases in the suite is controlled for", and therefore coverage "should not be used as a quality target because it is not a good indicator of test suite effectiveness".

Two details from that study transfer directly to research.

First, most of the apparent relationship between coverage and effectiveness is really a relationship between size and effectiveness. Bigger suites cover more and catch more; once you hold size constant, coverage stops carrying much signal. The research analogue: a longer discussion guide touches more topics and surfaces more findings, and it is tempting to read that as a better design. It is mostly just a bigger one.

Second, and more uncomfortable, the effect was strongest in the best-tested systems. For Joda Time, the correlation between coverage and effectiveness ran from 0.80 to 0.85 when suite size was ignored and "dropped to essentially zero when suite size was controlled for" - and the authors note that Joda Time and Closure had the largest and most comprehensive suites of the five, "which might indicate that the correlation becomes weaker as the suite improves". A mature research practice with a standard guide, a standard screener and standard coverage of the customer journey is exactly the practice where topic coverage tells you least about whether you would catch a mistake.

What mutation testing looks like when it is real

Mutation testing has a reputation for being academic. It is not. Goran Petrovic and Marko Ivankovic of Google described a deployment across Google's monolithic repository, which they put at approximately 2 billion lines of code, that evaluated more than 70,000 diffs, tested 1.1 million mutants, and surfaced 150,000 findings during code review for more than 13,000 code authors. Over 87 percent of test runs over mutants failed - that is, the mutant was killed. The interesting 13 percent are the ones that survived: places where a change to the program produced no complaint from anything.

Google also had to solve a problem that research teams will recognise immediately. Mutating every line produced too many uninteresting findings - the authors call them arid lines, code where a change is technically undetected but nobody cares - and the fix was a feedback loop where developers marked findings as not useful. That loop moved "the reported usefulness of the surfaced results" from 20 percent to 80 percent. The lesson for a research plan is the same: do not seed random falsehoods, seed the ones that would change a decision.

The design mutation score: a 45-minute protocol

Here is the full procedure. It needs your draft discussion guide, your screener, and one colleague.

Step 1. Write down the beliefs the study rests on. Not the research questions - the premises. A typical B2B discovery study rests on something like: the buyer is the user; the workaround they use today is a spreadsheet; the trigger to look for a tool is a headcount change; price is a secondary objection.

Step 2. Mutate each belief into a specific, plausible falsehood. Not the negation, which is usually too easy to spot. The good mutants are near misses, in the same spirit as DeMillo's near-correct programs. The buyer is the user mutates to the buyer is the user's manager, who has never opened the product. Their workaround is a spreadsheet mutates to their workaround is a colleague they ask in chat.

Step 3. For each mutant, walk the protocol line by line and ask which question would return a different answer if the mutant were true. Not which question is about this topic - which question would produce different words in the transcript. If the answer is none, the mutant survives.

Step 4. Score it. Mutants killed divided by mutants seeded. Ten seeded, four killed is a design mutation score of 40 percent, which is a normal first result and a bad place to start fieldwork.

Step 5. Fix the protocol, not the score. Every surviving mutant is a question you are missing. Add the question that separates the worlds, then re-score.

The crucial discipline in step 3 is that coverage does not count. We ask about their current workflow is coverage. We ask them to screen-share the last three things they did in the spreadsheet, and a chat-based workaround would be visible in that recording is detection.

Two killed out of four. The two survivors are not vague weaknesses; they are two specific questions that were missing, and you found them in twenty minutes without talking to anyone.

What sensitivity depends on, in numbers

There is a second, quieter arithmetic. Suppose a given participant would say something that contradicts your belief with probability p. The chance that at least one of n interviews surfaces it is 1 minus (1 minus p) to the power n.

This table is the reason sample-size questions feel so urgent, and it is also a trap, because every cell assumes p is greater than zero. The protocol sets p. If no question in your guide gives a participant an opening to contradict the belief, p is zero, the whole table collapses to zero, and no sample size rescues it. Running 30 interviews instead of 10 moves you down a column. Fixing a surviving mutant is what moves you between rows - and it is much cheaper.

That distinction is worth holding against two neighbouring ideas. Sample-size guidance, covered in How Many User Interviews Do You Need?, answers how many; the discovery-curve argument in Why Your Theme Discovery Curve Flattens explains why new themes stop appearing even when coverage is poor. Design sensitivity is a third question that comes before both: whether the instrument can register a contradiction at all.

How Koji helps

Traditional research makes design mutation testing theoretically attractive and practically impossible. If a study takes three weeks to field, nobody re-scores the guide and re-runs it; they ship the guide they have. The economics are what changed.

• Re-running the design is cheap. Koji's AI-moderated interviews run in parallel rather than in sequence, so a protocol revision costs a re-publish rather than a rescheduling exercise. When killing a mutant means adding one question and refielding, teams actually do it.
• Structured questions turn detection into something you can see. Koji supports six question types - open_ended, scale, single_choice, multiple_choice, ranking, and yes_no - and each produces a different shape of evidence. A surviving mutant is often a belief that only ever met an open_ended question; converting it to a ranking or single_choice item forces a discriminating answer instead of a sympathetic one. See Structured Questions in AI Interviews for how the six types map to report output.
• The AI interviewer probes where a human would move on. Koji's methodology frameworks encode the Mom Test rule against hypothetical questions directly into moderation: the framework's anti-patterns explicitly warn never ask whether someone would use X, because people misreport future behaviour. That turns a piece of interviewer discipline into a property of the instrument.
• Automatic thematic analysis scores every interview for depth and relevance on a 1 to 5 scale, which means a protocol that is producing shallow, non-discriminating answers shows up as a quality distribution, not as a vague feeling in the readout meeting.
• Custom AI consultants let you keep a permanent seeded-mutant list for a product area and re-run it against every new guide, which is the research equivalent of a regression suite.

Compared with a legacy survey tool, where the instrument is a static form and the only feedback is the response rate, an AI-native platform lets you interrogate the instrument before it meets a participant.

Common mistakes

Seeding mutants that are too easy. Our customers do not exist is not a mutant. If a falsehood would be obvious in the first ninety seconds of any interview, it tells you nothing. Aim for the near miss.

Counting topics as detection. The most common failure is scoring a mutant as killed because the guide mentions the topic. Ask what words would be different in the transcript.

Confusing sensitivity with sample size. A bigger n multiplies whatever detection probability the protocol already has. Zero times anything is still zero.

Scoring the guide and then not changing it. The score is diagnostic, not decorative. Inozemtseva and Holmes' warning against using coverage as a quality target applies here too: the moment a design mutation score becomes a number people report rather than a number people act on, it will be gamed with easy mutants.

Doing it alone. You cannot easily invent falsehoods about your own premises. Give the belief list to a colleague and let them write the mutants.

Frequently asked questions

What is design sensitivity in user research?

Design sensitivity is the probability that a study would have produced a different result if the belief it was testing were false. It is a property of the protocol - the screener, the questions, the prompts, the analysis plan - rather than of the sample size or the findings. A sensitive design can surprise you; an insensitive one returns the same shape of answer no matter what is true.

How is this different from sample size or saturation?

Sample size and saturation both ask how much data you need. Design sensitivity asks whether any amount of that data could contradict you. If your guide never gives a participant an opening to disagree with a premise, the contradiction rate per participant is zero and no sample size fixes it. Work out sensitivity first, then use the sample-size guidance in How Many User Interviews Do You Need?.

What is a good design mutation score?

There is no published benchmark for research protocols, so treat the number as relative rather than absolute: score the guide, fix the surviving mutants, re-score, and expect a first pass in the 30 to 50 percent range. What matters is the list of survivors, because each one names a missing question. A score reported without the survivor list is the failure mode Inozemtseva and Holmes warned about when they argued coverage should not be used as a quality target.

Does mutation testing really transfer from software to research?

Partly. The mechanic transfers exactly: seed a known fault, check whether the process detects it, count. The assumption underneath it - DeMillo, Lipton and Sayward's claim that complex errors are coupled to simple ones - is an empirical claim about programs, not about research designs, and nobody has tested it on discussion guides. Treat the score as a structured way to find missing questions rather than as a calibrated measure of study quality.

How long does this take on a real study?

About 45 minutes for a discovery guide: fifteen to write the belief list, fifteen for a colleague to write mutants, fifteen to walk the protocol. It is faster than a pilot study and answers a different question - a pilot tells you whether the guide runs smoothly, which is covered in Pilot Study in User Research, while mutation scoring tells you whether a smooth-running guide could ever contradict you.

Can AI-moderated interviews be mutation tested?

Yes, and more easily than human-moderated ones, because the protocol is explicit. With Koji, the interview plan, the probing rules and the structured question types are all written down, so walking a mutant through the protocol is a reading exercise rather than a guess about what a moderator would have done in the room. You can also field the revised guide the same day, which is what makes the loop worth closing.

The bottom line

A research plan that covers every topic and cannot contradict a single premise is a green test suite over untested code. Seed four or five specific, plausible falsehoods into your own beliefs, walk the protocol, and count the ones that survive. The survivors are your missing questions, and you can find them before fieldwork rather than a quarter after the decision.

Related Resources

• Structured Questions in AI Interviews - the six question types and what each one can discriminate
• Analysis of Competing Hypotheses - diagnosticity applied to evidence you have already collected
• How Many User Interviews Do You Need? - sample size once your design can detect something
• Pilot Study in User Research - pre-testing that a guide runs, as distinct from testing that it discriminates
• Why Your Theme Discovery Curve Flattens - why new themes stop appearing before coverage is complete
• The Research Risk Model - deciding which questions deserve this level of scrutiny