Falsifiable Research Questions: Can Your Study Produce the Answer You Do Not Want? (2026)

Answer first: a research question is falsifiable in the only sense that matters when you can name, in advance, the specific answer that would kill the belief - and when the protocol you are about to run is capable of producing that answer. Most product research questions fail the second half. They are open, warm and broad, which maximises the amount you learn if you are right and minimises the chance of discovering that you are wrong. The fix is counter-intuitive: for a named belief, fewer and narrower questions separate the two worlds far better than more and broader ones, and the difference is not marginal. A question that splits 60 percent against 5 percent settles the matter with about fourteen participants per group. A question that splits 80 percent against 75 percent needs roughly 1,100.

Koji makes the narrow version cheap to run, which is why teams using AI-moderated interviews can afford to ask the question that could embarrass them.

The question John Platt told everyone to ask

In 1964, the biophysicist John Platt published an essay in Science called "Strong Inference" arguing that the fastest-moving fields were not the ones with the best instruments but the ones with the best habit of mind. His method had three steps, applied "formally and explicitly and regularly": "Devising alternative hypotheses"; devising a crucial experiment "with alternative possible outcomes, each of which will, as nearly as possible, exclude one or more of the hypotheses"; and "Carrying out the experiment so as to get a clean result".

The part worth stealing is smaller than the method. Platt proposed a single test he called The Question, to be asked silently of any claim: "But sir, what experiment could disprove your hypothesis?" - and, of any study: "But sir, what hypothesis does your experiment disprove?"

His diagnosis of the theories that fail this test is the sharpest sentence in the essay. Of an explanation that could accommodate any observation, he wrote: "It predicts everything, and therefore does not predict anything". And the conclusion: "a theory is not a theory unless it can be disproved".

Run Platt's Question over your last research readout. For each finding, ask what participants would have had to say for you to have written the opposite. If the answer is that you cannot construct that transcript, the study did not test the belief; it illustrated it.

The sign inversion: richer is not more informative

Here is the part that contradicts most interview advice you have been given, and the contradiction is specific rather than general.

The standard prescription - stay open, ask non-leading questions, follow the thread, do not lead the witness - is correct for discovery, where you do not yet know what you are looking for. It is the wrong instrument for adjudication, where a named belief is load-bearing on a decision. For adjudication, breadth actively hurts, because the informativeness of an answer depends entirely on how differently it would come out in the two worlds.

Put numbers on it. A question is worth asking about a belief only to the extent that the answer rate differs depending on whether the belief is true or false.

The first two questions are not bad because they are leading. They are bad because both worlds predict almost the same answer, so the answer carries almost no information. In likelihood terms, an 80-versus-75 split is a likelihood ratio of 1.07: starting from an even-odds belief, one such answer moves you from 50 percent confidence to 51.6 percent. The 60-versus-5 question is a likelihood ratio of 12, which moves the same belief from 50 percent to 92 percent on a single observation.

A note on those sample sizes, because the arithmetic matters. The three figures come from a standard two-proportion power calculation at 80 percent power and a 5 percent two-sided significance level. The normal approximation used for large samples is unreliable at the small end, so the 60-versus-5 row was recomputed with an exact test: Fisher's exact test reaches 78 percent power at twelve per group and 88 percent at fourteen, so fourteen is the honest number and eleven, which the approximation gives, is optimistic. Correcting in the conservative direction still leaves a ratio of about 78 to 1 between the loose question and the sharp one. The conclusion does not depend on the approximation.

That ratio is the inversion in one number. Teams respond to an inconclusive study by adding participants. Adding participants is the expensive lever. Rewriting the question so the two worlds predict different answers is the cheap one, and it is worth roughly two orders of magnitude.

Coverage without a check is worthless, and software measured it

The same lesson has a quantitative literature in software testing, where a test that exercises code but checks nothing is a known pathology.

Yucheng Zhang and Ali Mesbah, in an ESEC/FSE 2015 study, composed 6,700 test suites from 5,892 test cases and 24,701 assertions across five real-world Java projects to isolate what actually makes a suite catch faults. Their framing sentence applies word for word to a discussion guide: "coverage alone is not the goal of software testing, since coverage without checking for correctness is meaningless".

Their findings: assertion quantity and assertion coverage are strongly correlated with a suite's fault-detection effectiveness, assertion quantity drives much of the apparent relationship between suite size and effectiveness, and "the correlation between statement coverage and effectiveness decreases dramatically when assertion coverage is controlled for".

Translate the vocabulary. Touching a topic is coverage. An assertion is a place where a specific expected answer is compared against the actual one. An interview that ranges widely and never compares anything to an expectation is a test that runs a lot of code and asserts nothing. It will produce a transcript, themes, and quotes - and it cannot fail.

Writing a question that can fail: the four-line pre-commitment

Before fieldwork, write four lines per load-bearing belief. This fits on a single page for a whole study and takes about twenty minutes.

1. The belief, stated so that it could be false. Not we want to understand onboarding friction. Rather: new admins abandon setup because SSO configuration requires their IT team, not because the flow is long.
2. The observation that would kill it. Six or more of ten admins who abandoned describe a blocker with no IT dependency, unprompted, before we name SSO.
3. The question that can produce that observation. Take me to where you stopped. What were you waiting on at that moment? Note that the question does not mention SSO. A question that names your hypothesis cannot produce evidence against it.
4. What we do if it dies. We drop the SSO self-serve project from the quarter and re-scope to flow length.

Line four is the one teams skip, and it is the one that makes the other three real. A belief whose falsification changes no decision does not need a study; it needs a note. That triage is what The Research Risk Model is for.

Note also what line two does: it commits to a threshold before you see the data. Deciding afterwards what would have counted as disconfirmation is one of the researcher degrees of freedom catalogued in P-Hacking and Researcher Degrees of Freedom, and it converts any study into a confirmation.

Where this sits next to the frameworks you already use

This is not a new methodology, and it is not a rival to the ones in the corpus. It is a property that a good instance of any of them has.

• The Mom Test is severity applied to the wording of a single question: asking about the past rather than the hypothetical future is exactly the move from an 80-versus-75 split to a 60-versus-5 one. See The Mom Test.
• Competing hypotheses is severity applied after the fact, to evidence already in hand - its notion of diagnosticity is the same quantity this article applies to a protocol before fieldwork. See Analysis of Competing Hypotheses.
• Design sensitivity is severity measured over a whole protocol rather than one question, using seeded falsehoods. See Research Design Sensitivity.
• Writing the hypothesis itself - the if-then-because structure, null and alternative - is covered in How to Write a Research Hypothesis. This article is about the step after: whether the protocol can return the alternative.

How Koji helps

The reason most teams do not run severe tests is not ignorance. It is that a narrow question feels like a wasted slot when a study costs three weeks. AI-native research changes that arithmetic.

• Koji's methodology frameworks encode severity as a rule, not a reminder. The built-in Mom Test framework carries explicit anti-patterns - among them, never asking whether someone would use a product, on the grounds that people misreport future behaviour, and never accepting a compliment as validation. The AI moderator applies them in every session, including the sessions run at 11pm by a founder who has never taken a research methods course.
• Structured questions are where assertions live. Koji's six question types - open_ended, scale, single_choice, multiple_choice, ranking, and yes_no - differ in how much they can discriminate. A ranking question forces a participant to place your feature against real alternatives; a yes_no question with a pre-registered threshold is an assertion in the software sense. An open_ended question is where you discover what you did not know to ask. A severe protocol uses both deliberately: see Structured Questions in AI Interviews.
• Parallel fieldwork makes the narrow question affordable. Because Koji runs AI-moderated interviews concurrently rather than one calendar slot at a time, reaching fourteen qualified participants per group is an afternoon, not a sprint. The 78-to-1 sample-size advantage of a sharp question is only useful if you can field it quickly.
• Real-time reporting shows the split as it forms. Because analysis is automatic, you see the 60-versus-5 separation - or its absence - while the study is still open, rather than after synthesis.
• Customisable AI consultants can hold the pre-commitment. Store the four lines with the study, and the readout can be generated against the threshold you set rather than the impression you formed.

Legacy survey platforms are built to maximise completed responses to a fixed form. They have no concept of a pre-committed kill criterion, which is why a SurveyMonkey questionnaire can return a 4.2 out of 5 average on a concept nobody will ever buy.

Common mistakes

Naming your hypothesis inside the question. Was SSO the reason you stopped? can only produce agreement or a polite dodge. Ask what they were waiting on.

Treating any disconfirmation as fatal. Severity is about the protocol's capacity to disconfirm, not about a rule that one contrary participant sinks a belief. Set the threshold in advance and hold to it in both directions.

Applying it to exploratory work. Early discovery has no named belief to adjudicate. Forcing a pre-commitment there just imports your assumptions sooner. Use severity when a belief is load-bearing on a decision.

Confusing severity with negativity. A severe test is not a hostile one. It is a test with two live outcomes, and it is just as capable of confirming strongly as of killing.

Pre-committing and then not looking. Write the threshold down where the readout will see it. A commitment nobody reads is the same as no commitment.

Frequently asked questions

What makes a research question falsifiable?

It is falsifiable when you can state in advance the specific observation that would make you abandon the belief, and the protocol you plan to run can actually produce that observation. Both halves are required. Plenty of research questions are falsifiable in principle and untestable by the study attached to them, which is Platt's point when he asks what experiment could disprove the hypothesis.

Is this the same as a null hypothesis?

Related but not identical. A null hypothesis is a formal statement used to structure a statistical test, covered in How to Write a Research Hypothesis. Severity is about the discriminating power of the design: two studies can share a null hypothesis while one of them is incapable of rejecting it. The question here is not how the hypothesis is written but whether the instrument can return the answer you do not want.

Does this mean I should stop asking open-ended questions?

No. Open-ended questions are how you discover the thing you did not know to ask, and Koji's open_ended type exists for exactly that. The claim is narrower: once a specific belief is carrying a decision, an open question is a weak instrument for adjudicating it, because both worlds produce a rich and sympathetic transcript.

How many participants does a severe test need?

Far fewer than a loose one. In the worked example above, a question whose answer splits 60 percent against 5 percent between the two worlds gives a clean read at about fourteen participants per group by an exact test, while a question splitting 80 against 75 needs roughly 1,100. The sample size is a property of the question, not of the method, which is why sample-size guidance such as How Many User Interviews Do You Need? has to be read alongside the wording you plan to use.

What if the belief survives the test?

Then you have learned something worth more than a confirmation, because the belief survived a test it could have failed. Record what the test was and what the threshold was, so that the next team to ask the question can see the evidence rather than the conclusion. Scoring those calls over time is what Calibration Scoring for Research Teams measures.

How do I keep a stakeholder from rewriting the threshold afterwards?

Put the four lines in the study brief before fieldwork, share it, and generate the readout against it. The threshold is much harder to move when it is a visible artefact with a date on it than when it lives in a researcher's memory. This is the same reason auditors document a control before testing it, discussed in Test the Process or Test the Output.

The bottom line

Ask Platt's Question of your own plan: what answer, from these participants, to these questions, would make me abandon this? If you cannot construct that transcript, add the question that could produce it. It is about eighty times cheaper than adding participants.

Related Resources

• Structured Questions in AI Interviews - which of the six question types can discriminate, and when
• Research Design Sensitivity - severity measured across a whole protocol with seeded falsehoods
• Negative Controls in User Research - what your process returns when the answer is known to be nothing
• Analysis of Competing Hypotheses - diagnosticity applied to evidence already collected
• The Mom Test - severity at the level of a single question's wording
• P-Hacking and Researcher Degrees of Freedom - what happens when the threshold is set afterwards