The Research Risk Model: How to Decide Which Questions Deserve a Rigorous Study

Answer first: the rigor a research question deserves is not set by how important the question is. It is set by what is left over once you account for two things you already have - how likely the belief is to be wrong on its own, and how likely the rest of your system is to catch the error without you. Auditing formalized this arithmetic decades ago, and it is the most useful thing a research team with more questions than capacity can borrow.

The short answer

Every research question carries one risk that actually matters: the risk that your team acts on a belief that turns out to be wrong. Auditing decomposes its equivalent risk into three parts and treats exactly one of them as something you buy.

You cannot lower the first two by studying harder. You set the third, and you set it to whatever brings the total down to a level you can live with. Rigor is a residual, not an input.

The consequence is counterintuitive, and it is the entire point of this article: a critically important question can rationally get a light study, and a question nobody is excited about can rationally get a heavy one.

The model auditing actually uses

The professional standards state it plainly. PCAOB AS 1101 defines audit risk as "the risk that the auditor expresses an inappropriate audit opinion when the financial statements are materially misstated" and specifies that "Audit risk is a function of the risk of material misstatement and detection risk."

The components are defined with unusual precision:

• Inherent risk (AS 1101.07) is the "susceptibility of an assertion to a misstatement, due to error or fraud, that could be material ... before consideration of any related controls."
• Control risk (AS 1101.07) is "the risk that a misstatement due to error or fraud ... will not be prevented or detected on a timely basis by the company's internal control."
• Detection risk (AS 1101.09) is "the risk that the procedures performed by the auditor will not detect a misstatement that exists and that could be material."

Then comes the sentence that does all the work. AS 1101.10: "The higher the risk of material misstatement, the lower the level of detection risk needs to be in order to reduce audit risk to an appropriately low level." And AS 1101.11 confirms who holds the lever: "The auditor reduces the level of detection risk through the nature, timing, and extent of the substantive procedures performed."

Read that as a research instruction and it says: the study you run is the last term in an equation whose first two terms were fixed before you were invited to the meeting.

Translating the three risks into product research

Inherent risk: how likely is this belief to be wrong on its own

Inherent risk is a property of the claim, assessed with the controls switched off. Some beliefs are simply more fragile than others. The reliable amplifiers:

• Novelty. No prior version of this exists in your product or anyone else's, so there is no base rate to lean on.
• Long causal chain. The belief connects a design decision to a business outcome through four intermediate steps, each of which could break.
• Stated futures. The claim depends on what people say they will do rather than on what they have already done.
• Motivated belief. Somebody senior needs this to be true. Incentive does not make a claim false, but it reliably suppresses the search for evidence that it is.
• Small-number extrapolation. The belief comes from a handful of loud accounts and is being applied to a whole segment.

Control risk: what else in your system would catch the error

This is the term research teams never assess, and it is the one that should change your plan most. The question is not "is this important" but "if we get this wrong, what in our system finds out, how fast, and at what cost?"

High control risk, meaning nothing downstream would catch it:

• The decision is a contract, a price change published to existing customers, a public commitment, or a data migration.
• The feature ships to everyone at once with no flag and no staged rollout.
• No instrumented metric would move in a readable way if the belief were false.
• The affected segment is small enough that aggregate dashboards will never show it.

Low control risk, meaning the system is already watching:

• Staged rollout behind a flag with a real holdout.
• A weekly metric that would visibly bend within a month.
• A sales or support channel that complains loudly and quickly.
• The change is cheap to reverse and nobody has built on top of it yet.

Detection risk: the only term your study sets

Detection risk is what you buy with sample size, method, question design, segment coverage and analytic care. It is the only place your research budget goes. Everything above determines how much of it you need to buy.

The allocation table

Work the three columns and the study shape falls out.

Two rows in that table are the argument. The onboarding question feels important and gets the lightest treatment, because the system will tell you within a week. The renewal question and the clinician question get the heaviest treatment, not because they are the most senior asks, but because a wrong answer would sit undetected for months.

Why this contradicts "test the riskiest assumption first"

The Riskiest Assumption Test is good advice and it is not this advice. It ranks assumptions by how fatal they are to the idea and tells you to test the most fatal one before you build. See the Riskiest Assumption Test guide and assumption testing for that ranking, which this article does not replace.

The risk model asks a question those frameworks do not ask: what else would catch this? They score the assumption in isolation. The risk model scores the assumption against the system it lives in. The two give different answers whenever your product already has good detection, which for any team with feature flags, staged rollouts and instrumented funnels is most of the time.

Put concretely: an assumption can be genuinely fatal and still deserve a small study, because a fatal error that surfaces in nine days at the cost of one rollback is not the same object as a fatal error that surfaces in nine months at the cost of a segment. A pure fatality ranking treats them identically. The risk model does not.

This also cuts the other way, and this is the direction teams get wrong more often. A minor-sounding question - do our SMB customers understand the usage cap? - can carry very high control risk, because confused SMB customers do not file tickets, they quietly stop expanding. Nothing in the system would ever tell you. That question deserves more rigor than its importance score suggests.

The evidence rule that comes with the model

AS 2301.09a states the requirement directly: the auditor must "Obtain more persuasive audit evidence the higher the auditor's assessment of risk." AS 2301.37 adds: "As the assessed risk of material misstatement increases, the evidence from substantive procedures that the auditor should obtain also increases."

Persuasiveness is not only sample size. It is method, directness and independence. Raising persuasiveness on a high-risk question can mean any of: talking to people who have actually done the thing rather than people who say they would, covering the segment that would disconfirm rather than the one that would confirm, or adding a second source that could genuinely have disagreed. That last move has its own rules, covered in evidence corroboration and source independence.

The one-way-door adjustment

There is one modifier that overrides the arithmetic. If the decision is irreversible, control risk is high by definition, because reversal is the mechanism by which most product errors get corrected cheaply. A pricing page you can revert, a flag you can flip and a copy change you can redeploy are all two-way doors, and a two-way door is itself a control.

The clean version of the rule: assess control risk by asking what the correction costs, not whether a correction is possible. Almost everything is technically reversible. Very little is cheaply reversible once customers have adapted to it.

What changes when a study costs hours instead of weeks

The risk model was built for a profession where evidence is expensive, and it is usually taught as a rationing device: you have 400 hours, spend them where risk is highest. That framing has a hidden failure mode. When every study is expensive, the low-inherent-risk, low-control-risk questions never get studied at all, and teams lose the base rates that would let them assess inherent risk on the next question.

When the marginal cost of a study collapses, the model stops being a rationing device and becomes a design device. You are no longer choosing which questions to answer. You are choosing how much assurance each answer carries.

This is what changes with AI-moderated research. A Koji study runs conversational interviews with adaptive follow-up questions in voice or text, without a moderator on a calendar, and returns analysis as responses land - so the fifteen-minute directional read on the icon question and the sixty-participant study on the pricing question become the same kind of object, ordered differently. Traditional survey platforms force the opposite trade: because a survey is cheap and an interview is expensive, teams downgrade high-risk questions into multiple-choice instruments and call the result rigor.

Koji's structured questions are the mechanism that lets one instrument carry both registers. A study can mix open_ended questions, where the AI probes reasoning and returns coded themes with supporting quotes, with closed types that aggregate deterministically: scale for magnitude, single_choice and multiple_choice for selection, ranking for relative preference, and yes_no for a clean binary. On a low-risk question the closed types may be all you need. On a high-risk question the open_ended follow-ups are where the disconfirming evidence lives, and the closed types give you a stable denominator to read the qualitative material against. See the structured questions guide for how the six types behave in reports.

Common mistakes

Scoring importance and calling it risk. Importance belongs to the decision. Risk belongs to the belief. A high-importance decision resting on a well-evidenced belief needs less new research than a medium-importance decision resting on a guess.

Assessing control risk optimistically. Teams credit themselves with detection they do not have. The test is historical: name the last three times your system caught a wrong product belief before a customer did. If you cannot, your control risk is high.

Letting the model justify skipping research entirely. Low detection-risk requirements mean a smaller study, not no study. A study of zero has a detection risk of one.

Re-running the assessment per project instead of per question. A single project usually contains one question that needs a powered design and four that need a conversation. Assess the questions.

Confusing this with an operating-maturity score. This is not a ResearchOps maturity model and it does not grade your team. It grades one question at a time, and a mature team will still rate some questions high-risk.

Frequently asked questions

What is the research risk model in one sentence?

It is the audit risk model applied to product research: the total risk of acting on a wrong belief is a function of inherent risk (how fragile the belief is), control risk (whether your existing system would catch the error), and detection risk (how likely your study is to miss it) - and only the third is something your study sets.

How is this different from the Riskiest Assumption Test?

The Riskiest Assumption Test ranks assumptions by how fatal they are and tells you which one to test first. The risk model tells you how much assurance that test needs, and its answer depends on what else in your system would catch the error. A fatal assumption inside a well-instrumented, easily reversed rollout can rationally get a small study.

Does low control risk mean I can skip research?

No. It means you can accept higher detection risk, which translates into a smaller or faster study, not the absence of one. Skipping research sets detection risk to certainty, which puts total risk back at inherent risk regardless of how good your controls are.

How do I assess inherent risk without data?

Use the amplifiers rather than a number. A belief is high inherent risk when it is novel, depends on stated future behavior, travels through a long causal chain, is extrapolated from a few accounts, or is one that a senior stakeholder needs to be true. Two or more amplifiers is a strong signal.

Is this the same as deciding how many interviews to run?

No, and keeping them separate matters. This model tells you how much assurance a question needs. Converting that into a participant count is a separate question covered in how many interviews are enough and survey sample size.

Where should the risk assessment be recorded?

In the brief, before fielding, alongside the assurance level the study is committing to. An assessment written after the results are in is a rationalization, not an allocation, and it cannot constrain what the report is allowed to claim.

Related Resources

• The Riskiest Assumption Test (RAT) - which assumption to test first, the ranking this model sits on top of
• Assurance levels for research - declaring how sure a study is entitled to be, before you field it
• Process controls vs output checks - how control strength changes the work a study must do
• Evidence corroboration and source independence - why a second source only helps if it could have disagreed
• The structured questions guide - the six question types and what each one is good for
• Assumption testing - mapping and validating product assumptions
• The product pre-mortem - surfacing the failure modes worth assessing