Evidence or Ammunition: What a Study Is For When No Result Could Change the Decision

Answer first: when a study is commissioned after the decision is settled, it is not a wasted study - it is a study with a different job. Research has at least three relationships to a decision: it can make the decision, inform it, or support one already taken. Only the first two are evidence. The third is legitimation, and it is so common that management researchers gave it a name: decision-based evidence making. The practical skill is not refusing to do it. It is knowing which one you are being asked for, before fieldwork starts, and negotiating the terms accordingly.

The inversion: a zero-value study is a fact about the organisation

Its companion piece in this library, the value of information test, ends with a clean rule: if no realistic result would change the action, the information is worth nothing and you should not run the study.

That rule is correct and incomplete, because teams run those studies anyway - constantly, deliberately, and often at the explicit request of senior people who are not stupid and not acting in bad faith. A method that says "this should not happen" and then watches it happen every quarter is missing something.

Here is the inversion. In the value-of-information frame, a study that cannot change the decision is waste to be cancelled. Read the other way round, the same study is information about the decision system: somebody wants a research artefact for a decision that is already made, and why is a question with a real, discoverable answer. The zero-value study is not the problem. It is the symptom, and it is a remarkably reliable one - it tells you where the real decision was taken, by whom, and on what basis.

The same test, read in the opposite direction. Cancel the study, and you also throw away the diagnosis.

Three relationships between evidence and a decision

The most useful framing for practitioners comes from Peter Tingling and Michael Brydon, writing in MIT Sloan Management Review in 2010. Their central observation is uncomfortable and worth quoting: many managers think they have committed their organisations to evidence-based decision making, but have instead, without realising it, committed to decision-based evidence making.

They separate three uses of evidence, and the distinction is the whole article:

None of these is illegitimate on its face. A leader who has made a defensible call on judgement and needs to bring an organisation with them has a real problem, and evidence genuinely helps solve it. The failure is not that the third category exists. The failure is that it is almost never named, so a researcher plans a study for category one and delivers into category three, where the honest and inconvenient parts of the finding will be dropped.

The academic backbone: research does more than one job

The underlying literature is older and broader than product research, and it is worth knowing because it stops you treating this as a pathology of your particular company.

In 1979, Carol Weiss published "The Many Meanings of Research Utilization" in Public Administration Review (39:426-431), setting out seven distinct models of how research gets used in policy. Three of them describe most of what happens in product organisations:

• The problem-solving model. Research is commissioned to answer a defined question and the answer feeds the decision. This is the model everyone assumes is operating.
• The political model. Positions are already taken and research becomes ammunition for a side. Weiss's point, which is sharper than the cynical reading, is that this is not necessarily corrupt: a study that arms an under-resourced faction with good evidence has done real work, provided the evidence is sound.
• The enlightenment model. No single study changes any single decision. Concepts, framings and generalisations seep into how an organisation thinks over years. Weiss argued this is the dominant route by which social research actually influences policy - and it is invisible to any measurement scheme that asks "which decision did this study change?"

The enlightenment model is the reason the value-of-information test needs a companion rather than a monopoly. A body of research can be enormously valuable while no individual study in it clears a switch-point test, because its effect is cumulative and diffuse. A research programme judged only on per-study decision impact will systematically cancel the work that changes what the organisation believes is worth deciding about in the first place.

Weiss also names a tactical use, which is worth keeping separate: research commissioned not for its findings but for its existence - to buy time, to deflect a critic, to signal seriousness. The tell is that nobody asks what the result was.

The diagnostic: five questions, asked before fieldwork

You can identify which use you are being asked for in one short conversation, and you should have it before you write a single question. Ask the requester:

Two or more answers in the right-hand column and you are in category three. That is not a reason to refuse. It is a reason to change what you agree to deliver.

Note the fourth question especially. "Who else has to be convinced?" having a specific name is the single strongest signal, because it identifies the actual audience for the artefact - and an artefact built for a named skeptic is a different artefact from one built for a decision.

What to do when you spot it

Four moves, in rough order of preference.

1. Rename the deliverable and price it honestly. If the job is to help a leader carry a room, say so and build for it: fewer questions, sharper quotes, real customer voices on the specific objection the room will raise. This is a legitimate and useful deliverable, it is much cheaper than a decision-grade study, and calling it what it is protects the research function's credibility for the studies that are decision-grade.

2. Find the live decision next door. The headline decision is settled; the ones downstream of it usually are not. Pricing is fixed, but the migration path for existing customers is open. The feature is being built, but the default state is undecided. Redirect the same budget onto the question where a result still changes an action - this is almost always available, and it converts a zero-value study into a real one.

3. Pre-commit publication. The specific harm of legitimation research is not that it is run, it is that inconvenient findings vanish from it. Agree before fieldwork that the full result is circulated to a named list whatever it says. A requester who accepts this is in category two whether they realised it or not. A requester who refuses has told you exactly what the study is for, which is worth knowing. This is the organisational version of the file-drawer problem: an evidence base that only remembers convenient results is worse than no evidence base, because it carries false authority.

4. Accept it, and log it. Sometimes the right call is to run the supporting study, deliver it well, and record in the decision log that this was a support artefact rather than an input. That record is what stops a legitimation study being cited two years later as though it had been evidence - which is how a decision nobody would defend today acquires a paper trail that makes it look tested.

What not to do: run the study as though it were category one, discover the inconvenient result, and stage a confrontation at the readout. The result will be discounted, you will be seen as obstructive, and the finding will not survive the meeting. The time to have that argument is before fieldwork, when it is about method rather than about someone's judgement in front of their peers.

The one case where you should run it anyway

There is a case where a study with no switch point is still worth running on its own merits, and it is not a compromise.

When a decision is already taken but the reasoning is contested and will be revisited, research does not change this decision - it changes the next one. Weiss's enlightenment model in miniature: the study establishes a shared factual base that the organisation will still be arguing from in six months, when the decision genuinely reopens. Migrations, pricing architecture and platform bets all behave this way. The action does not move now; the option set the next time it is discussed does.

Two conditions distinguish this from self-deception. First, the question has to survive the specific decision - it is about customers, not about this quarter's plan. Second, somebody has to name the future moment at which it will be reconsidered. Without a named review point, "it will inform future thinking" is the standard euphemism for tactical use.

How Koji changes the calculus

The economics matter here more than the ethics, because much legitimation research exists precisely because good research is slow. When the honest study takes six weeks and the decision is on Friday, the only research that can exist is research that confirms.

Compressing the cycle changes which studies are possible:

• Speed reopens the category. An AI-moderated study that fields and reports in a day can run before the decision hardens rather than after it. The most effective intervention against decision-based evidence making is not a policy, it is being fast enough to be consulted while the question is still open.
• The brief makes the use explicit. Koji's AI consultant produces a structured brief from a plain-language goal - problem framing, target participant, typed question plan. Adding one line to it ("this study feeds decision X, owned by Y, taken on Z") converts the diagnostic above into a durable artefact rather than a conversation nobody remembers.
• Structured questions constrain the edit. Six question types - open_ended, scale, single_choice, multiple_choice, ranking and yes_no - produce results that are hard to soften. A scale distribution or a ranking order is difficult to present selectively in a way that survives someone opening the report. Open-ended findings are far easier to quote around, which is why legitimation studies gravitate to them. See structured questions.
• Published reports have one version. When the report is a link rather than a deck that gets edited between meetings, the pre-commitment in move three enforces itself. See publishing and sharing reports.
• The AI moderator does not read the room. A human moderator who knows which answer the sponsor wants will, without meaning to, probe the confirming answers harder. Koji asks its follow-ups from the participant's answer, identically for every participant - which is the same mechanism that makes confirmation bias harder to introduce at the collection stage.

Common mistakes

1. Treating category three as dishonest. It is not. Persuasion is a real organisational task and evidence genuinely serves it. Treating every support request as corruption is the fastest way to be excluded from the conversations where decisions are still open.
2. Diagnosing after the readout. By then you have spent the budget and staked your credibility on a finding that was never going to be acted on. The diagnostic is a pre-fieldwork instrument.
3. Confusing enlightenment with vagueness. "It will inform our thinking" is either a genuine enlightenment case with a named review point, or it is a tactical study with a nice name. Ask which.
4. Letting a support study enter the evidence base unlabelled. This is the durable harm. Label it in the repository, or it will be cited as evidence later - see the roadmap evidence audit for how untraceable claims accumulate.
5. Assuming the requester knows which one they want. Most do not. Tingling and Brydon's point is precisely that the confusion is usually sincere. The five questions are a tool for the requester as much as for you.

Frequently asked questions

Is it always wrong to run research that supports a decision already made?

No. Helping a leader bring an organisation along behind a defensible judgement is legitimate work, and good evidence does it better than assertion. Two conditions make it honest: the finding must be reported in full whatever it says, and the artefact must be labelled so that nobody later mistakes it for the input to a decision it did not inform. The wrong is not producing supporting material, it is producing it under the description of evidence.

How can I tell the difference before I have spent the budget?

Ask the five questions in the diagnostic table above, in a short conversation with whoever owns the decision. The strongest single signal is whether they can name a result that would change their mind, stated as a specific threshold. The second strongest is whether the decision date falls before or after the study reports. Both take one minute to establish and neither requires you to accuse anyone of anything.

What if I discover mid-study that the decision has already been taken?

Stop and renegotiate the deliverable rather than continuing on the original terms. Often the best available move is to redirect the remaining fieldwork onto a downstream question that is still open - the migration path, the default setting, the communication. Where that is not possible, finish the study, report it in full, and record in the decision log that it post-dates the decision, so its status is unambiguous to whoever finds it later.

Does Carol Weiss's enlightenment model justify research with no decision attached?

It justifies research whose value is cumulative rather than per-study, which is a narrower claim. The distinguishing test is whether the question survives the current decision cycle - genuinely about customers rather than about this quarter's plan - and whether somebody can name a future moment when it will be revisited. Without those two, "enlightenment" is being used as a label for work nobody intends to act on.

How does this relate to the value-of-information test?

They are the same test read in opposite directions. Value of information asks whether a result could change the action, and cancels the study when it cannot. This article asks what a study is for when the answer is no, and treats that answer as a finding about the organisation rather than as waste. Run the value test first; if it comes back zero, run the diagnostic here before you cancel anything.

Will faster research actually reduce legitimation studies?

It removes their most common cause, which is timing. A great deal of supporting research exists because the honest study could not have been delivered before the decision hardened, so the only research that could exist was research produced afterwards. When a study fields and reports in a day, it can be consulted while the question is open. It will not remove the studies that are genuinely political or tactical, and no tool will.

Related Resources

• Expected Value of Information - the test this article inverts: should this study exist at all?
• Structured Questions Guide - the six question types, and why typed results are harder to quote selectively
• Publication Bias and the File-Drawer Problem - what happens to an evidence base that only keeps convenient results
• The Roadmap Evidence Audit - tracing every claim on a roadmap back to a real source
• Outcome Bias - why the result of a decision is not evidence about the decision
• Confirmation Bias in User Research - the collection-stage version of the same pressure