{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-11-19T08:59:10.433Z"},"content":[{"type":"documentation","id":"4d90ffdb-d9de-43f0-b345-c854af52383f","slug":"value-of-information-research-decisions","title":"Expected Value of Information: How to Decide Whether a Study Is Worth Running (2026)","url":"https://www.koji.so/docs/value-of-information-research-decisions","summary":"The expected value of information decides whether a study should exist at all. It equals the probability your current belief is wrong multiplied by the cost of acting on a wrong belief, so a study whose results cannot change the action is worth nothing regardless of the decision's importance. Health economics formalises this as EVPI, the expected cost of uncertainty, which places an upper bound on what further research can be worth; in a York/NICE pilot the value of research across six appraisals ranged from GBP 2.8 million to GBP 865 million and contradicted expert research recommendations in some cases. A 2026 study of 4,857 online experiments by Abadie, Imbens and colleagues found that doubling statistical precision raised the value of evidence-based decision making by only 8.33%, while a 50% more varied idea portfolio raised it by 34.32%, and that significance-based decision rules discard roughly 27-30% of attainable value. Cheap AI-moderated interviews lower the cost side, which widens the set of decisions that deserve evidence.","content":"**Answer first: the question that decides whether a study is worth running is not \"how important is this?\" or \"can we afford it?\" It is \"what would we do differently depending on the answer?\" If no realistic result changes the action, the value of the information is zero and the correct sample size is zero, no matter how large the decision is. Health economics has a formal name and a formula for this - the expected value of perfect information - and it is the cleanest available answer to the question every research team is asked and almost none can answer: which studies should we not run?**\n\n## The test that comes before sample size\n\nMost research planning starts one step too late. Teams argue about how many interviews are enough, what confidence level to target, whether the sample is representative. Those are all questions about how *precisely* to answer. They assume the answer is worth having.\n\n![Two colleagues talking through what they would do differently depending on what a study finds](https://images.unsplash.com/photo-1580894736036-7a68513983ec?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3w5NDUzMDN8MHwxfHNlYXJjaHw4fHx0ZWFtJTIwbWVldGluZyUyMHdoaXRlYm9hcmQlMjBkaXNjdXNzaW9ufGVufDF8MHx8fDE3OTM5NTUwMTR8MA&ixlib=rb-4.1.0&q=80&w=1080)\n\n_Photo by [ThisisEngineering](https://unsplash.com/@thisisengineering?utm_source=koji&utm_medium=referral) on [Unsplash](https://unsplash.com/?utm_source=koji&utm_medium=referral)_\n\nThe prior question is whether the answer can change anything. Formally:\n\n**Value of information = (probability your current belief is wrong) x (cost of acting on the wrong belief)**\n\nBoth terms have to be non-zero. A question you are almost certainly right about has low value even if the stakes are enormous. A question you are genuinely uncertain about has low value if being wrong costs nothing, or if the decision is already fixed. The product of a small number and a large number is still small, and the product of anything and zero is zero.\n\nThis is the whole idea, and it is a deliberately uncomfortable one, because it means some of the most interesting questions your team can ask are worth exactly nothing to answer.\n\n## Where this comes from: the expected cost of uncertainty\n\nThe formal machinery is called value of information (VOI) analysis, and it is standard practice in health technology assessment, where a national body must decide both whether to fund a treatment and whether to fund more research about it. The framing translates directly to product work.\n\nThe clearest statement of the logic comes from the Centre for Health Economics at the University of York, in the pilot study Karl Claxton and colleagues produced to support research recommendations for NICE:\n\n> \"Decisions based on existing information will be uncertain, and there will always be a chance that the wrong decision will be made. If the wrong decision is made, there will be costs in terms of health benefit and resources forgone. Therefore, the expected cost of uncertainty is determined jointly by the probability that a decision based on existing information will be wrong and the consequences of a wrong decision.\"\n\nThat expected cost of uncertainty has a second name:\n\n> \"The expected costs of uncertainty can be interpreted as the expected value of perfect information (EVPI), since perfect information can eliminate the possibility of making the wrong decision.\"\n\nAnd then the sentence that makes it operational, because it converts a philosophical point into a budget line:\n\n> \"this is also the maximum that the health care system should be willing to pay for additional evidence to inform this decision in the future, and it places an upper bound on the value of conducting further research.\"\n\nThe decision rule that falls out is blunt: \"If the costs of investigation exceed the EVPI, then the proposed research will not be cost-effective.\" The York team put it in portfolio terms - population EVPI \"can be used to rule out research recommendations which will not be worth while.\"\n\nThe empirical result of that pilot is the part worth remembering. Across six technology appraisals, the value of further research ranged from **GBP 2.8 million to GBP 865 million** - a factor of roughly 300 between the cheapest and most valuable question in the same portfolio. And the analysis contradicted the experts: in some cases \"the analysis indicated that the original research recommendations should not be regarded as a priority,\" with liquid-based cytology screening the worked example at the low end, against GBP 865 million for clopidogrel in stroke patients.\n\nExpert committees had recommended further research on both. The arithmetic said one of those recommendations was not worth making. That is the failure mode this method exists to catch, and there is no reason to think product research committees are better calibrated than NICE appraisal committees.\n\n## The four inputs you actually need\n\nYou do not need a Bayesian model to run this. You need four numbers you can estimate in a fifteen-minute conversation with whoever owns the decision.\n\n![Table of the four inputs for a value of information check: the decision, the options, the switch point and the cost of being wrong, with how each one fails.](/visuals/value-of-information-four-inputs-table/value-of-information-four-inputs-table.png \"Write these four lines before a single question. An empty switch point means a study worth nothing.\")\n\n| Input | The question to ask | Typical failure |\n| --- | --- | --- |\n| The decision | What action does this study feed, and who takes it? | \"It's for general understanding\" - which means no decision, so no value |\n| The options | What are the two or three things we might actually do? | Only one option is real; the others are decoration |\n| The switch point | What result would make us pick a different option? | Nobody can name one, at any value |\n| The cost of being wrong | If we pick wrong, what does it cost over the life of the decision? | Stated as \"a lot\" rather than a number with a denominator and a time horizon |\n\nWrite those four lines down before you write a single question. If line three comes back empty, you have found a zero-value study, and the honest recommendation is to skip it.\n\n## A worked example\n\nA team wants to research a pricing change: move the mid tier from EUR 29 to EUR 39.\n\n- **Decision:** ship the new price on 1 October, or hold at EUR 29.\n- **Options:** two. There is no third.\n- **Switch point:** the team will hold if more than 20% of current mid-tier accounts say they would downgrade or churn at the new price.\n- **Cost of being wrong:** the mid tier holds roughly 4,000 accounts. A 10-point error in the churn response is about 400 accounts at EUR 29-39 per month, so somewhere between EUR 140,000 and EUR 190,000 of annual recurring revenue.\n\nNow the arithmetic. The team's current belief is that about 12% would leave, and they are genuinely unsure - they would not be shocked by 5% or by 25%. The switch point at 20% sits inside that range. That is exactly the condition under which information has value: **the plausible range of answers straddles the switch point.** A study costing a few hundred euros against a five- or six-figure exposure clears the bar by two orders of magnitude. Run it.\n\nChange one thing and the answer inverts. Suppose the team's belief is 3% with high confidence, and the switch point is still 20%. Now no realistic result crosses the line. The study is interesting and worthless in the same breath. Either accept the decision as made, or - the better move - go find the question where the range does straddle a switch point. In this case that is probably not \"will they churn\" but \"which accounts will churn and what would have kept them,\" which feeds a different decision with a different switch point.\n\n## The three shapes of a zero-value study\n\n| Shape | What it looks like | The tell |\n| --- | --- | --- |\n| No switch point | Every result leads to the same action | Ask \"what would you do if the answer were the opposite?\" and get a shrug |\n| Already decided | Contracts signed, launch dated, headcount moved | The study's deadline is after the decision's deadline |\n| Reversible at low cost | A two-way door you can walk back in a week | The rollback costs less than the study |\n\nThe third one is the least obvious and the most common source of wasted research. Research is a substitute for the ability to change your mind cheaply. When the change is genuinely cheap to reverse, shipping *is* the experiment, and the study is a slower, weaker version of the thing you could just do. This is why the value-of-information test and a good rollback plan are substitutes for one another, and why teams with strong feature flags should rationally do less pre-launch research and more post-launch measurement than teams without them.\n\n![Flowchart for deciding whether a study is worth running: four yes or no questions on switch point, open decision, reversibility and range, then run it.](/visuals/is-this-study-worth-running-flowchart/is-this-study-worth-running-flowchart.png \"Four questions, asked before anyone talks about sample size.\")\n\nThe second shape is a different animal, and it deserves its own treatment - a study commissioned after the decision is settled is not a mistake so much as a signal about the organisation. We take that apart in [evidence or ammunition](/docs/research-as-ammunition-legitimation).\n\n## What the newest evidence says: precision is not where the value is\n\nThere is a fresh and unusually direct piece of evidence here. In February 2026, Alberto Abadie (MIT), Guido Imbens (Stanford), Anish Agarwal (Columbia) and colleagues at Amazon published an empirical framework for estimating the value of evidence-based decision making, applied to the Upworthy Research Archive - **4,857 online experiments** comparing headline-and-image packages.\n\nTwo of their counterfactuals should change how research budgets are set.\n\nFirst, **precision buys surprisingly little.** Halving the estimation variance - roughly a 29% reduction in standard errors, which in practice means doubling your sample - increased the value of evidence-based decision making by **8.33%**, from 1.4057 to 1.5228.\n\nSecond, **the spread of your ideas buys a great deal.** Increasing the heterogeneity of true effects by 50% raised the value by **34.32%**; halving it cut the value by **44.22%**.\n\nRead those two together. Running bigger studies on the same portfolio of ideas produces single-digit gains. Generating a more varied portfolio of ideas - some genuinely bold, some genuinely different - produces gains several times larger. If your research programme is entirely devoted to measuring the same class of small changes more precisely, the arithmetic says you are optimising the wrong term. This is the strongest available argument for cheap, fast, high-variance discovery work over expensive confirmatory work, and it comes out of a portfolio of nearly five thousand real experiments.\n\nThe same paper adds a third finding worth flagging to anyone who ships on p-values: decision rules that require statistical significance \"can leave substantial value unrealized and, in some cases, generate negative expected value.\" In their data, a 5% significance rule delivered 1.0287 against 1.4057 for a value-based rule - a reduction of about **27%**, rising to roughly **30%** of attainable value discarded under their nonparametric estimates. Significance is a rule about error control, not about value, and the two come apart.\n\n## What changes when a study costs hours instead of weeks\n\nThe value-of-information test has a moving part most write-ups ignore: the cost side. Research is worth doing when its value exceeds its cost, so anything that lowers the cost widens the set of decisions that deserve evidence.\n\nThis is where AI-moderated research changes the arithmetic rather than just the workflow. A traditional moderated study - recruit, schedule, moderate, transcribe, code, synthesise - carries enough cost that it can only be justified for decisions with large exposure. Platforms like Koji run the interviews automatically, in voice or text, with the AI generating its own follow-up questions in the moment, and return an analysed report without a moderator. A study that used to cost several weeks and several thousand euros can cost a day and a fraction of that.\n\nThe consequence is not \"do more research.\" It is that **the threshold moves**, and a whole class of mid-sized decisions - the ones previously resolved by the loudest opinion in the room because research could not be justified - now clear the bar. Koji starts free with 10 credits and no card, interviews run as low as €1 per qualified interview, and conversations that fall below the quality gate are never charged, so the cost side of the calculation is a number you can look up rather than estimate.\n\nTwo Koji features map directly onto the four inputs above:\n\n- **The research brief.** Koji's AI consultant turns a plain-language goal into a structured brief with problem framing, target participant and a typed question plan. The brief is where you write the decision and the switch point down, which is the step teams skip. See [how to write a research brief](/docs/how-to-write-research-brief).\n- **Structured questions.** Six types - `open_ended`, `scale`, `single_choice`, `multiple_choice`, `ranking` and `yes_no` - let you put the switch point in the instrument itself. \"Would you downgrade at EUR 39?\" is a `yes_no`; \"how likely, 0-10\" is a `scale`; the reason is an `open_ended` the AI probes. A switch point defined on a typed question can be read straight off the report instead of argued about. See [structured questions](/docs/structured-questions-guide).\n\n![Koji draft research brief listing three interview questions with their question types](/email/shots/shot-card-brief.jpg \"The brief is where the decision and the switch point get written down, before fieldwork starts.\")\n\n## How this differs from two neighbouring methods\n\nTwo related articles in this library answer questions that sound similar and are not. Worth being precise, because using the wrong one wastes the effort.\n\n| Method | The question it answers | When to use it |\n| --- | --- | --- |\n| Value of information (this article) | Should this study exist at all? | Before any planning, per decision |\n| [The research risk model](/docs/research-risk-model-rigor-allocation) | Given that we are studying it, how much rigour does it deserve? | After VOI clears, to size the study |\n| [The re-research audit](/docs/re-research-audit-duplicate-studies) | Do we already own this answer? | Before both, as a cheap first filter |\n\nThe order is: do we already know it, would knowing it change anything, and only then how carefully should we find out. Most teams run only the third check.\n\n## Common mistakes\n\n1. **Treating importance as value.** The most important question in the company can have zero information value if the decision is settled or if no answer changes the action. Importance sets the *cost of being wrong*, which is only one of the two terms.\n2. **Setting the switch point after seeing the data.** A threshold chosen after the fact is not a threshold, it is a rationalisation. Write it in the brief. This is the same discipline that [blind analysis](/docs/blind-analysis-research) enforces on the analysis side.\n3. **Pricing the study but not the delay.** A four-week study on a decision that must be made in three weeks has zero value regardless of quality, because the decision will be taken without it.\n4. **Forgetting that value falls as certainty rises.** The second study on the same question is worth much less than the first, because it can only move a belief that has already moved. This is the formal reason [re-research](/docs/re-research-audit-duplicate-studies) is expensive.\n5. **Assuming the range straddles the switch point.** It usually does not. Ask people for their honest range first; if the whole range sits on one side of the line, you have your answer without fieldwork.\n\n## Frequently asked questions\n\n### What is the expected value of perfect information in plain terms?\n\nIt is the most you should ever be willing to pay to remove all uncertainty from a decision. It is calculated as the probability that your current belief is wrong multiplied by the cost of acting on a wrong belief. Because perfect information is the best information can possibly be, that number is a ceiling: any real study, which gives partial information, is worth less. If a proposed study costs more than the ceiling, it cannot be worth running.\n\n### How do I estimate the cost of being wrong without a financial model?\n\nUse the exposure, not the profit. Count the units affected - accounts, users, deals - over the period the decision holds, and multiply by a per-unit value you already track. You are not producing an accounting figure, you are producing an order of magnitude, and orders of magnitude are enough to separate a study worth hundreds from one worth hundreds of thousands. If two people on the team estimate independently and land within a factor of three, that is precise enough to act on.\n\n### Does this mean we should only research decisions with big numbers attached?\n\nNo, and this is the most common misreading. A small decision where you are genuinely uncertain and the plausible answers straddle your switch point can easily be worth more research than a huge decision where you are confident or already committed. The test is the product of the two terms, not the size of either one. Small, frequent, reversible decisions are usually better served by shipping and measuring than by studying.\n\n### What if the stakeholder cannot name a result that would change their mind?\n\nThen you have learned something more valuable than the study would have told you, and you should say so plainly and early. Sometimes the honest answer is that the decision is made and the request is for support rather than evidence - see [evidence or ammunition](/docs/research-as-ammunition-legitimation) for how to tell the difference and what to do about it. Sometimes it means the decision has not been framed sharply enough yet, and the right next step is a framing conversation rather than fieldwork.\n\n### How does this relate to sample size and statistical power?\n\nValue of information decides whether to run the study; power decides how big it has to be to detect the effect you care about. They answer different questions in sequence, and running the power calculation first is how teams end up precisely measuring things that do not matter. Once value of information clears, [statistical power and minimum detectable effect](/docs/statistical-power-minimum-detectable-effect) tells you what size instrument the switch point requires.\n\n### Does cheaper research make this test unnecessary?\n\nIt makes it more important, not less. When studies are expensive, the budget itself filters out low-value work, badly but automatically. When a study costs a day and a few credits, that filter disappears and teams can generate large volumes of research that answers nothing, which then has to be read, synthesised and stored. The value-of-information test is what replaces the discipline that cost used to impose for free.\n\n## Related Resources\n\n- [The Research Risk Model](/docs/research-risk-model-rigor-allocation) - how much rigour a question deserves once you have decided to study it\n- [The Re-Research Audit](/docs/re-research-audit-duplicate-studies) - the cheapest first filter: do you already own the answer?\n- [Evidence or Ammunition](/docs/research-as-ammunition-legitimation) - what a study is for when no result could change the decision\n- [Structured Questions Guide](/docs/structured-questions-guide) - the six question types, and how to put a switch point in the instrument\n- [Statistical Power and Minimum Detectable Effect](/docs/statistical-power-minimum-detectable-effect) - sizing the study once it clears the value test\n- [How to Write a Research Brief](/docs/how-to-write-research-brief) - where the decision and the switch point get written down\n- [The Questions Nobody Asks You](/docs/unasked-research-questions-suppressed-demand) - the decisions that never reach the value test at all\n- [We Already Knew That](/docs/we-already-knew-that-obvious-findings) - measuring the prior that decides whether a study is worth running\n- [The Range Your Finding Is Valid For](/docs/extrapolating-beyond-observed-range) - when the decision sits outside everything you measured\n","category":"Research Operations","lastModified":"2026-11-06T15:31:05.656002+00:00","metaTitle":"Expected Value of Information: Is This Study Worth Running? (2026)","metaDescription":"Run the value-of-information test before planning research: probability you are wrong times cost of being wrong. Includes EVPI, worked examples and 2026 evidence.","keywords":["expected value of information","is user research worth it","when not to run a study","EVPI","value of information analysis","research prioritization","should we run this study","research decision framework"],"aiSummary":"The expected value of information decides whether a study should exist at all. It equals the probability your current belief is wrong multiplied by the cost of acting on a wrong belief, so a study whose results cannot change the action is worth nothing regardless of the decision's importance. Health economics formalises this as EVPI, the expected cost of uncertainty, which places an upper bound on what further research can be worth; in a York/NICE pilot the value of research across six appraisals ranged from GBP 2.8 million to GBP 865 million and contradicted expert research recommendations in some cases. A 2026 study of 4,857 online experiments by Abadie, Imbens and colleagues found that doubling statistical precision raised the value of evidence-based decision making by only 8.33%, while a 50% more varied idea portfolio raised it by 34.32%, and that significance-based decision rules discard roughly 27-30% of attainable value. Cheap AI-moderated interviews lower the cost side, which widens the set of decisions that deserve evidence."}],"pagination":{"total":1,"returned":1,"offset":0}}