{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-11-19T08:59:04.729Z"},"content":[{"type":"documentation","id":"03bb77bf-8e41-4d0e-bef0-9e2ae0754f45","slug":"outcome-bias-research-decision-quality","title":"Outcome Bias: Why You Cannot Grade a Decision by Its Result (2026)","url":"https://www.koji.so/docs/outcome-bias-research-decision-quality","summary":"Outcome bias is the tendency to judge a decision by how it turned out even when the evaluator has exactly the information the decision maker had. Baron and Hershey established it in 1988 with a bypass-surgery scenario in which only the ending varied; a 2022 registered replication with 692 participants found effect sizes of d = 1.10 and d = 0.77, with mean ratings of about 1.8 for the successful version against 0.45 to 0.90 for the identical failed one, and participants who had said outcomes should not matter still showed the bias at d = 0.64. Unlike sampling or wording error, this has no technical fix: decision quality is unobservable in principle because outcome equals decision quality plus luck, one equation with two unknowns at n equals one. Proper scoring rules remain valid across many claims where luck averages out. The only auditable object is a decision record written before the outcome, reviewed in a fixed four-question order that discloses the result last.","content":"**Answer first: you cannot grade a decision by how it turned out, because the thing you want to grade - decision quality - is never observed. Only outcomes are, and an outcome is decision quality plus luck. Show people identical decisions with different endings and they rate the lucky one as better, by roughly a full point on a seven-point scale, with an effect size above 1.0. They do it even when they have just said outcomes should not matter. This is the last and hardest of the measurement problems, because unlike sampling error or wording effects, it has no technical fix: the counterfactual you would need does not exist. The only auditable object is the decision record you wrote before you knew.**\n\n## The experiment that established it\n\nJonathan Baron and John Hershey published \"Outcome bias in decision evaluation\" in the *Journal of Personality and Social Psychology* (54(4):569-579, 1988). The design is almost annoyingly simple, which is why it has held up.\n\nParticipants read about a 55-year-old man with a heart condition considering a bypass operation. The operation would relieve his pain and raise his life expectancy from about 65 to about 70. They were told that \"8% of the people who have this operation die from the operation itself.\" They then rated the decision to operate on a seven-point scale running from **3, \"clearly correct and the opposite decision would be inexcusable,\" to -3, \"incorrect and inexcusable.\"**\n\nEvery participant saw the same information the decision maker had. The only thing that varied was how it ended: the patient recovered, or the patient died. Baron and Hershey found that the decision was rated better when the outcome was good - effect sizes of about d = 0.21 when the patient decided and d = 0.53 when the physician did.\n\nThe result has since been re-run at modern sample sizes. A registered replication of Experiment 1 published in the *International Review of Social Psychology* (2022) recruited **692 participants** and found the effect substantially larger than the original:\n\n![Bar chart: a surgery decision rated 1.81 if the patient lived and 0.45 if he died when a physician decided, and 1.76 against 0.90 when the patient did.](/visuals/outcome-bias-surgery-decision-ratings-replication/outcome-bias-surgery-decision-ratings-replication.png \"Same facts, same choice, a much lower grade when it ended badly. · Source: Registered replication of Baron and Hershey (1988), International Review of Social Psychology, 2022\")\n\n| Condition | Outcome | n | Mean rating | SD |\n| --- | --- | --- | --- | --- |\n| Physician decided | Survived | 173 | 1.81 | 0.84 |\n| Physician decided | Died | 172 | 0.45 | 1.55 |\n| Patient decided | Survived | 171 | 1.76 | 0.79 |\n| Patient decided | Died | 176 | 0.90 | 1.36 |\n\nEffect sizes were **d = 1.10** in the physician condition and **d = 0.77** in the patient condition. The authors described it as \"a successful replication with signal and direction consistent with that of the target article's findings.\"\n\nThen the finding that should end the argument. A subgroup of participants (n = 44) separately reported that outcomes \"definitely\" or \"probably\" should not influence how a decision is evaluated. **They showed the bias anyway: d = 0.64, p = .03.**\n\nKnowing about outcome bias does not protect you from outcome bias. That is what makes it a structural problem rather than a training problem.\n\n## The capstone claim: the quantity you want is never observed\n\nThis library contains a long series of articles about things that make a measurement wrong. [Total survey error](/docs/total-survey-error-research-budget) prices seven of them against a fixed budget. [Sampling bias](/docs/sampling-bias-research) is about who got in. [Statistical power](/docs/statistical-power-minimum-detectable-effect) is about whether the instrument could have seen the effect. Every one of them shares a structure: there is a true value out there, your estimate misses it, and the article tells you how to miss by less.\n\nDecision quality does not have that structure, and this is the point of this article.\n\n**Decision quality is not a hard-to-measure quantity. It is an unobservable one.** A good decision is one that was correct given what could be known at the time, over the distribution of things that might have happened. Exactly one of those things did happen. The others - the ones that define whether the decision was right - did not, and never will. There is no larger sample, no better instrument and no cleaner design that recovers them, because the counterfactual is not missing data. It does not exist.\n\nWhat you observe instead is the outcome, which is:\n\n**Outcome = decision quality + luck**\n\nTwo unknowns, one equation, one observation. No amount of rigour solves an underdetermined system. This is why outcome bias is not a lapse in judgement that better-trained reviewers avoid: reviewers are asked to report a quantity they cannot see, and they substitute the only correlated quantity available.\n\nThat substitution is not irrational, either. Over many decisions, outcome really is evidence about quality - the luck term averages out. The error is applying that logic to n = 1, where it does not.\n\n## Why this is not fixed by measuring harder\n\nThree properties make outcome bias resistant to the standard remedies.\n\n**It survives knowing about it.** The replication's d = 0.64 among people who had just endorsed the correct principle is the cleanest demonstration available. Compare this with most measurement problems, where naming the mechanism largely defuses it.\n\n**It is asymmetric in organisations.** Bad outcomes get reviewed; good ones do not. A launch that succeeds is never subjected to a post-mortem asking whether the reasoning was sound, so bad reasoning that got lucky is never corrected and enters the culture as a precedent. This is the [survivorship](/docs/survivorship-bias-customer-research) mechanism applied to decisions rather than customers, and it means the correction is systematically one-sided.\n\n**It has second-order effects on behaviour.** People who expect to be judged on outcomes take decisions that produce defensible outcomes rather than good ones. The characteristic symptom is risk aversion on exactly the high-variance bets that portfolio evidence says carry the returns - which is where this connects to [the base rate for a product bet](/docs/product-idea-base-rate-outside-view). If two-thirds of well-reasoned ideas fail, and failure is punished at the individual level, the rational individual response is to stop proposing ideas that could fail visibly. The organisation gets a safer, worse portfolio, and no single person did anything unreasonable.\n\n## Reconciling this with calibration scoring\n\nThere is an apparent contradiction with [calibration scoring for research teams](/docs/research-calibration-brier-score), which argues for grading research claims against what actually happened using proper scoring rules. If outcomes cannot grade decisions, how can outcomes grade forecasts?\n\nThe reconciliation is about **n**, and it is worth stating precisely because it determines which tool you reach for.\n\n| | Grading one decision | Grading a track record |\n| --- | --- | --- |\n| What you observe | One outcome | Many outcomes |\n| The luck term | Dominates; cannot be separated | Averages toward zero |\n| Valid method | Review the process and the record | Proper scoring rules across all claims |\n| Invalid method | \"It worked, so it was a good call\" | \"Their last call was wrong, so they are miscalibrated\" |\n\nA Brier score over fifty claims is informative precisely because the noise cancels. A Brier score over one claim is a coin flip with a decimal point. So the rule is: **score the record, review the decision.** Both articles are right within their n, and the failure mode in each direction is the other one's method applied at the wrong scale.\n\nThis also explains why a single spectacular success or failure is such poor evidence about a person or a team, and why organisations that promote on the strength of one shipped hit are running the invalid method in the left-hand column.\n\n## What you can grade: the decision record\n\nIf the outcome cannot be the object of review, something else has to be, and it has to be created before the outcome is known. That artefact is a decision record, and it is the only defence with any evidence behind it.\n\nThe fields that make a decision reviewable:\n\n![Table of seven decision record fields, from the decision and its owner to the expected outcome, the switch point and the review date, with why each matters.](/visuals/decision-record-template-fields/decision-record-template-fields.png \"Write it before the outcome is known, or it records the reconstruction.\")\n\n| Field | What it captures | Why it is needed later |\n| --- | --- | --- |\n| The decision and its owner | What was decided, by whom, when | Prevents retrospective reassignment of authorship |\n| The options considered | The two or three real alternatives | A decision with one option was not a decision |\n| What was known | The evidence available, with links | Stops hindsight importing facts that arrived later |\n| What was uncertain | The things nobody knew, stated plainly | The luck term, named in advance |\n| The expected outcome | A probability or a range, not a hope | Makes the claim scoreable across many decisions |\n| The switch point | What would have changed the call | Ties back to [the value of information](/docs/value-of-information-research-decisions) |\n| The review date | When this gets looked at again | Turns a document into a loop |\n\nThe discipline that gives this its power is that the record is written **before** the outcome. Hindsight reliably rewrites what people believe they knew, so a record reconstructed after the fact is not evidence about the decision - it is evidence about the reconstruction.\n\n## The four questions of an honest decision review\n\nWith a record in hand, a review can ask questions that have answers:\n\n1. **Given only what is in the record, was this the best available option?** Not \"did it work.\" Ban outcome information from the first pass entirely - the same logic that makes [blind analysis](/docs/blind-analysis-research) work.\n2. **Was anything knowable at the time missing from the record?** This is the real learning question, and it is the one that improves future decisions. A study that could have been run cheaply and was not is a genuine process failure; an unknowable fact is not.\n3. **Did the outcome fall inside the stated range?** If a range was written down and the result landed outside it, the model of the world was wrong, and that is separable from the decision being wrong.\n4. **Only now: what did the outcome cost or earn, and does anything need repairing?** A real question, but a separate one from decision quality, and running it fourth prevents it colouring the first three.\n\n![Process of four decision review questions in order: was it the best option on the record, what was missing, did the outcome land in range, then what it cost.](/visuals/honest-decision-review-four-questions/honest-decision-review-four-questions.png \"Most post-mortems start at question four. The order is the fix.\")\n\nA review that starts at question four - which is what almost every post-mortem does, because that is what triggered the meeting - will produce outcome bias reliably. The order is the intervention.\n\n## How Koji supports this\n\nThe hard part of the decision record is not the template. It is that the evidence it points at has to exist, be linked, and still be readable a year later when the review happens.\n\n- **Studies stay attached to the decision.** A Koji report is a durable, shareable link rather than a deck that gets edited between meetings, so \"what was known\" in a decision record points at the actual instrument, sample and result rather than a summary of them. See [publishing and sharing reports](/docs/publishing-sharing-reports).\n- **Structured questions make expectations scoreable.** Six question types - `open_ended`, `scale`, `single_choice`, `multiple_choice`, `ranking` and `yes_no` - turn \"customers will like this\" into a distribution on a defined scale. A stated expectation of \"median 7 or better on the 0-10 scale\" can be checked at review time; \"we thought it would land well\" cannot.\n- **Speed lets you record the expectation before you commit.** The reason expectations go unrecorded is usually that the decision moves faster than the research. When a study fields and reports in a day - Koji runs the interviews itself in voice or text and writes its own follow-up questions from each answer - the record can be written while the decision is still open rather than reconstructed after it.\n- **The AI moderator does not know the outcome.** Any follow-up research after a launch is conducted by a moderator with no stake in whether the launch is judged a success, which removes one channel through which a known outcome shapes the evidence collected about it.\n\n## Common mistakes\n\n1. **Reviewing only failures.** The asymmetry is the mechanism. If you only review bad outcomes, you only ever correct the reasoning that got unlucky, and lucky bad reasoning becomes precedent. Sample some successes.\n2. **Writing the decision record after the outcome.** This is not a record, it is a reconstruction, and hindsight has already edited it.\n3. **Judging a person on one call.** Valid at large n, invalid at n = 1. See the table above.\n4. **Confusing a bad outcome with a bad decision, or a good outcome with a good one.** Both errors are the same error. The second is more dangerous because nobody complains about it.\n5. **Recording a hope instead of an expectation.** \"We expect this to do well\" cannot be scored. \"We expect 20-30% of trials to convert, and would be surprised outside 15-40%\" can.\n6. **Assuming awareness is enough.** The replication participants who said outcomes should not matter still showed d = 0.64. Structure beats intention here; the four-question order is the structure.\n\n## Frequently asked questions\n\n### What is outcome bias?\n\nOutcome bias is the tendency to judge the quality of a decision by how it turned out, even when the person judging has exactly the information the decision maker had. Baron and Hershey demonstrated it in 1988 using a medical scenario in which only the ending varied, and a 2022 replication with 692 participants found effect sizes of d = 0.77 to d = 1.10 - a decision rated around 1.8 on a seven-point scale when it worked and as low as 0.45 when the identical decision failed.\n\n### How is outcome bias different from hindsight bias?\n\nHindsight bias is about knowledge: once you know what happened, you misremember having predicted it. Outcome bias is about evaluation: you judge the decision more harshly or more kindly because of the result, even while correctly recalling the information available beforehand. They usually appear together and reinforce each other, but the remedies differ - hindsight bias is countered by a written record of what was believed, outcome bias by reviewing the decision before the outcome is disclosed.\n\n### Can we not just train people to ignore outcomes?\n\nThe evidence says no, at least not by awareness alone. In the 2022 replication, participants who explicitly reported that outcomes definitely or probably should not influence evaluation still showed the bias at d = 0.64. Structural remedies work better than intentions: write the decision record before the result is known, review the reasoning before disclosing the outcome, and sample successes for review rather than only failures.\n\n### If outcomes cannot grade decisions, how can Brier scores grade forecasts?\n\nThe difference is sample size. Over many claims, the luck component of outcomes averages toward zero and what remains is signal about judgement, which is exactly what a proper scoring rule extracts. Over a single claim, luck dominates and no scoring rule can separate it from quality. The working rule is to score the track record and review the individual decision, and to treat any confident judgement of one decision from its outcome as unsupported.\n\n### What goes in a decision record?\n\nThe decision and its owner, the options genuinely considered, what was known with links to the evidence, what was uncertain, an expected outcome stated as a probability or range, the result that would have changed the call, and a date for review. The single most important property is that it is written before the outcome is known - a record reconstructed afterwards documents the reconstruction rather than the decision.\n\n### Does this mean we should stop holding teams accountable for results?\n\nNo. It means separating two accountabilities that usually get merged. Teams remain accountable for outcomes at the portfolio level, where luck averages out and the numbers mean something. Individual decisions are held accountable for process: were the options considered, was cheap available evidence gathered, was the expectation stated in advance. Merging the two produces predictable risk aversion, which is expensive in a domain where roughly one idea in three works and the returns sit in the tail.\n\n## Related Resources\n\n- [Calibration Scoring for Research Teams](/docs/research-calibration-brier-score) - the method that is valid across many claims, where this one is not\n- [The Base Rate for a Product Bet](/docs/product-idea-base-rate-outside-view) - what the portfolio evidence says about how often good decisions still fail\n- [Expected Value of Information](/docs/value-of-information-research-decisions) - the switch point that belongs in every decision record\n- [Structured Questions Guide](/docs/structured-questions-guide) - the six question types that turn an expectation into something scoreable\n- [Blind Analysis](/docs/blind-analysis-research) - withholding the outcome so the judgement comes first\n- [Evidence or Ammunition](/docs/research-as-ammunition-legitimation) - the related failure where the decision precedes the evidence\n- [Authority Gradient in Research Teams](/docs/authority-gradient-research-teams) - why the objection that would have caught it never carried\n","category":"Research Operations","lastModified":"2026-11-19T08:18:35.740426+00:00","metaTitle":"Outcome Bias: Why a Decision Cannot Be Graded by Its Result (2026)","metaDescription":"People rate identical decisions better when they happen to work, at d over 1.0, even after saying outcomes should not matter. What to review instead of the result.","keywords":["outcome bias","judge a decision by its outcome","decision quality","resulting decision making","process versus outcome","decision record","hindsight bias","post-mortem bias"],"aiSummary":"Outcome bias is the tendency to judge a decision by how it turned out even when the evaluator has exactly the information the decision maker had. Baron and Hershey established it in 1988 with a bypass-surgery scenario in which only the ending varied; a 2022 registered replication with 692 participants found effect sizes of d = 1.10 and d = 0.77, with mean ratings of about 1.8 for the successful version against 0.45 to 0.90 for the identical failed one, and participants who had said outcomes should not matter still showed the bias at d = 0.64. Unlike sampling or wording error, this has no technical fix: decision quality is unobservable in principle because outcome equals decision quality plus luck, one equation with two unknowns at n equals one. Proper scoring rules remain valid across many claims where luck averages out. The only auditable object is a decision record written before the outcome, reviewed in a fixed four-question order that discloses the result last."}],"pagination":{"total":1,"returned":1,"offset":0}}