{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-08-03T18:01:00.715Z"},"content":[{"type":"documentation","id":"b31ac21a-572e-49b8-8fa1-1452076905dc","slug":"ai-incident-postmortem-user-research","title":"AI Incident Postmortems: How to Investigate Model Failures with User Evidence (2026)","url":"https://www.koji.so/docs/ai-incident-postmortem-user-research","summary":"An AI incident postmortem is a written, blameless investigation of a model or agent failure that records impact, timeline, contributing causes, and owned corrective actions. AI incidents differ from ordinary software incidents because they usually produce no error code - the system returns a confident, well-formed, wrong answer - so telemetry cannot scope the harm. This guide covers what counts as an incident, how to classify severity, the EU AI Act Article 73 reporting clocks, the postmortem template adapted for probabilistic systems, and how to gather user evidence fast enough to meet those clocks using Koji.","content":"# AI Incident Postmortems: How to Investigate Model Failures with User Evidence (2026)\n\n**Short answer:** An AI incident postmortem is a blameless written investigation of a model or agent failure that records impact, timeline, contributing causes, and owned corrective actions. It differs from an ordinary software postmortem in one decisive way: AI systems usually fail without an error. There is no exception, no 500, no alert - just a confident, fluent, wrong answer returned as a success. That means your telemetry can tell you what the model said, but only your users can tell you what it cost.\n\nMost teams discover this the hard way, mid-incident, when someone asks \"how many customers were affected?\" and the honest answer is that nobody knows.\n\n## What counts as an AI incident\n\nDraw the line explicitly, before you need it. An AI incident is any deployed model behaviour that:\n\n- causes or nearly causes harm to a person, financially, physically, reputationally, or psychologically;\n- produces a materially wrong result that a user **acted on**;\n- violates a stated policy, a permission boundary, or a legal obligation;\n- degrades an agreed quality metric beyond a defined threshold for a defined period.\n\nNote what is absent from that list: downtime, exceptions, and error rates. The Responsible AI Collaborative's AI Incident Database - which indexes real-world AI harms and near-harms and has now issued incident IDs past 1,600 - is instructive here. Read through it and the pattern is unmistakable: very few entries are outages. They are systems working exactly as built, on inputs nobody anticipated, for users nobody profiled.\n\n**The near-harm is worth logging too.** Aviation safety culture, from which this practice descends, treats the near miss as the cheapest possible lesson. An AI system that produced a dangerous recommendation which a user happened to catch is a free incident - all of the signal, none of the damage.\n\n## Why AI incidents need user evidence\n\nIn a conventional incident, the blast radius is a query: which requests failed, between which timestamps, for which accounts. In an AI incident, the same query returns everything and tells you nothing, because every one of those requests succeeded.\n\nWhether a wrong output became a harm depends entirely on the human step that followed it:\n\n| Model behaviour | Log signature | Actual outcome | Determined by |\n|---|---|---|---|\n| Wrong dosage guidance | 200 OK | User noticed, ignored it | The user |\n| Wrong dosage guidance | 200 OK | User followed it | The user |\n| Fabricated citation | 200 OK | Caught in review | The reviewer |\n| Fabricated citation | 200 OK | Published | The reviewer |\n| Over-refusal | 200 OK | User rephrased, succeeded | The user |\n| Over-refusal | 200 OK | User abandoned the task | The user |\n\nIdentical telemetry, opposite severity. This is the structural reason an AI postmortem that consults only logs will systematically under-report harm - and why the phrase \"we saw no elevated error rate\" is not evidence of anything.\n\nThe corroborating industry signal: the 2025 Stack Overflow Developer Survey found **66% of developers name AI output that is \"almost right, but not quite\" as their single largest frustration, and 45% report that debugging AI-generated code takes longer** than writing it themselves. Near-misses are the dominant failure mode, and near-misses are precisely what monitoring cannot see.\n\n## Blameless, properly understood\n\nGoogle's SRE practice popularised the blameless postmortem, and it is probably the most misread term in the discipline. Blameless does **not** mean no consequences, no accountability, or no follow-up. It means assuming everyone involved had good intentions and acted on the best information available to them at the time, and pointing the investigation at the system rather than the person.\n\nThe reason this is operational rather than sentimental: removing blame gives people the confidence to escalate early. In AI incidents, where detection depends on a human noticing something subtly wrong, the cost of a culture where people hesitate to raise a concern is measured in weeks of undetected harm.\n\nTwo adaptations for AI systems:\n\n1. **The cause is rarely a commit.** Expect a combination - a prompt revision, a retrieval index rebuild, an upstream model version change, and a shift in the input distribution, none individually sufficient. Resist the pressure to name one.\n2. **\"The model did it\" is not a root cause.** It is a restatement of the incident. Push through to the design decision that let a probabilistic component take an irreversible action without a check.\n\n## The reporting clock\n\nThis is no longer purely voluntary practice. Under **Article 73 of the EU AI Act**, providers of high-risk AI systems must report serious incidents to the market surveillance authority of the member state where the incident occurred, as soon as a causal link to their system is established or reasonably likely. The deadlines are tiered by severity:\n\n| Situation | Deadline from awareness |\n|---|---|\n| Serious incident (general) | 15 days |\n| Death may have been caused | 10 days |\n| Widespread infringement, or serious disruption to critical infrastructure | 2 days |\n\nArticle 73(5) explicitly permits an **initial incomplete report** with fuller information to follow - which is the mechanism that makes these deadlines survivable. Authorities may order market surveillance measures within days of receiving a report, so the quality of your initial evidence directly shapes what happens next.\n\nA two-day clock is the part worth internalising. Two days is not enough time to recruit, schedule, moderate, transcribe, and synthesise a conventional research study. It is enough time to run an AI-moderated one.\n\n## The AI postmortem template\n\n| Section | What goes in it | AI-specific note |\n|---|---|---|\n| Summary | Two sentences: what happened, who was affected | Quantify affected users, not affected requests |\n| Impact | Harm distribution by severity | Requires user evidence, not logs |\n| Detection | How and when you found out | Record whether a user or a monitor found it - if it was always users, that is itself a finding |\n| Timeline | First bad output to full mitigation | Include the silent period before detection |\n| Contributing causes | Plural, always | Prompt, retrieval, model version, distribution shift, missing guardrail |\n| What went well | Genuinely - what limited the damage | Usually a human check somewhere |\n| Action items | Owned, dated, split mitigative vs preventative | Add an evaluation-set item to every AI postmortem |\n| Evidence appendix | Sample outputs, participant quotes, severity counts | This is your regulatory filing material |\n\nSplit action items into **mitigative** (closes this specific gap) and **preventative** (addresses the whole class of failure). AI postmortems should almost always generate a third kind: an **evaluation** item - the failure, converted into permanent test cases and added to your golden set so the regression can never silently return.\n\n## Severity classification\n\nScore every confirmed harm on one scale, defined in advance:\n\n| Level | Definition | Reporting implication |\n|---|---|---|\n| S1 - Harmful | User suffered material harm; irreversible or costly | Regulatory reporting likely; executive notification |\n| S2 - Blocking | User could not complete a critical task; no workaround | Full postmortem required |\n| S3 - Degraded | Task completed but with wasted effort or lost trust | Postmortem if systemic |\n| S4 - Cosmetic | Noticed, no consequence | Log and aggregate |\n\nThe count that matters in the summary is **S1 and S2 users**, not total requests. Executives and regulators both ask the same first question, and \"0.3% of requests\" is not an answer to it.\n\n## Running the user-evidence study with Koji\n\nThe traditional path - recruit affected users, schedule interviews, moderate 12 of them, transcribe, tag, synthesise - takes two to three weeks. That is longer than every deadline in the table above and longer than the patience of any executive during an active incident. It is also why most postmortems quietly substitute a support-ticket sample for real evidence, which biases everything toward the users who complained loudly.\n\nThe AI-native approach:\n\n**1. Launch within hours, not weeks.** Define the exposed population from your incident window, invite a representative sample, and run an always-on AI-moderated study. Participants respond on their own schedule, which removes the scheduling bottleneck entirely - the single largest source of delay in conventional research.\n\n**2. Probe what the user actually did.** This is the question a survey cannot ask well, because the useful follow-up depends on the answer. Koji's AI moderator asks it: *\"You said the recommendation looked off - what did you do next?\"* and then follows the answer wherever it goes. That branch is the difference between knowing an output was wrong and knowing whether it caused harm.\n\n**3. Produce a comparable severity distribution.** Koji's six structured question types turn testimony into countable evidence:\n\n| Question type | Incident use |\n|---|---|\n| `single_choice` | Which of these did you experience? |\n| `yes_no` | Did you act on the output before realising it was wrong? |\n| `scale` | How much impact did this have on you? (1-5) |\n| `ranking` | Order these consequences by how much they mattered |\n| `multiple_choice` | What did you do after you noticed? |\n| `open_ended` | Describe what happened in your own words |\n\nThe `yes_no` acted-on-it question is the one that converts a quality problem into a harm count - and it is the number your postmortem summary and any regulatory filing both need. See the [structured questions guide](/docs/structured-questions-guide) for how to sequence these without leading participants.\n\n**4. Cluster harms automatically.** Thematic analysis groups hundreds of open-ended accounts into named harm categories, each linked back to verbatim quotes and specific participants. Manual tagging of 60 transcripts is roughly a week of researcher time; this is the step that makes a real evidence base compatible with a two-day clock.\n\n**5. Keep it as a regression study.** The same study, re-run after the fix, is your verification that the mitigation worked from the user's side rather than the dashboard's.\n\nBecause Koji is AI-native rather than a scheduling layer over human moderators, a 50-participant incident study runs in the window where it can still change the outcome. That is the whole argument: research that arrives after the postmortem is filed is not evidence, it is history.\n\n## Sample sizes during an incident\n\n| Goal | Participants | Turnaround |\n|---|---|---|\n| Confirm the failure is real and characterise it | 10-15 | Same day |\n| Scope the harm distribution defensibly | 40-60 | 2-3 days |\n| Verify the mitigation from the user side | 30-40 | Post-fix |\n| Track trust recovery | 40+ | Monthly for a quarter |\n\n## Five mistakes to avoid\n\n1. **Reporting requests instead of people.** Percentages of traffic conceal concentrated harm on a small, specific group.\n2. **Sampling only complainants.** Support tickets over-represent articulate, high-engagement users and under-represent the ones who silently left.\n3. **Naming a single root cause.** AI incidents are almost always a conjunction. A single named cause usually means the investigation stopped early.\n4. **Closing without an evaluation item.** If the failure did not become a permanent test case, you have licensed its return.\n5. **Waiting for a complete picture before filing.** Article 73(5) exists precisely so you do not have to. File the initial report, then complete it.\n\n## Frequently asked questions\n\n**What counts as an AI incident?**\nAny deployed model behaviour that causes or nearly causes harm, produces a materially wrong result a user acted on, violates a policy or legal obligation, or degrades a quality metric past an agreed threshold. It does not require an error, an exception, or downtime - most AI incidents return HTTP 200.\n\n**How is an AI incident postmortem different from a normal one?**\nThree ways: there is usually no error signal, so detection depends on users rather than alerts; the cause is rarely a single commit but a conjunction of prompt, retrieval, model version, and distribution shift; and the blast radius cannot be read from logs, because whether a wrong output caused harm depends on what the user did with it.\n\n**What does blameless actually mean?**\nAssuming everyone involved acted with good intent on the best information available at the time, and directing the investigation at systems rather than individuals. It does not mean no consequences. Its practical purpose is that people escalate early instead of hiding problems.\n\n**What are the EU AI Act reporting deadlines for serious incidents?**\nUnder Article 73, providers of high-risk systems report to the market surveillance authority of the member state where the incident occurred: 15 days generally, 10 days where a death may have been caused, and 2 days for widespread infringement or serious disruption to critical infrastructure. Article 73(5) permits an initial incomplete report.\n\n**How do you scope who was affected when telemetry cannot tell you?**\nYou sample. Identify the population plausibly exposed during the incident window, draw a representative sample, and ask them what they saw and what they did next. A structured study of 40-60 affected users characterises the harm distribution far better than any log query.\n\n**How does Koji help during an incident?**\nSpeed is the binding constraint when a 15-day or 2-day clock is running. Koji launches an AI-moderated study to affected users within hours, probes what each person actually did with the wrong output, produces a comparable severity distribution through structured questions, and clusters responses into named harm categories - evidence in days rather than weeks.\n\n## Related Resources\n\n- [Root Cause Analysis for Customer Research](/docs/root-cause-analysis-guide) - the general RCA method this adapts\n- [The Five Whys Technique](/docs/five-whys-technique-user-research) - drilling past the first plausible cause\n- [Human Evaluation of AI Outputs](/docs/human-evaluation-ai-outputs) - the ongoing quality practice that catches incidents earlier\n- [AI Red Teaming with Real Users](/docs/ai-red-teaming-with-users) - finding these failures before they become incidents\n- [The EU AI Act and User Research](/docs/eu-ai-act-user-research-compliance) - the wider compliance picture\n- [AI Governance for Customer Research](/docs/ai-governance-frameworks-research) - ISO 42001 and the NIST AI RMF\n- [Structured Questions in AI Interviews](/docs/structured-questions-guide) - turning testimony into countable evidence\n\n---\n\n**Sources:** EU AI Act Article 73 (Reporting of Serious Incidents) and the European Commission draft guidance on serious incident reporting; Google SRE, *Postmortem Culture: Learning from Failure*; AI Incident Database, Responsible AI Collaborative; 2025 Stack Overflow Developer Survey.","category":"Research Operations","lastModified":"2026-08-02T03:20:44.868648+00:00","metaTitle":"AI Incident Postmortems: Investigating Failures with User Evidence","metaDescription":"How to run a blameless postmortem for an AI product failure: what counts as an incident, why telemetry alone cannot scope harm, the EU AI Act reporting clocks, and how to gather user evidence in days with Koji.","keywords":["ai incident postmortem","blameless postmortem","ai incident response","model failure investigation","ai incident reporting","eu ai act article 73","ai regression","root cause analysis ai"],"aiSummary":"An AI incident postmortem is a written, blameless investigation of a model or agent failure that records impact, timeline, contributing causes, and owned corrective actions. AI incidents differ from ordinary software incidents because they usually produce no error code - the system returns a confident, well-formed, wrong answer - so telemetry cannot scope the harm. This guide covers what counts as an incident, how to classify severity, the EU AI Act Article 73 reporting clocks, the postmortem template adapted for probabilistic systems, and how to gather user evidence fast enough to meet those clocks using Koji.","aiPrerequisites":["Familiarity with incident response or root cause analysis","Basic understanding of how AI models fail in production"],"aiLearningOutcomes":["Distinguish an AI incident from an ordinary software bug","Classify AI incident severity and scope affected users","Apply blameless postmortem principles to probabilistic systems","Meet EU AI Act Article 73 reporting deadlines with defensible evidence","Run a rapid user-evidence study during an active incident with Koji"],"aiDifficulty":"intermediate","aiEstimatedTime":"13 min"}],"pagination":{"total":1,"returned":1,"offset":0}}