AI Incident Postmortems: How to Investigate Model Failures with User Evidence (2026)

Short answer: An AI incident postmortem is a blameless written investigation of a model or agent failure that records impact, timeline, contributing causes, and owned corrective actions. It differs from an ordinary software postmortem in one decisive way: AI systems usually fail without an error. There is no exception, no 500, no alert - just a confident, fluent, wrong answer returned as a success. That means your telemetry can tell you what the model said, but only your users can tell you what it cost.

Most teams discover this the hard way, mid-incident, when someone asks "how many customers were affected?" and the honest answer is that nobody knows.

What counts as an AI incident

Draw the line explicitly, before you need it. An AI incident is any deployed model behaviour that:

• causes or nearly causes harm to a person, financially, physically, reputationally, or psychologically;
• produces a materially wrong result that a user acted on;
• violates a stated policy, a permission boundary, or a legal obligation;
• degrades an agreed quality metric beyond a defined threshold for a defined period.

Note what is absent from that list: downtime, exceptions, and error rates. The Responsible AI Collaborative's AI Incident Database - which indexes real-world AI harms and near-harms and has now issued incident IDs past 1,600 - is instructive here. Read through it and the pattern is unmistakable: very few entries are outages. They are systems working exactly as built, on inputs nobody anticipated, for users nobody profiled.

The near-harm is worth logging too. Aviation safety culture, from which this practice descends, treats the near miss as the cheapest possible lesson. An AI system that produced a dangerous recommendation which a user happened to catch is a free incident - all of the signal, none of the damage.

Why AI incidents need user evidence

In a conventional incident, the blast radius is a query: which requests failed, between which timestamps, for which accounts. In an AI incident, the same query returns everything and tells you nothing, because every one of those requests succeeded.

Whether a wrong output became a harm depends entirely on the human step that followed it:

Identical telemetry, opposite severity. This is the structural reason an AI postmortem that consults only logs will systematically under-report harm - and why the phrase "we saw no elevated error rate" is not evidence of anything.

The corroborating industry signal: the 2025 Stack Overflow Developer Survey found 66% of developers name AI output that is "almost right, but not quite" as their single largest frustration, and 45% report that debugging AI-generated code takes longer than writing it themselves. Near-misses are the dominant failure mode, and near-misses are precisely what monitoring cannot see.

Blameless, properly understood

Google's SRE practice popularised the blameless postmortem, and it is probably the most misread term in the discipline. Blameless does not mean no consequences, no accountability, or no follow-up. It means assuming everyone involved had good intentions and acted on the best information available to them at the time, and pointing the investigation at the system rather than the person.

The reason this is operational rather than sentimental: removing blame gives people the confidence to escalate early. In AI incidents, where detection depends on a human noticing something subtly wrong, the cost of a culture where people hesitate to raise a concern is measured in weeks of undetected harm.

Two adaptations for AI systems:

1. The cause is rarely a commit. Expect a combination - a prompt revision, a retrieval index rebuild, an upstream model version change, and a shift in the input distribution, none individually sufficient. Resist the pressure to name one.
2. "The model did it" is not a root cause. It is a restatement of the incident. Push through to the design decision that let a probabilistic component take an irreversible action without a check.

The reporting clock

This is no longer purely voluntary practice. Under Article 73 of the EU AI Act, providers of high-risk AI systems must report serious incidents to the market surveillance authority of the member state where the incident occurred, as soon as a causal link to their system is established or reasonably likely. The deadlines are tiered by severity:

Article 73(5) explicitly permits an initial incomplete report with fuller information to follow - which is the mechanism that makes these deadlines survivable. Authorities may order market surveillance measures within days of receiving a report, so the quality of your initial evidence directly shapes what happens next.

A two-day clock is the part worth internalising. Two days is not enough time to recruit, schedule, moderate, transcribe, and synthesise a conventional research study. It is enough time to run an AI-moderated one.

The AI postmortem template

Split action items into mitigative (closes this specific gap) and preventative (addresses the whole class of failure). AI postmortems should almost always generate a third kind: an evaluation item - the failure, converted into permanent test cases and added to your golden set so the regression can never silently return.

Severity classification

Score every confirmed harm on one scale, defined in advance:

The count that matters in the summary is S1 and S2 users, not total requests. Executives and regulators both ask the same first question, and "0.3% of requests" is not an answer to it.

Running the user-evidence study with Koji

The traditional path - recruit affected users, schedule interviews, moderate 12 of them, transcribe, tag, synthesise - takes two to three weeks. That is longer than every deadline in the table above and longer than the patience of any executive during an active incident. It is also why most postmortems quietly substitute a support-ticket sample for real evidence, which biases everything toward the users who complained loudly.

The AI-native approach:

1. Launch within hours, not weeks. Define the exposed population from your incident window, invite a representative sample, and run an always-on AI-moderated study. Participants respond on their own schedule, which removes the scheduling bottleneck entirely - the single largest source of delay in conventional research.

2. Probe what the user actually did. This is the question a survey cannot ask well, because the useful follow-up depends on the answer. Koji's AI moderator asks it: "You said the recommendation looked off - what did you do next?" and then follows the answer wherever it goes. That branch is the difference between knowing an output was wrong and knowing whether it caused harm.

3. Produce a comparable severity distribution. Koji's six structured question types turn testimony into countable evidence:

The yes_no acted-on-it question is the one that converts a quality problem into a harm count - and it is the number your postmortem summary and any regulatory filing both need. See the structured questions guide for how to sequence these without leading participants.

4. Cluster harms automatically. Thematic analysis groups hundreds of open-ended accounts into named harm categories, each linked back to verbatim quotes and specific participants. Manual tagging of 60 transcripts is roughly a week of researcher time; this is the step that makes a real evidence base compatible with a two-day clock.

5. Keep it as a regression study. The same study, re-run after the fix, is your verification that the mitigation worked from the user's side rather than the dashboard's.

Because Koji is AI-native rather than a scheduling layer over human moderators, a 50-participant incident study runs in the window where it can still change the outcome. That is the whole argument: research that arrives after the postmortem is filed is not evidence, it is history.

Sample sizes during an incident

Five mistakes to avoid

1. Reporting requests instead of people. Percentages of traffic conceal concentrated harm on a small, specific group.
2. Sampling only complainants. Support tickets over-represent articulate, high-engagement users and under-represent the ones who silently left.
3. Naming a single root cause. AI incidents are almost always a conjunction. A single named cause usually means the investigation stopped early.
4. Closing without an evaluation item. If the failure did not become a permanent test case, you have licensed its return.
5. Waiting for a complete picture before filing. Article 73(5) exists precisely so you do not have to. File the initial report, then complete it.

Frequently asked questions

What counts as an AI incident? Any deployed model behaviour that causes or nearly causes harm, produces a materially wrong result a user acted on, violates a policy or legal obligation, or degrades a quality metric past an agreed threshold. It does not require an error, an exception, or downtime - most AI incidents return HTTP 200.

How is an AI incident postmortem different from a normal one? Three ways: there is usually no error signal, so detection depends on users rather than alerts; the cause is rarely a single commit but a conjunction of prompt, retrieval, model version, and distribution shift; and the blast radius cannot be read from logs, because whether a wrong output caused harm depends on what the user did with it.

What does blameless actually mean? Assuming everyone involved acted with good intent on the best information available at the time, and directing the investigation at systems rather than individuals. It does not mean no consequences. Its practical purpose is that people escalate early instead of hiding problems.

What are the EU AI Act reporting deadlines for serious incidents? Under Article 73, providers of high-risk systems report to the market surveillance authority of the member state where the incident occurred: 15 days generally, 10 days where a death may have been caused, and 2 days for widespread infringement or serious disruption to critical infrastructure. Article 73(5) permits an initial incomplete report.

How do you scope who was affected when telemetry cannot tell you? You sample. Identify the population plausibly exposed during the incident window, draw a representative sample, and ask them what they saw and what they did next. A structured study of 40-60 affected users characterises the harm distribution far better than any log query.

How does Koji help during an incident? Speed is the binding constraint when a 15-day or 2-day clock is running. Koji launches an AI-moderated study to affected users within hours, probes what each person actually did with the wrong output, produces a comparable severity distribution through structured questions, and clusters responses into named harm categories - evidence in days rather than weeks.

Related Resources

• Root Cause Analysis for Customer Research - the general RCA method this adapts
• The Five Whys Technique - drilling past the first plausible cause
• Human Evaluation of AI Outputs - the ongoing quality practice that catches incidents earlier
• AI Red Teaming with Real Users - finding these failures before they become incidents
• The EU AI Act and User Research - the wider compliance picture
• AI Governance for Customer Research - ISO 42001 and the NIST AI RMF
• Structured Questions in AI Interviews - turning testimony into countable evidence

---

Sources: EU AI Act Article 73 (Reporting of Serious Incidents) and the European Commission draft guidance on serious incident reporting; Google SRE, Postmortem Culture: Learning from Failure; AI Incident Database, Responsible AI Collaborative; 2025 Stack Overflow Developer Survey.