AI Failure Mode Analysis: An FMEA Framework for AI Products (2026)

Answer first: Failure Mode and Effects Analysis (FMEA) is a structured, prospective risk method — you list every way a system can fail, score each failure on severity, occurrence and detection, and fix the highest-priority ones before launch. It works on AI products, but only if you change three things: build your failure list from an AI-specific taxonomy rather than a component tree, score occurrence and detection from real user evidence instead of engineering intuition, and treat "detection" as the hardest score to earn, because AI failures are usually silent. Koji runs the evidence half of that loop: AI-moderated interviews with real users, structured questions that convert straight into occurrence and severity scores, and thematic analysis that surfaces the failure modes nobody put on the list.

What FMEA is, in one paragraph

FMEA came out of aerospace and automotive engineering. You decompose a system into functions, ask "how could each function fail?", and for every failure mode you record three things: what happens to the user (severity), how often it happens (occurrence), and how likely you are to catch it before the user does (detection). Each is scored 1–10. Classic FMEA multiplies them into a Risk Priority Number (RPN = S × O × D, range 1–1000) and works the list top-down.

Note that the automotive industry has already moved past raw RPN. The 2019 AIAG & VDA FMEA Handbook replaced the RPN ranking with an Action Priority (AP) table — High, Medium, Low — precisely because multiplying three ordinal scales produces misleading ties: a severity-10 / occurrence-1 / detection-2 failure (RPN 20) is not really equivalent to a severity-2 / occurrence-2 / detection-5 failure (RPN 20). For AI products this matters even more, because catastrophic-but-rare is the shape of most of your real risk.

Why hardware FMEA breaks on AI products

That last row is the one teams get wrong. In a mechanical FMEA, a detection score of 2 means "we have an automated inspection gate". In an AI FMEA, a detection score of 2 means you have an eval that catches this failure class and evidence that users notice and report it. If neither is true, your detection score is 9 or 10 no matter how good your monitoring dashboard looks.

Step 1: Build the failure list from a taxonomy, not a brainstorm

The single biggest quality difference between a useful AI FMEA and a theatrical one is where the failure list comes from. Brainstorms produce the failures your team already worries about. Taxonomies produce the ones you have not thought of.

Two anchors worth starting from:

• Microsoft's Taxonomy of Failure Mode in Agentic AI Systems. The v2.0 update, published in June 2026 and grounded in twelve months of red team engagements against deployed agentic systems, added seven new categories: agentic supply chain compromise, goal hijacking, inter-agent trust escalation, computer-use-agent visual attacks, session context contamination, MCP/plugin abuse, and capability or architecture disclosure. It splits failures along two axes that are worth copying into your own worksheet: security vs safety, and novel to agentic systems vs materially amplified by them.
• System-level LLM taxonomies such as Failure Modes in LLM Systems (arXiv:2511.19933), which classify failures by where in the pipeline they originate — retrieval, prompt assembly, generation, tool call, post-processing — rather than by symptom. This is the view that tells you which team owns the fix.

The Microsoft red team's most striking operational finding is worth pinning above the worksheet: human-in-the-loop bypass was the most consistently exploited failure mode, at very high frequency. In other words, the mitigation most teams write down in the "detection" column — "a human reviews it" — is itself the most reliably defeated control in the system. Score it accordingly.

Supply-chain exposure is not hypothetical either: 99 CVEs were published for MCP-related software in 2025, and the January 2026 OpenClaw release saw a security audit surface 512 vulnerabilities with more than 1,800 exposed instances leaking credentials in the first week.

Step 2: Score severity from user harm, not from engineering effort

Severity is the one score that should never come from your engineering team. It is a statement about what happens to a person.

Anchoring severity in user consequence is exactly what usability issue severity ratings do for UX defects, and the same discipline applies here. If your severity column is really a story-points column in disguise, the whole FMEA is decoration.

Step 3: Get occurrence and detection from real users

This is where most AI FMEAs quietly fail. Occurrence gets guessed. Detection gets flattered.

Three sources give you defensible numbers:

1. Offline evals give you occurrence on the distribution you thought to test. Your golden evaluation set is the input here.
2. Production telemetry gives you occurrence on the real distribution — but only for failures that leave a trace. Silent failures leave none.
3. User research gives you the two numbers nothing else can: how often users hit the failure, and whether they notice it. That second one is the detection score.

The gap between the second and third source is enormous, and there is a well-known measurement to prove it. In METR's 2025 randomized controlled trial (arXiv:2507.09089), 16 experienced open-source developers worked 246 real tasks with and without AI tooling. They completed tasks 19% slower with AI — and afterwards estimated that AI had made them 20% faster. If a population of expert engineers can be 39 percentage points wrong about their own outcomes, your telemetry-plus-intuition estimate of "users would notice" is not evidence. You have to go and ask.

Step 4: Prioritise with Action Priority, then write mitigations

Rank by Action Priority rather than raw RPN: any severity 9–10 failure is High regardless of occurrence; high-occurrence and high-detection-difficulty combinations come next. Then, for each High item, write a mitigation that changes one specific score, and say which:

• Reduce severity — add an undo, a confirmation step, a scoped permission, a fallback to a deterministic path.
• Reduce occurrence — retrieval grounding, prompt hardening, model change, input validation, guardrails.
• Improve detection — an eval that covers the class, a monitored signal, an in-product report affordance, and a research cadence that catches what the signal misses.

Re-score after mitigation. An FMEA with no post-mitigation column is a list, not an analysis.

The modern approach: how Koji supplies the evidence half

Traditional FMEA facilitation is a two-day workshop with eight people in a room and a spreadsheet, and every occurrence and detection number in it is a guess dressed as a consensus. The AI-native version replaces the guessing with fieldwork you can actually run inside a sprint.

Koji runs the user-evidence half of the analysis:

• AI-moderated interviews at scale. Run 40 interviews with people who use the AI feature in production, in a few days rather than a few weeks, and probe each failure mode with real follow-up questions rather than a checkbox. A human moderator running the same study serially would need a month.
• Structured questions that map directly onto FMEA columns. Koji supports six structured question types — open_ended, scale, single_choice, multiple_choice, ranking, yes_no — and each one has an FMEA job. A scale question gives you a severity distribution. A yes_no question ("did you notice the answer was wrong before you acted on it?") gives you the detection score you cannot get anywhere else. A ranking question tells you which failure users would pay to have fixed first. And every structured answer is followed by AI probing, so you get the number and the reason. See the structured questions guide for how the six types behave.
• Automatic thematic analysis. Themes surface across the whole set as interviews complete, which is how you find the failure modes that were never on the taxonomy — the ones that matter most, because nothing in your eval suite is looking for them.
• Real-time reporting. Occurrence rates, severity distributions and verbatim quotes update as responses land, so the FMEA worksheet can be filled in during the review rather than three weeks after it.
• Customisable AI consultants. Give the interviewer your product's domain context so it can probe a hallucination the way a domain expert would, instead of accepting the first vague answer.

Where a traditional survey tool like SurveyMonkey gives you a frequency count and no explanation, and an eval harness gives you an explanation with no user, an AI-native platform gives you both in the same study — and you do not need a PhD in research methods to run it.

Worked example: an AI support assistant

Note the detection scores. The two failures with detection 8 and 9 are exactly the ones telemetry cannot see, and exactly the ones a 30-interview study resolves in a week.

Where FMEA sits next to your other AI research

FMEA is the prospective method. It complements two others rather than replacing them:

• Red teaming is adversarial — you actively try to make the system misbehave. It populates your failure list.
• Incident postmortems are retrospective — you reconstruct a failure that already reached users. Every postmortem should add a row to the FMEA and, usually, correct a detection score that was too optimistic.
• FMEA is prospective and comprehensive — it forces you to reason about the failures that have not happened yet.

Run all three and the worksheet becomes a living risk register rather than a launch-gate artifact.

Common mistakes

• Scoring detection off your dashboard. If a failure is silent, users do not report it and telemetry does not log it. Detection is 9.
• Letting engineers set severity. Severity is user harm. Ask users.
• Ranking by RPN alone. Use Action Priority so severity-10 items cannot be arithmetic-ed down the list.
• One FMEA per launch. Model versions, prompt changes and new tools all invalidate occurrence scores. Re-run occurrence evidence at least quarterly.
• A failure list built only from brainstorm. Start from a published taxonomy, then let user research add rows.
• No owner per row. An unowned mitigation is a wish.

Frequently asked questions

What is FMEA in the context of AI products? FMEA (Failure Mode and Effects Analysis) is a structured method for listing every way a system can fail, scoring each failure on severity, occurrence and detection, and fixing the highest-priority ones before they reach users. Applied to AI, the failure modes are behaviours — hallucination, over-refusal, goal drift, context loss, tool misuse — rather than broken components, and the scores must be grounded in user evidence because AI failures are probabilistic and often silent.

How is RPN calculated, and should I still use it? RPN = Severity × Occurrence × Detection, each scored 1–10, giving a range of 1–1000. It is still widely taught, but the 2019 AIAG & VDA FMEA Handbook replaced it with an Action Priority table because multiplying three ordinal scales creates false equivalences. For AI products, prefer Action Priority so that rare-but-catastrophic failures cannot be averaged out of the top of your list.

Where do occurrence and detection numbers come from for a brand-new AI feature? Occurrence from your evaluation set for known failure classes, plus production telemetry once you have traffic. Detection almost always has to come from user research — you need to know whether people notice the failure, which no log records. A study of 30–50 users with a yes_no detection question and open-ended probing gives you a defensible score in days.

How is AI FMEA different from red teaming? Red teaming is adversarial and exploratory: skilled testers try to elicit harmful behaviour, which is how you discover failure modes. FMEA is prospective and systematic: you take the full failure list, score every item on user impact and detectability, and prioritise mitigations. Red teaming feeds the FMEA; it does not replace it.

How many failure modes should an AI FMEA contain? For a single AI feature, 20–40 rows is typical; for an agentic system with tool access, 60–100. If you have fewer than 15, you brainstormed instead of working from a taxonomy. If reviewing the list takes more than two hours, split it by subsystem.

How often should we re-run it? Re-score occurrence and detection on every model or prompt change that alters behaviour, and re-run the underlying user evidence at least quarterly. Add a row after every incident postmortem. An FMEA older than one model version is describing a system you no longer ship.

Can Koji run this analysis for us? Koji runs the evidence half — the part that is otherwise guesswork. You define the failure modes; Koji runs AI-moderated interviews with real users to establish how often each failure is hit, how badly it hurts, and whether anyone notices, then returns occurrence and severity distributions with supporting quotes. You keep ownership of scoring and mitigation.

Related Resources

• Structured Questions in AI Interviews — the six question types and how each maps to a research goal
• AI Red Teaming with Real Users — the adversarial method that populates your failure list
• AI Incident Postmortems — the retrospective sibling of FMEA
• Evaluation Datasets for AI Products — building the golden set that supplies occurrence data
• Human Evaluation of AI Outputs — how to score model outputs with people
• Usability Issue Severity Ratings — anchoring severity in user harm
• AI Governance for Customer Research — where FMEA fits in ISO 42001 and the NIST AI RMF