{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-08-06T01:25:32.246Z"},"content":[{"type":"documentation","id":"b42892b4-f30a-4125-b6c7-214b66ab957f","slug":"ai-graceful-degradation-fallback-research","title":"Graceful Degradation for AI Features: Researching What Users Experience When the Model Cannot Deliver (2026)","url":"https://www.koji.so/docs/ai-graceful-degradation-fallback-research","summary":"Graceful degradation is what an AI feature does when it cannot deliver: the model is unavailable, too slow, too unsure, or out of scope. Each is a separate design problem needing its own fallback. A degradation ladder runs from full capability through reduced capability, labelled stale output, and deterministic non-AI fallback, down to honest unavailability. Nielsen three response-time limits (0.1s, 1s, 10s) still govern the latency budget, and for streaming AI the binding metric is time-to-first-token. EU AI Act Article 15(4) names backup and fail-safe plans; NIST AI RMF MANAGE 2.4 requires mechanisms to supersede or deactivate underperforming systems. Research the degraded state by staging failures deliberately and recruiting from real incidents.","content":"Graceful degradation is what your AI feature does when it cannot do the thing it promises. The model is down, too slow, too unsure of itself, or simply out of its depth. Most teams design the happy path in detail and let the failure path fall out of whatever the stack trace happened to produce. That is backwards. The degraded state is the one that sets a user expectations for every interaction afterwards, and it is the state your most frustrated users see most often.\n\nThe practical fix is to stop treating \"the AI broke\" as one event. It is four different events with four different user experiences, and each one needs its own designed fallback and its own research. This guide covers how to build a degradation ladder, what regulators now expect of it, and how to put real users in front of the degraded state before your customers find it for you.\n\n## Degradation is not escalation\n\nThese get conflated constantly, and the confusion produces products that have neither.\n\nEscalation routes the user to a human being. It is the right answer when a human is available, the stakes justify the cost, and the user wants one. We cover the design and research of that moment in depth in [the AI-to-human handoff guide](/docs/ai-human-handoff-escalation-research).\n\nGraceful degradation is what happens when there is no human to route to. It is 2am, or the user is on a self-serve plan, or the queue is 40 minutes long, or the failure is an infrastructure outage that a support agent cannot fix either. Degradation is the design of reduced service, not transferred service.\n\nA product that only has escalation has an expensive failure mode. A product that only has degradation has an infuriating one. Mature AI products have both, and they route between them on the basis of what actually broke.\n\n## The four failure classes\n\nTreat these as separate design problems. They present differently to the user, they have different tells, and a fallback that fixes one does nothing for the others.\n\n| Failure class | What triggers it | What users see if you did not design it | What to research |\n| --- | --- | --- | --- |\n| **Unavailable** | Provider outage, rate limit, quota exhaustion, region failure | Spinner, then a generic red error toast, then a lost draft | Whether the users work survives, and whether they trust the feature next week |\n| **Too slow** | Cold start, queue depth, long context, retrieval timeout | An indefinite spinner; the user tabs away and never returns | Where the abandonment threshold actually sits for this task |\n| **Low confidence** | Ambiguous input, thin retrieval, out-of-distribution query | A confident, fluent, wrong answer | Whether users can tell a hedged answer from a certain one, and what they do differently |\n| **Out of scope** | The request is real but the feature was never built for it | A refusal that reads as a malfunction, or a hallucinated attempt | Whether the user understands it is a boundary, not a bug |\n\nThe most dangerous of these is low confidence, because it is the only one where the system does not visibly fail. The other three announce themselves. A low-confidence answer looks exactly like a high-confidence answer unless you deliberately make it look different, which is why it belongs in the same research programme as [over-reliance and automation bias](/docs/ai-overreliance-automation-bias-research).\n\n## The degradation ladder\n\nThe useful mental model is a ladder, not a switch. Each rung delivers less than the one above it and costs the user more, and your job is to fall exactly one rung rather than straight to the bottom.\n\n| Rung | What the system does | What the user loses | Good for |\n| --- | --- | --- | --- |\n| 1. Full capability | The feature works as designed | Nothing | The happy path |\n| 2. Reduced capability | Smaller or faster model, shorter context, fewer features | Quality and nuance | Rate limits, cost spikes, latency pressure |\n| 3. Stale but labelled | Serve the cached or last-known-good result, marked as such | Freshness | Read-heavy features, dashboards, summaries |\n| 4. Deterministic fallback | Drop to the non-AI path: keyword search, template, manual form | Convenience | Anything that had a pre-AI version |\n| 5. Honest unavailability | Say so plainly, preserve the work, give a return path | The task, for now | Genuine hard outages |\n\nRung 4 is the one teams forget, and it is usually the cheapest one they have. If your AI search replaced a keyword search, the keyword search still exists somewhere in the codebase. If your AI summariser replaced a template, the template still renders. Falling back to the boring old thing is almost always better than falling to rung 5, and it is the rung that most reliably keeps the user in the product.\n\nRung 3 has a hard rule attached: stale content must be labelled as stale. An unlabelled cached answer is not degradation, it is a silent correctness bug, and it is the failure mode most likely to end up in [an incident postmortem](/docs/ai-incident-postmortem-user-research).\n\n## The latency budget is older and stricter than you think\n\nThe response-time thresholds that govern degradation are not new AI findings. Nielsen Norman Group still publishes the three limits that Robert Miller established in 1968 and Stuart Card and colleagues confirmed in 1991:\n\n- **0.1 second** is \"the limit for having the user feel that the system is reacting instantaneously, meaning that no special feedback is necessary except to display the result.\"\n- **1.0 second** is \"the limit for the user flow of thought to stay uninterrupted, even though the user will notice the delay.\"\n- **10 seconds** is \"the limit for keeping the user attention focused on the dialogue. For longer delays, users will want to perform other tasks while waiting.\"\n\nJakob Nielsen notes that \"the basic advice regarding response times has been about the same for thirty years.\" That matters, because it means these are properties of human attention, not of 1990s hardware, and no amount of model improvement will move them.\n\nTwo consequences follow directly, and they are the most useful thing on this page for a team shipping a streaming LLM feature.\n\n**The metric that governs the 1-second limit is time-to-first-token, not total response time.** A response that starts streaming in 700ms and finishes in six seconds sits inside the flow-of-thought limit. A response that returns complete in four seconds after four seconds of blank spinner has already broken it. Teams that optimise total latency and ignore TTFT are optimising the wrong number.\n\n**Past 10 seconds, the design problem changes from reassurance to task-switching.** Users are going to leave the tab. A better spinner does not help. What helps is letting them leave properly: background the job, keep the draft, and tell them how they will be notified. Anything over ten seconds needs an exit, not an animation.\n\n## What regulators now expect\n\nDegradation has quietly become a compliance surface, which is new. Two anchors matter.\n\n**EU AI Act Article 15** requires that high-risk AI systems \"achieve an appropriate level of accuracy, robustness, and cybersecurity, and that they perform consistently in those respects throughout their lifecycle.\" Article 15(4) is the operative one for this topic: systems \"shall be as resilient as possible regarding errors, faults or inconsistencies that may occur within the system or the environment in which the system operates, in particular due to their interaction with natural persons or other systems,\" and states that robustness may be achieved through \"technical redundancy solutions, which may include backup or fail-safe plans.\"\n\nRead that carefully. The regulation names backup and fail-safe plans as an acceptable route to compliance. A documented degradation ladder is not merely good practice; for a high-risk system it is close to being the artefact the text is describing. Our [EU AI Act guide for user research](/docs/eu-ai-act-user-research-compliance) covers where the wider obligations land.\n\n**The NIST AI Risk Management Framework** approaches the same ground from the governance side. MANAGE 2.4 requires that mechanisms are in place and applied, with responsibilities assigned and understood, to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use. That is a kill switch requirement, and a kill switch with no designed rung beneath it simply relocates the failure. If you are building the governance case, pair this with [our AI governance frameworks guide](/docs/ai-governance-frameworks-research) and the rollout controls in [staged rollout for AI features](/docs/ai-staged-rollout-user-research).\n\n## The design principle worth internalising\n\nGoogle People and AI Research team put the underlying idea more cleanly than any framework does. In the PAIR Guidebook chapter on errors and graceful failure, they argue that \"the trick isn't to avoid failure, but to find it and make it just as user-centered\" as the rest of the experience, and that \"when an AI system fails, often the easiest path forward is to let the user take over.\"\n\nThe second quote is the one that generates most of rung 4. The fallback you need usually already exists in the form of whatever the user would have done manually. Your job is to hand them the manual path with their context intact, not to invent a new degraded experience from scratch.\n\nThe PAIR framing carries a second implication that teams routinely miss: what counts as an error in an AI system is defined by the user expectation, not by your monitoring. A response that returns HTTP 200 with a fluent, irrelevant answer is a failure. Your error dashboard will show a clean green line through it. This is precisely why degradation cannot be researched from telemetry alone, and why it needs the same treatment as [AI failure mode analysis](/docs/ai-failure-mode-analysis).\n\n## How to research the degraded state\n\nThe degraded state is hard to research for a mundane reason: you cannot reliably wait for it to happen to a participant. You have to induce it.\n\n**Stage the failure deliberately.** Run a session where the feature is forced into each rung. Most teams can do this with the same feature flags they already use for rollout. Force rung 5 for one task, rung 4 for another, rung 3 for a third. What you are looking for is not whether the user liked the message. It is whether they finished the job, what they believed had happened, and what they say they will do the next time.\n\n**Ask about the previous failure, not a hypothetical one.** Users are poor at predicting their reaction to a failure they have not experienced, and good at describing one they have. If your telemetry can identify sessions that actually hit a timeout or an outage, recruit from that list. This is the [critical incident technique](/docs/critical-incident-technique) applied to reliability, and it produces far better data than \"what would you do if the AI stopped working?\"\n\n**Separate the four classes in your instrument.** A single question about \"when the AI does not work\" collapses four different experiences into one unusable average. Ask about each class separately.\n\nThe measures that actually distinguish a good degradation design from a bad one:\n\n1. **Task completion in the degraded state.** Did they finish by another route, or abandon? This is the only measure that matters at rung 4.\n2. **Attribution.** Do they believe the product is broken, the feature is limited, or they did something wrong? Self-blame is the worst outcome and the most common.\n3. **Work preservation.** Did their draft, upload, or context survive? Losing user input during a failure does more trust damage than the failure.\n4. **Return intent.** Will they try the feature again next week? A single unhandled failure can permanently retire a feature in a users mind.\n5. **Detection of the stale or reduced state.** At rungs 2 and 3, did they notice they were getting a lesser answer? If not, your labelling failed.\n6. **Perceived honesty.** Did the product feel candid or evasive? This is what determines whether you get a second chance, and it connects directly to [algorithm aversion and trust repair](/docs/algorithm-aversion-trust-repair).\n\n## Turning that into a study you can actually run\n\nTraditional research tooling makes this study expensive: you need to catch users near a failure, schedule sessions, moderate them, and hand-code the transcripts. By the time you have booked the sessions, the outage is three weeks in the past and the memory has gone soft.\n\nKoji is built for exactly this shape of problem. An AI-moderated interview can be triggered from the event that matters, run itself at 2am while the incident is still fresh, and probe each answer with follow-up questions rather than accepting the first sentence. Where a traditional survey tool would collect \"it was frustrating\" and stop, Koji AI interviewer asks why, asks what the user did next, and asks what they expected the product to do instead.\n\nA practical structure for a degradation study, using Koji six structured question types alongside open-ended conversation:\n\n| Question | Type | What it gives you |\n| --- | --- | --- |\n| \"Walk me through what happened the last time this feature did not work.\" | `open_ended` | The incident narrative, with AI follow-up probing |\n| \"Were you able to finish what you were doing?\" | `yes_no` | Clean completion rate for the degraded state |\n| \"How clear was it what had gone wrong?\" | `scale` | Comparable clarity score across failure classes |\n| \"What did you think had happened?\" | `single_choice` | Attribution: product broken / feature limited / my mistake / network |\n| \"Which of these would have helped most?\" | `multiple_choice` | Demand for each rung of your ladder |\n| \"Rank these in order of how much they would reassure you.\" | `ranking` | Priority ordering for the fallback backlog |\n\nThat mix is the point. The structured questions give you numbers you can track release over release, and the open-ended questions give you the reasoning behind the numbers, all from a single conversation. Because the metrics live in structured fields rather than in free speech, they are also immune to transcription error, which matters more than teams expect on voice studies.\n\nEverything is thematically analysed automatically as responses arrive, so a study that would have meant a week of manual tagging produces a live report you can watch fill up. Trigger the study from a PostHog cohort of users who hit an error, from a Segment audience, or from a Zendesk ticket, and the research runs itself while your team works on the fix.\n\n## Where to start this week\n\nYou do not need a research programme to make progress here. You need one hour and an honest inventory:\n\n1. List every AI feature you ship. For each, write down what happens today at rungs 2 through 5. Most teams discover they have rung 1 and rung 5 and nothing in between.\n2. Find the deterministic fallback that already exists in your codebase. It is usually the pre-AI feature you never deleted.\n3. Measure time-to-first-token, not just p95 total latency.\n4. Check that stale or cached output is labelled.\n5. Run a short study against users who hit a real failure in the last 30 days.\n\nThe teams that do this well are not the ones with the most reliable models. They are the ones whose users never had a reason to stop trusting the feature.\n\n---\n\n**Try it yourself.** Koji gives you 10 free credits when you sign up, which is enough to run a real degradation study with AI-moderated text or voice interviews. No sales call, no seat licence, no research degree required.\n\n## Frequently asked questions\n\n### What is the difference between graceful degradation and escalation to a human?\nEscalation transfers the user to a human being; degradation delivers reduced service when no human is available. A user hitting a provider outage at 2am on a self-serve plan cannot be escalated to anyone, and a support agent cannot fix an infrastructure failure either. Mature AI products design both paths and route between them based on what actually broke: a capability gap escalates, an availability failure degrades.\n\n### What are the five rungs of a degradation ladder?\nFull capability; reduced capability (a smaller or faster model, shorter context); stale but clearly labelled output from cache; deterministic fallback to the non-AI path such as keyword search or a template; and honest unavailability that preserves the user work and gives a return path. The goal is to fall exactly one rung rather than straight to the bottom. Rung four is the one teams forget, and it usually already exists in the codebase as the pre-AI feature nobody deleted.\n\n### What latency counts as too slow for an AI feature?\nNielsen Norman Group publishes three limits established by Miller in 1968 and confirmed by Card and colleagues in 1991: 0.1 second feels instantaneous, 1 second is the limit for uninterrupted flow of thought, and 10 seconds is the limit for holding attention on the task. For a streaming AI response the binding metric is time-to-first-token, not total duration: a reply that starts in 700ms and finishes in six seconds respects the flow limit, while one that returns complete after four seconds of blank spinner does not. Past ten seconds you need a task-switching affordance, not a better spinner.\n\n### Does the EU AI Act require a fallback plan?\nFor high-risk systems, Article 15(1) requires an appropriate level of accuracy, robustness and cybersecurity performed consistently across the lifecycle, and Article 15(4) requires systems to be as resilient as possible regarding errors, faults or inconsistencies, noting that robustness may be achieved through technical redundancy solutions which may include backup or fail-safe plans. The regulation names backup and fail-safe plans directly, so a documented degradation ladder is close to the artefact the text describes. NIST AI RMF MANAGE 2.4 covers the same ground from the governance side.\n\n### Why can I not just measure degradation from telemetry?\nBecause the most dangerous failure class does not register as a failure. A low-confidence answer returns HTTP 200 with fluent text, so your error dashboard shows a clean green line while users act on wrong information. The Google PAIR Guidebook makes the point that what counts as an error in an AI system is defined by the user expectation, not by the stack trace. Telemetry finds outages and timeouts; only research finds the confident wrong answer and the silent stale cache.\n\n### How do I research a failure state that happens unpredictably?\nInduce it rather than wait for it. Use the same feature flags you use for staged rollout to force each rung during a session, and test the four failure classes separately rather than asking one generic question about the AI not working. In parallel, recruit from telemetry: users who actually hit a timeout or outage in the last 30 days give far better data than users asked to imagine one. That is the critical incident technique applied to reliability.\n\n### What should I measure in a degradation study?\nSix things: task completion by another route, attribution (do users blame the product, the feature boundary, or themselves), whether their draft or upload survived, return intent for next week, whether they noticed a reduced or stale answer at all, and perceived honesty. Self-blame and lost work do the most trust damage. In Koji you can capture completion with a `yes_no` question, clarity with a `scale` question, attribution with `single_choice`, and demand for each rung with `multiple_choice` and `ranking`, while `open_ended` questions carry the reasoning behind every number.\n\n## Related Resources\n\n- [Structured Questions in AI Interviews](/docs/structured-questions-guide) - the six question types and when to use each\n- [AI-to-Human Handoff: Designing the Escalation Moment](/docs/ai-human-handoff-escalation-research) - what to do when there is a human to route to\n- [Staged Rollout for AI Features](/docs/ai-staged-rollout-user-research) - shadow mode, canary releases, and kill switches\n- [Algorithm Aversion and Trust Repair](/docs/algorithm-aversion-trust-repair) - winning users back after a failure\n- [AI Failure Mode Analysis](/docs/ai-failure-mode-analysis) - an FMEA framework for AI products\n- [AI Incident Postmortems](/docs/ai-incident-postmortem-user-research) - investigating model failures with user evidence\n- [The EU AI Act and User Research](/docs/eu-ai-act-user-research-compliance) - what the regulation actually requires\n","category":"Research Methods","lastModified":"2026-08-05T03:22:36.243858+00:00","metaTitle":"Graceful Degradation for AI Features: Research the Fallback (2026)","metaDescription":"AI features fail four ways: unavailable, too slow, too unsure, out of scope. Build a degradation ladder, meet EU AI Act Art 15(4), and research the degraded state with real users.","keywords":["graceful degradation ai","ai fallback design","degraded mode ux","ai outage user experience","ai latency time to first token","eu ai act article 15 resilience","ai error recovery research","fallback ux research"],"aiSummary":"Graceful degradation is what an AI feature does when it cannot deliver: the model is unavailable, too slow, too unsure, or out of scope. Each is a separate design problem needing its own fallback. A degradation ladder runs from full capability through reduced capability, labelled stale output, and deterministic non-AI fallback, down to honest unavailability. Nielsen three response-time limits (0.1s, 1s, 10s) still govern the latency budget, and for streaming AI the binding metric is time-to-first-token. EU AI Act Article 15(4) names backup and fail-safe plans; NIST AI RMF MANAGE 2.4 requires mechanisms to supersede or deactivate underperforming systems. Research the degraded state by staging failures deliberately and recruiting from real incidents.","aiPrerequisites":["Basic familiarity with how your AI feature is deployed","Access to feature flags or a staging environment"],"aiLearningOutcomes":["Distinguish graceful degradation from human escalation","Design a five-rung degradation ladder for any AI feature","Apply the three response-time limits to streaming AI responses","Map EU AI Act Article 15(4) and NIST AI RMF MANAGE 2.4 onto fallback design","Run a study that induces failure rather than waiting for it","Measure attribution, work preservation, and return intent"],"aiDifficulty":"intermediate","aiEstimatedTime":"13 min"}],"pagination":{"total":1,"returned":1,"offset":0}}