Graceful Degradation for AI Features: Researching What Users Experience When the Model Cannot Deliver (2026)

Graceful degradation is what your AI feature does when it cannot do the thing it promises. The model is down, too slow, too unsure of itself, or simply out of its depth. Most teams design the happy path in detail and let the failure path fall out of whatever the stack trace happened to produce. That is backwards. The degraded state is the one that sets a user expectations for every interaction afterwards, and it is the state your most frustrated users see most often.

The practical fix is to stop treating "the AI broke" as one event. It is four different events with four different user experiences, and each one needs its own designed fallback and its own research. This guide covers how to build a degradation ladder, what regulators now expect of it, and how to put real users in front of the degraded state before your customers find it for you.

Degradation is not escalation

These get conflated constantly, and the confusion produces products that have neither.

Escalation routes the user to a human being. It is the right answer when a human is available, the stakes justify the cost, and the user wants one. We cover the design and research of that moment in depth in the AI-to-human handoff guide.

Graceful degradation is what happens when there is no human to route to. It is 2am, or the user is on a self-serve plan, or the queue is 40 minutes long, or the failure is an infrastructure outage that a support agent cannot fix either. Degradation is the design of reduced service, not transferred service.

A product that only has escalation has an expensive failure mode. A product that only has degradation has an infuriating one. Mature AI products have both, and they route between them on the basis of what actually broke.

The four failure classes

Treat these as separate design problems. They present differently to the user, they have different tells, and a fallback that fixes one does nothing for the others.

The most dangerous of these is low confidence, because it is the only one where the system does not visibly fail. The other three announce themselves. A low-confidence answer looks exactly like a high-confidence answer unless you deliberately make it look different, which is why it belongs in the same research programme as over-reliance and automation bias.

The degradation ladder

The useful mental model is a ladder, not a switch. Each rung delivers less than the one above it and costs the user more, and your job is to fall exactly one rung rather than straight to the bottom.

Rung 4 is the one teams forget, and it is usually the cheapest one they have. If your AI search replaced a keyword search, the keyword search still exists somewhere in the codebase. If your AI summariser replaced a template, the template still renders. Falling back to the boring old thing is almost always better than falling to rung 5, and it is the rung that most reliably keeps the user in the product.

Rung 3 has a hard rule attached: stale content must be labelled as stale. An unlabelled cached answer is not degradation, it is a silent correctness bug, and it is the failure mode most likely to end up in an incident postmortem.

The latency budget is older and stricter than you think

The response-time thresholds that govern degradation are not new AI findings. Nielsen Norman Group still publishes the three limits that Robert Miller established in 1968 and Stuart Card and colleagues confirmed in 1991:

• 0.1 second is "the limit for having the user feel that the system is reacting instantaneously, meaning that no special feedback is necessary except to display the result."
• 1.0 second is "the limit for the user flow of thought to stay uninterrupted, even though the user will notice the delay."
• 10 seconds is "the limit for keeping the user attention focused on the dialogue. For longer delays, users will want to perform other tasks while waiting."

Jakob Nielsen notes that "the basic advice regarding response times has been about the same for thirty years." That matters, because it means these are properties of human attention, not of 1990s hardware, and no amount of model improvement will move them.

Two consequences follow directly, and they are the most useful thing on this page for a team shipping a streaming LLM feature.

The metric that governs the 1-second limit is time-to-first-token, not total response time. A response that starts streaming in 700ms and finishes in six seconds sits inside the flow-of-thought limit. A response that returns complete in four seconds after four seconds of blank spinner has already broken it. Teams that optimise total latency and ignore TTFT are optimising the wrong number.

Past 10 seconds, the design problem changes from reassurance to task-switching. Users are going to leave the tab. A better spinner does not help. What helps is letting them leave properly: background the job, keep the draft, and tell them how they will be notified. Anything over ten seconds needs an exit, not an animation.

What regulators now expect

Degradation has quietly become a compliance surface, which is new. Two anchors matter.

EU AI Act Article 15 requires that high-risk AI systems "achieve an appropriate level of accuracy, robustness, and cybersecurity, and that they perform consistently in those respects throughout their lifecycle." Article 15(4) is the operative one for this topic: systems "shall be as resilient as possible regarding errors, faults or inconsistencies that may occur within the system or the environment in which the system operates, in particular due to their interaction with natural persons or other systems," and states that robustness may be achieved through "technical redundancy solutions, which may include backup or fail-safe plans."

Read that carefully. The regulation names backup and fail-safe plans as an acceptable route to compliance. A documented degradation ladder is not merely good practice; for a high-risk system it is close to being the artefact the text is describing. Our EU AI Act guide for user research covers where the wider obligations land.

The NIST AI Risk Management Framework approaches the same ground from the governance side. MANAGE 2.4 requires that mechanisms are in place and applied, with responsibilities assigned and understood, to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use. That is a kill switch requirement, and a kill switch with no designed rung beneath it simply relocates the failure. If you are building the governance case, pair this with our AI governance frameworks guide and the rollout controls in staged rollout for AI features.

The design principle worth internalising

Google People and AI Research team put the underlying idea more cleanly than any framework does. In the PAIR Guidebook chapter on errors and graceful failure, they argue that "the trick isn't to avoid failure, but to find it and make it just as user-centered" as the rest of the experience, and that "when an AI system fails, often the easiest path forward is to let the user take over."

The second quote is the one that generates most of rung 4. The fallback you need usually already exists in the form of whatever the user would have done manually. Your job is to hand them the manual path with their context intact, not to invent a new degraded experience from scratch.

The PAIR framing carries a second implication that teams routinely miss: what counts as an error in an AI system is defined by the user expectation, not by your monitoring. A response that returns HTTP 200 with a fluent, irrelevant answer is a failure. Your error dashboard will show a clean green line through it. This is precisely why degradation cannot be researched from telemetry alone, and why it needs the same treatment as AI failure mode analysis.

How to research the degraded state

The degraded state is hard to research for a mundane reason: you cannot reliably wait for it to happen to a participant. You have to induce it.

Stage the failure deliberately. Run a session where the feature is forced into each rung. Most teams can do this with the same feature flags they already use for rollout. Force rung 5 for one task, rung 4 for another, rung 3 for a third. What you are looking for is not whether the user liked the message. It is whether they finished the job, what they believed had happened, and what they say they will do the next time.

Ask about the previous failure, not a hypothetical one. Users are poor at predicting their reaction to a failure they have not experienced, and good at describing one they have. If your telemetry can identify sessions that actually hit a timeout or an outage, recruit from that list. This is the critical incident technique applied to reliability, and it produces far better data than "what would you do if the AI stopped working?"

Separate the four classes in your instrument. A single question about "when the AI does not work" collapses four different experiences into one unusable average. Ask about each class separately.

The measures that actually distinguish a good degradation design from a bad one:

1. Task completion in the degraded state. Did they finish by another route, or abandon? This is the only measure that matters at rung 4.
2. Attribution. Do they believe the product is broken, the feature is limited, or they did something wrong? Self-blame is the worst outcome and the most common.
3. Work preservation. Did their draft, upload, or context survive? Losing user input during a failure does more trust damage than the failure.
4. Return intent. Will they try the feature again next week? A single unhandled failure can permanently retire a feature in a users mind.
5. Detection of the stale or reduced state. At rungs 2 and 3, did they notice they were getting a lesser answer? If not, your labelling failed.
6. Perceived honesty. Did the product feel candid or evasive? This is what determines whether you get a second chance, and it connects directly to algorithm aversion and trust repair.

Turning that into a study you can actually run

Traditional research tooling makes this study expensive: you need to catch users near a failure, schedule sessions, moderate them, and hand-code the transcripts. By the time you have booked the sessions, the outage is three weeks in the past and the memory has gone soft.

Koji is built for exactly this shape of problem. An AI-moderated interview can be triggered from the event that matters, run itself at 2am while the incident is still fresh, and probe each answer with follow-up questions rather than accepting the first sentence. Where a traditional survey tool would collect "it was frustrating" and stop, Koji AI interviewer asks why, asks what the user did next, and asks what they expected the product to do instead.

A practical structure for a degradation study, using Koji six structured question types alongside open-ended conversation:

That mix is the point. The structured questions give you numbers you can track release over release, and the open-ended questions give you the reasoning behind the numbers, all from a single conversation. Because the metrics live in structured fields rather than in free speech, they are also immune to transcription error, which matters more than teams expect on voice studies.

Everything is thematically analysed automatically as responses arrive, so a study that would have meant a week of manual tagging produces a live report you can watch fill up. Trigger the study from a PostHog cohort of users who hit an error, from a Segment audience, or from a Zendesk ticket, and the research runs itself while your team works on the fix.

Where to start this week

You do not need a research programme to make progress here. You need one hour and an honest inventory:

1. List every AI feature you ship. For each, write down what happens today at rungs 2 through 5. Most teams discover they have rung 1 and rung 5 and nothing in between.
2. Find the deterministic fallback that already exists in your codebase. It is usually the pre-AI feature you never deleted.
3. Measure time-to-first-token, not just p95 total latency.
4. Check that stale or cached output is labelled.
5. Run a short study against users who hit a real failure in the last 30 days.

The teams that do this well are not the ones with the most reliable models. They are the ones whose users never had a reason to stop trusting the feature.

---

Try it yourself. Koji gives you 10 free credits when you sign up, which is enough to run a real degradation study with AI-moderated text or voice interviews. No sales call, no seat licence, no research degree required.

Frequently asked questions

What is the difference between graceful degradation and escalation to a human?

Escalation transfers the user to a human being; degradation delivers reduced service when no human is available. A user hitting a provider outage at 2am on a self-serve plan cannot be escalated to anyone, and a support agent cannot fix an infrastructure failure either. Mature AI products design both paths and route between them based on what actually broke: a capability gap escalates, an availability failure degrades.

What are the five rungs of a degradation ladder?

Full capability; reduced capability (a smaller or faster model, shorter context); stale but clearly labelled output from cache; deterministic fallback to the non-AI path such as keyword search or a template; and honest unavailability that preserves the user work and gives a return path. The goal is to fall exactly one rung rather than straight to the bottom. Rung four is the one teams forget, and it usually already exists in the codebase as the pre-AI feature nobody deleted.

What latency counts as too slow for an AI feature?

Nielsen Norman Group publishes three limits established by Miller in 1968 and confirmed by Card and colleagues in 1991: 0.1 second feels instantaneous, 1 second is the limit for uninterrupted flow of thought, and 10 seconds is the limit for holding attention on the task. For a streaming AI response the binding metric is time-to-first-token, not total duration: a reply that starts in 700ms and finishes in six seconds respects the flow limit, while one that returns complete after four seconds of blank spinner does not. Past ten seconds you need a task-switching affordance, not a better spinner.

Does the EU AI Act require a fallback plan?

For high-risk systems, Article 15(1) requires an appropriate level of accuracy, robustness and cybersecurity performed consistently across the lifecycle, and Article 15(4) requires systems to be as resilient as possible regarding errors, faults or inconsistencies, noting that robustness may be achieved through technical redundancy solutions which may include backup or fail-safe plans. The regulation names backup and fail-safe plans directly, so a documented degradation ladder is close to the artefact the text describes. NIST AI RMF MANAGE 2.4 covers the same ground from the governance side.

Why can I not just measure degradation from telemetry?

Because the most dangerous failure class does not register as a failure. A low-confidence answer returns HTTP 200 with fluent text, so your error dashboard shows a clean green line while users act on wrong information. The Google PAIR Guidebook makes the point that what counts as an error in an AI system is defined by the user expectation, not by the stack trace. Telemetry finds outages and timeouts; only research finds the confident wrong answer and the silent stale cache.

How do I research a failure state that happens unpredictably?

Induce it rather than wait for it. Use the same feature flags you use for staged rollout to force each rung during a session, and test the four failure classes separately rather than asking one generic question about the AI not working. In parallel, recruit from telemetry: users who actually hit a timeout or outage in the last 30 days give far better data than users asked to imagine one. That is the critical incident technique applied to reliability.

What should I measure in a degradation study?

Six things: task completion by another route, attribution (do users blame the product, the feature boundary, or themselves), whether their draft or upload survived, return intent for next week, whether they noticed a reduced or stale answer at all, and perceived honesty. Self-blame and lost work do the most trust damage. In Koji you can capture completion with a yes_no question, clarity with a scale question, attribution with single_choice, and demand for each rung with multiple_choice and ranking, while open_ended questions carry the reasoning behind every number.

Related Resources

• Structured Questions in AI Interviews - the six question types and when to use each
• AI-to-Human Handoff: Designing the Escalation Moment - what to do when there is a human to route to
• Staged Rollout for AI Features - shadow mode, canary releases, and kill switches
• Algorithm Aversion and Trust Repair - winning users back after a failure
• AI Failure Mode Analysis - an FMEA framework for AI products
• AI Incident Postmortems - investigating model failures with user evidence
• The EU AI Act and User Research - what the regulation actually requires