Acceptance Criteria for AI Features: How to Define "Good Enough to Ship" (2026)

Answer first: Acceptance criteria for a deterministic feature are binary — the button works or it doesn't. Acceptance criteria for an AI feature must be distributional: a rate, measured over a defined sample, at a threshold set by what each kind of error costs a specific user. "The model is accurate" is not a criterion. "On the 300-case golden set, ≤2% of outputs are confidently wrong in a way a novice user cannot detect, and ≥80% of users complete the task unaided" is. The threshold is a product judgement grounded in user research, not a data-science default.

Teams that skip this step do not avoid the decision — they make it implicitly, at 5pm on launch day, based on whoever is most confident in the room.

Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, naming poor data quality, inadequate risk controls, escalating costs, and unclear business value among the causes. MIT's Project NANDA report, The GenAI Divide: State of AI in Business 2025 — built on 150 executive interviews, 350 employee surveys, and 300 public deployments — found that despite $30–40bn in enterprise investment, roughly 95% of generative AI pilots produced no measurable P&L impact, with just 5% extracting real value.

Those two numbers describe the same failure from different angles: an enormous number of AI features get built without anyone having written down, in advance, what would make them worth shipping.

Why AI acceptance criteria are a different problem

User acceptance testing verifies that software meets a specification. Every case passes or fails, and the same input gives the same output tomorrow. Three properties of AI features break that model:

The last row is the one that bites hardest. A model version bump, a prompt refactor, or a provider silently updating a checkpoint can move every rate you measured. Acceptance criteria therefore have to be re-runnable, not a one-time sign-off — which means they have to be cheap enough to run every release.

And the fourth row is the one teams argue about. Someone demos a bad output in a review meeting and the launch stalls; someone else demos a great one and it ships. Both anecdotes carry the same evidential weight, which is nearly none. Rates over a fixed sample end that argument.

Step 1: Enumerate the errors before you set any threshold

You cannot set a threshold on "wrong." You need error classes, because they have wildly different costs.

Google's People + AI Guidebook is blunt about this: "Weighing the cost of false positives and false negatives is a critical decision that will shape your users' experiences." It warns against the default: "It is tempting to weigh both equally by default. However, that's not likely to match the consequences in real life for users." Its illustration is exactly right — "is a false alarm worse than one that doesn't go off when there's a fire? Both are incorrect, but one is much more dangerous."

For most AI product features, the classes that matter are:

1. Confidently wrong, user-detectable. Annoying. Costs trust and a bit of time.
2. Confidently wrong, user-undetectable. The dangerous class. The user acts on it. This is where automation bias lives, and it is the class that deserves the tightest threshold.
3. Refusal or non-answer when the request was fine. The over-blocking cost — see AI guardrail testing.
4. Right but unusable. Correct output in the wrong format, wrong length, wrong register.
5. Slow. Correct but past the point the user gave up.

Classes 2 and 3 are the two that offline evals systematically under-count, because both are defined by a user's reaction, not by a string comparison. Your FMEA gives you the candidate list; user research gives you the frequencies and the costs.

Step 2: Set the threshold from user cost, not from a benchmark

There is no universal quality bar. The People + AI Guidebook makes the point precisely: "a recommendations system that's useful 60% of the time could be seen as a failure or a success, depending on the user and the purpose of the system."

Four questions set the number:

What does the user lose when it's wrong? Thirty seconds, or a misfiled tax return? Recoverable cost tolerates a loose bar; irreversible cost does not.

Can the user tell? If detection is easy and correction is cheap, a 10% error rate can be fine. If the error is invisible, 2% may be too high. This is the single most under-weighted input in most ship decisions.

What is the alternative they'd use? The bar is not perfection, it is the status quo. If people currently do this manually with a 15% error rate and it takes 40 minutes, a 7% error rate in 20 seconds is a clear win — but only if error class 2 isn't worse.

Who eats the cost — the user or you? A wrong suggestion the user rejects costs them a click. A wrong automated action costs you a support ticket and possibly a refund.

The Guidebook offers a template worth stealing verbatim for the monitoring half of this:

"If { specific success metric } for { your team's AI-driven feature } { drops below/goes above } { meaningful threshold } we will { take a specific action }."

Fill it in before launch, and the go/no-go conversation has already happened.

Step 3: Write criteria in the four-part form

A usable AI acceptance criterion has four parts. Miss any one and it is unsignable.

{ Error class } occurs in ≤ { rate } of { defined sample }, measured by { method }.

Bad: "The summariser is accurate." Bad: "No hallucinations." (An absolute on an unbounded input space — unmeasurable, so it will be quietly dropped.) Good: "On the 300-case golden set, outputs containing a factual claim not supported by the source document occur in ≤3%, as judged by two independent human raters with disagreements adjudicated." Good: "In moderated task testing (n≥40), ≥80% of participants complete the task without escalating to support, and ≤5% report acting on an output they later discovered was wrong." Good: "Median time-to-first-token ≤1.5s; p95 ≤4s. Above p95, task abandonment exceeds 20% in observed sessions."

Note that two of those three require people, not a test harness. That is the point.

Step 4: Build the sample the criteria are measured on

A rate is only as meaningful as the set it is measured over. Two artifacts, and you need both:

The golden set — a fixed, versioned collection of representative cases with agreed correct handling. Rates on it are comparable across releases, which is what makes regression detectable. Build it from real user inputs, not invented ones. See Evaluation Datasets for AI Products for how to construct one without baking in your own blind spots.

The user sample — real people doing real tasks, because classes 2, 3 and 4 are defined by human reaction. Twelve people will not give you a defensible rate; the sample has to be large enough that a percentage means something. This is precisely where traditional research economics break the process: nobody schedules 60 moderated sessions for every release, so the criteria that need users quietly get replaced by criteria that don't.

Step 5: Separate the ship bar from the aspiration

Write three columns, not one. Teams that write a single column end up with an aspiration and no ship decision.

Only the Blocker column is a veto. Everything else is a roadmap. Making that separation explicit is what prevents the launch meeting from relitigating each number under time pressure — and, per Gartner's list, "unclear business value" is a governance failure, not a modelling one.

How Koji helps

The bottleneck in this whole process is step 4: the criteria that matter most are the ones that require observing people, and observing people has traditionally been too slow to run per release. That is the constraint AI-native research removes.

Task-based studies at release cadence. Koji runs AI-moderated interviews continuously and in parallel, so a 60-participant task study takes an afternoon rather than three weeks of scheduling. Run the same study against each release candidate and your unaided-completion criterion becomes a real, repeatable number instead of a hopeful sentence.

Structured questions produce the rates; open-ended follow-ups explain them. Koji supports six question types — open_ended, scale, single_choice, multiple_choice, ranking, and yes_no — in one conversational study. A yes_no ("did you have to correct the output?") gives you the numerator. A scale ("how confident were you the output was right?") cross-tabbed against actual correctness gives you the detectability estimate that error class 2 depends on. An open_ended follow-up tells you why, with the AI probing until the answer is specific. See Structured Questions in AI Interviews.

The confidence-versus-correctness cross-tab is the criterion nobody measures. Ask participants how confident they were, then compare against whether they were right. High confidence plus wrong output is your undetectable-error rate, measured directly. Traditional survey tools like SurveyMonkey give you the confidence rating and nothing to cross it against; a manual interview round gives you the explanation but not enough n to compute a rate. Koji gives you both from the same sessions.

Quality scoring keeps the sample honest. Koji scores every interview 1–5 against your research goals, so a rate isn't polluted by half-completed sessions — and only conversations scoring 3 or higher consume credits. See understanding quality scores.

Automatic thematic analysis and re-runnable reports. Because the analysis is automated, running the same acceptance study against release candidate 4 costs roughly what it cost against candidate 1. That is the property that turns acceptance criteria from a one-time sign-off into a regression gate.

You don't need a research team to do this. That is the actual unlock: a PM can define, run, and read an acceptance study in a day.

Common mistakes

Absolute criteria. "No hallucinations," "zero harmful outputs." Unmeasurable on an unbounded input space, so they get dropped silently, and the team ships with no bar at all rather than a loose one.

One rate for all errors. Collapsing classes 1–5 into "accuracy" hides the only class that can actually hurt someone.

Benchmark thresholds. Borrowing 95% from a paper because it sounds rigorous. The paper's users had different costs.

Measuring only what the harness can measure. Detectability, refusal-annoyance, and format usability all require humans. If your criteria happen to contain only the machine-measurable ones, that is a signal about your research capacity, not about what matters.

No re-run plan. Criteria signed once, never re-measured, then a model provider updates a checkpoint. Tie the study to the release, and see research refresh cadence.

Skipping the pre-mortem. Before setting thresholds, run a product pre-mortem — imagining the failed launch surfaces error classes the FMEA missed.

Frequently asked questions

How is this different from user acceptance testing? UAT verifies a deterministic system against a specification: each case passes or fails, and results are stable over time. AI features fail probabilistically over an unbounded input space, so a single failing case proves little and a passing suite yesterday does not guarantee one today. AI acceptance criteria are therefore rates over a fixed sample, with thresholds set by user cost, and they must be cheap enough to re-run every release.

What error rate is acceptable for an AI feature? There is no universal answer — Google's People + AI Guidebook notes that a system useful 60% of the time can be a success or a failure depending on the user and purpose. Set it from four inputs: what the user loses when it's wrong, whether they can detect the error, what alternative they'd otherwise use, and who bears the cost. Errors users cannot detect deserve a far tighter threshold than errors they can.

Should I optimise for precision or recall? Whichever error costs your users more. The People + AI Guidebook warns against weighting them equally by default because that rarely matches real consequences — a false alarm and a missed fire are both errors, but one is much more dangerous. Decide by user cost, write it down, and make it an explicit acceptance criterion rather than a data-science default.

How many participants do I need to measure an acceptance rate? Enough that a percentage is meaningful — far more than the 5–8 typical of discovery work. For a threshold like "≤5% of users act on a wrong output," a sample in the dozens is a practical floor, and more is better. This is exactly why acceptance criteria requiring users get dropped when research is slow and expensive; running the study in an afternoon changes what you can commit to.

Who owns the acceptance criteria? Product owns the thresholds because they are judgements about user cost; data science owns the measurement method; research owns the sample and the evidence. If data science sets the thresholds alone you get benchmark numbers disconnected from user consequence; if product sets them alone you get unmeasurable absolutes.

What if we can't hit the bar? Then you change the product, not the bar. Options include narrowing the intended use, adding a human review step, surfacing uncertainty, making the error easier to detect and cheaper to correct, or shipping to a smaller segment first. Quietly lowering the threshold to fit the model is how the undetectable-error class reaches production.

Do we need to re-run acceptance criteria after a model upgrade? Yes. A model version change, a prompt refactor, or a silent provider-side checkpoint update can move every rate you measured. Treat the criteria as a regression gate tied to the release, which is only sustainable if running the study is cheap.

Related Resources

• Structured Questions in AI Interviews — the six question types behind the confidence-versus-correctness cross-tab
• Evaluation Datasets for AI Products — building the fixed sample your rates are measured on
• AI Failure Mode Analysis — enumerating the error classes before you threshold them
• AI Guardrail Testing — measuring the false-refusal criterion
• AI Over-Reliance and Automation Bias — why undetectable errors need the tightest bar
• User Acceptance Testing Guide — the deterministic counterpart to this process
• Human Evaluation of AI Outputs — rating methods and inter-rater reliability
• The Product Pre-Mortem — surfacing error classes before you set thresholds

Want acceptance criteria backed by real users instead of a benchmark? Start free with 10 credits and run your first task study today.