{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-08-05T09:57:37.884Z"},"content":[{"type":"documentation","id":"ad37ae4a-5fda-419f-944c-b9c23ca69cb1","slug":"ai-feature-acceptance-criteria","title":"Acceptance Criteria for AI Features: How to Define \"Good Enough to Ship\" (2026)","url":"https://www.koji.so/docs/ai-feature-acceptance-criteria","summary":"Acceptance criteria for AI features must be distributional — a rate over a defined sample at a threshold set by user cost — rather than the binary per-case checks used for deterministic software. Enumerate error classes first (especially confidently-wrong-but-undetectable), set thresholds from what users lose and whether they can detect the error, write criteria in the form \"error class occurs in at most X% of sample Y measured by method Z\", and separate blocker thresholds from targets.","content":"**Answer first:** Acceptance criteria for a deterministic feature are binary — the button works or it doesn't. Acceptance criteria for an AI feature must be **distributional**: a rate, measured over a defined sample, at a threshold set by what each kind of error costs a specific user. \"The model is accurate\" is not a criterion. \"On the 300-case golden set, ≤2% of outputs are confidently wrong in a way a novice user cannot detect, and ≥80% of users complete the task unaided\" is. The threshold is a product judgement grounded in user research, not a data-science default.\n\nTeams that skip this step do not avoid the decision — they make it implicitly, at 5pm on launch day, based on whoever is most confident in the room.\n\n**Gartner** predicted that **at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025**, naming poor data quality, inadequate risk controls, escalating costs, and **unclear business value** among the causes. **MIT's Project NANDA report, *The GenAI Divide: State of AI in Business 2025*** — built on 150 executive interviews, 350 employee surveys, and 300 public deployments — found that despite $30–40bn in enterprise investment, roughly **95% of generative AI pilots produced no measurable P&L impact**, with just 5% extracting real value.\n\nThose two numbers describe the same failure from different angles: an enormous number of AI features get built without anyone having written down, in advance, what would make them worth shipping.\n\n## Why AI acceptance criteria are a different problem\n\n[User acceptance testing](/docs/user-acceptance-testing-guide) verifies that software meets a specification. Every case passes or fails, and the same input gives the same output tomorrow. Three properties of AI features break that model:\n\n| Property | Deterministic feature | AI feature |\n|---|---|---|\n| Same input, same output | Yes | Not guaranteed |\n| Failure is reproducible | Yes | Often not |\n| Test coverage | Enumerable paths | Unbounded input space |\n| A single failing case | Blocks release | Says almost nothing on its own |\n| Passing yesterday means passing today | Yes | Not after a model or prompt change |\n\nThe last row is the one that bites hardest. A model version bump, a prompt refactor, or a provider silently updating a checkpoint can move every rate you measured. Acceptance criteria therefore have to be re-runnable, not a one-time sign-off — which means they have to be cheap enough to run every release.\n\nAnd the fourth row is the one teams argue about. Someone demos a bad output in a review meeting and the launch stalls; someone else demos a great one and it ships. Both anecdotes carry the same evidential weight, which is nearly none. Rates over a fixed sample end that argument.\n\n## Step 1: Enumerate the errors before you set any threshold\n\nYou cannot set a threshold on \"wrong.\" You need error classes, because they have wildly different costs.\n\nGoogle's **People + AI Guidebook** is blunt about this: **\"Weighing the cost of false positives and false negatives is a critical decision that will shape your users' experiences.\"** It warns against the default: *\"It is tempting to weigh both equally by default. However, that's not likely to match the consequences in real life for users.\"* Its illustration is exactly right — *\"is a false alarm worse than one that doesn't go off when there's a fire? Both are incorrect, but one is much more dangerous.\"*\n\nFor most AI product features, the classes that matter are:\n\n1. **Confidently wrong, user-detectable.** Annoying. Costs trust and a bit of time.\n2. **Confidently wrong, user-undetectable.** The dangerous class. The user acts on it. This is where [automation bias](/docs/ai-overreliance-automation-bias-research) lives, and it is the class that deserves the tightest threshold.\n3. **Refusal or non-answer when the request was fine.** The over-blocking cost — see [AI guardrail testing](/docs/ai-guardrail-testing-false-refusals).\n4. **Right but unusable.** Correct output in the wrong format, wrong length, wrong register.\n5. **Slow.** Correct but past the point the user gave up.\n\nClasses 2 and 3 are the two that offline evals systematically under-count, because both are defined by a *user's* reaction, not by a string comparison. Your [FMEA](/docs/ai-failure-mode-analysis) gives you the candidate list; user research gives you the frequencies and the costs.\n\n## Step 2: Set the threshold from user cost, not from a benchmark\n\nThere is no universal quality bar. The People + AI Guidebook makes the point precisely: **\"a recommendations system that's useful 60% of the time could be seen as a failure or a success, depending on the user and the purpose of the system.\"**\n\nFour questions set the number:\n\n**What does the user lose when it's wrong?** Thirty seconds, or a misfiled tax return? Recoverable cost tolerates a loose bar; irreversible cost does not.\n\n**Can the user tell?** If detection is easy and correction is cheap, a 10% error rate can be fine. If the error is invisible, 2% may be too high. This is the single most under-weighted input in most ship decisions.\n\n**What is the alternative they'd use?** The bar is not perfection, it is the status quo. If people currently do this manually with a 15% error rate and it takes 40 minutes, a 7% error rate in 20 seconds is a clear win — but only if error class 2 isn't worse.\n\n**Who eats the cost — the user or you?** A wrong suggestion the user rejects costs them a click. A wrong automated action costs you a support ticket and possibly a refund.\n\nThe Guidebook offers a template worth stealing verbatim for the monitoring half of this:\n\n> \"If **{ specific success metric }** for **{ your team's AI-driven feature }** { drops below/goes above } **{ meaningful threshold }** we will **{ take a specific action }**.\"\n\nFill it in before launch, and the go/no-go conversation has already happened.\n\n## Step 3: Write criteria in the four-part form\n\nA usable AI acceptance criterion has four parts. Miss any one and it is unsignable.\n\n**{ Error class } occurs in ≤ { rate } of { defined sample }, measured by { method }.**\n\nBad: *\"The summariser is accurate.\"*\nBad: *\"No hallucinations.\"* (An absolute on an unbounded input space — unmeasurable, so it will be quietly dropped.)\nGood: *\"On the 300-case golden set, outputs containing a factual claim not supported by the source document occur in ≤3%, as judged by two independent human raters with disagreements adjudicated.\"*\nGood: *\"In moderated task testing (n≥40), ≥80% of participants complete the task without escalating to support, and ≤5% report acting on an output they later discovered was wrong.\"*\nGood: *\"Median time-to-first-token ≤1.5s; p95 ≤4s. Above p95, task abandonment exceeds 20% in observed sessions.\"*\n\nNote that two of those three require people, not a test harness. That is the point.\n\n## Step 4: Build the sample the criteria are measured on\n\nA rate is only as meaningful as the set it is measured over. Two artifacts, and you need both:\n\n**The golden set** — a fixed, versioned collection of representative cases with agreed correct handling. Rates on it are comparable across releases, which is what makes regression detectable. Build it from real user inputs, not invented ones. See [Evaluation Datasets for AI Products](/docs/ai-evaluation-dataset-golden-set) for how to construct one without baking in your own blind spots.\n\n**The user sample** — real people doing real tasks, because classes 2, 3 and 4 are defined by human reaction. Twelve people will not give you a defensible rate; the sample has to be large enough that a percentage means something. This is precisely where traditional research economics break the process: nobody schedules 60 moderated sessions for every release, so the criteria that need users quietly get replaced by criteria that don't.\n\n## Step 5: Separate the ship bar from the aspiration\n\nWrite three columns, not one. Teams that write a single column end up with an aspiration and no ship decision.\n\n| Criterion | Blocker (must hold to ship) | Target (where we want to be) | Monitor (watch, don't block) |\n|---|---|---|---|\n| Undetectable-wrong rate | ≤2% | ≤0.5% | Weekly, by segment |\n| Detectable-wrong rate | ≤12% | ≤5% | Weekly |\n| False refusal rate | ≤5% | ≤2% | Per release |\n| Unaided task completion | ≥75% | ≥90% | Monthly study |\n| p95 latency | ≤4s | ≤2s | Continuous |\n\nOnly the Blocker column is a veto. Everything else is a roadmap. Making that separation explicit is what prevents the launch meeting from relitigating each number under time pressure — and, per Gartner's list, \"unclear business value\" is a governance failure, not a modelling one.\n\n## How Koji helps\n\nThe bottleneck in this whole process is step 4: the criteria that matter most are the ones that require observing people, and observing people has traditionally been too slow to run per release. That is the constraint AI-native research removes.\n\n**Task-based studies at release cadence.** Koji runs AI-moderated interviews continuously and in parallel, so a 60-participant task study takes an afternoon rather than three weeks of scheduling. Run the same study against each release candidate and your unaided-completion criterion becomes a real, repeatable number instead of a hopeful sentence.\n\n**Structured questions produce the rates; open-ended follow-ups explain them.** Koji supports six question types — `open_ended`, `scale`, `single_choice`, `multiple_choice`, `ranking`, and `yes_no` — in one conversational study. A `yes_no` (\"did you have to correct the output?\") gives you the numerator. A `scale` (\"how confident were you the output was right?\") cross-tabbed against actual correctness gives you the detectability estimate that error class 2 depends on. An `open_ended` follow-up tells you *why*, with the AI probing until the answer is specific. See [Structured Questions in AI Interviews](/docs/structured-questions-guide).\n\n**The confidence-versus-correctness cross-tab is the criterion nobody measures.** Ask participants how confident they were, then compare against whether they were right. High confidence plus wrong output is your undetectable-error rate, measured directly. Traditional survey tools like SurveyMonkey give you the confidence rating and nothing to cross it against; a manual interview round gives you the explanation but not enough n to compute a rate. Koji gives you both from the same sessions.\n\n**Quality scoring keeps the sample honest.** Koji scores every interview 1–5 against your research goals, so a rate isn't polluted by half-completed sessions — and only conversations scoring 3 or higher consume credits. See [understanding quality scores](/docs/understanding-quality-scores).\n\n**Automatic thematic analysis and re-runnable reports.** Because the analysis is automated, running the same acceptance study against release candidate 4 costs roughly what it cost against candidate 1. That is the property that turns acceptance criteria from a one-time sign-off into a regression gate.\n\nYou don't need a research team to do this. That is the actual unlock: a PM can define, run, and read an acceptance study in a day.\n\n## Common mistakes\n\n**Absolute criteria.** \"No hallucinations,\" \"zero harmful outputs.\" Unmeasurable on an unbounded input space, so they get dropped silently, and the team ships with no bar at all rather than a loose one.\n\n**One rate for all errors.** Collapsing classes 1–5 into \"accuracy\" hides the only class that can actually hurt someone.\n\n**Benchmark thresholds.** Borrowing 95% from a paper because it sounds rigorous. The paper's users had different costs.\n\n**Measuring only what the harness can measure.** Detectability, refusal-annoyance, and format usability all require humans. If your criteria happen to contain only the machine-measurable ones, that is a signal about your research capacity, not about what matters.\n\n**No re-run plan.** Criteria signed once, never re-measured, then a model provider updates a checkpoint. Tie the study to the release, and see [research refresh cadence](/docs/research-refresh-cadence).\n\n**Skipping the pre-mortem.** Before setting thresholds, run a [product pre-mortem](/docs/product-pre-mortem) — imagining the failed launch surfaces error classes the FMEA missed.\n\n## Frequently asked questions\n\n**How is this different from user acceptance testing?**\nUAT verifies a deterministic system against a specification: each case passes or fails, and results are stable over time. AI features fail probabilistically over an unbounded input space, so a single failing case proves little and a passing suite yesterday does not guarantee one today. AI acceptance criteria are therefore rates over a fixed sample, with thresholds set by user cost, and they must be cheap enough to re-run every release.\n\n**What error rate is acceptable for an AI feature?**\nThere is no universal answer — Google's People + AI Guidebook notes that a system useful 60% of the time can be a success or a failure depending on the user and purpose. Set it from four inputs: what the user loses when it's wrong, whether they can detect the error, what alternative they'd otherwise use, and who bears the cost. Errors users cannot detect deserve a far tighter threshold than errors they can.\n\n**Should I optimise for precision or recall?**\nWhichever error costs your users more. The People + AI Guidebook warns against weighting them equally by default because that rarely matches real consequences — a false alarm and a missed fire are both errors, but one is much more dangerous. Decide by user cost, write it down, and make it an explicit acceptance criterion rather than a data-science default.\n\n**How many participants do I need to measure an acceptance rate?**\nEnough that a percentage is meaningful — far more than the 5–8 typical of discovery work. For a threshold like \"≤5% of users act on a wrong output,\" a sample in the dozens is a practical floor, and more is better. This is exactly why acceptance criteria requiring users get dropped when research is slow and expensive; running the study in an afternoon changes what you can commit to.\n\n**Who owns the acceptance criteria?**\nProduct owns the thresholds because they are judgements about user cost; data science owns the measurement method; research owns the sample and the evidence. If data science sets the thresholds alone you get benchmark numbers disconnected from user consequence; if product sets them alone you get unmeasurable absolutes.\n\n**What if we can't hit the bar?**\nThen you change the product, not the bar. Options include narrowing the intended use, adding a human review step, surfacing uncertainty, making the error easier to detect and cheaper to correct, or shipping to a smaller segment first. Quietly lowering the threshold to fit the model is how the undetectable-error class reaches production.\n\n**Do we need to re-run acceptance criteria after a model upgrade?**\nYes. A model version change, a prompt refactor, or a silent provider-side checkpoint update can move every rate you measured. Treat the criteria as a regression gate tied to the release, which is only sustainable if running the study is cheap.\n\n## Related Resources\n\n- [Structured Questions in AI Interviews](/docs/structured-questions-guide) — the six question types behind the confidence-versus-correctness cross-tab\n- [Evaluation Datasets for AI Products](/docs/ai-evaluation-dataset-golden-set) — building the fixed sample your rates are measured on\n- [AI Failure Mode Analysis](/docs/ai-failure-mode-analysis) — enumerating the error classes before you threshold them\n- [AI Guardrail Testing](/docs/ai-guardrail-testing-false-refusals) — measuring the false-refusal criterion\n- [AI Over-Reliance and Automation Bias](/docs/ai-overreliance-automation-bias-research) — why undetectable errors need the tightest bar\n- [User Acceptance Testing Guide](/docs/user-acceptance-testing-guide) — the deterministic counterpart to this process\n- [Human Evaluation of AI Outputs](/docs/human-evaluation-ai-outputs) — rating methods and inter-rater reliability\n- [The Product Pre-Mortem](/docs/product-pre-mortem) — surfacing error classes before you set thresholds\n\n*Want acceptance criteria backed by real users instead of a benchmark? [Start free with 10 credits](https://www.koji.so) and run your first task study today.*","category":"Research Methods","lastModified":"2026-08-05T03:24:45.421037+00:00","metaTitle":"AI Feature Acceptance Criteria: Defining the Quality Bar to Ship (2026)","metaDescription":"How to write distributional acceptance criteria for AI features, weight false positives against false negatives by user cost, set the ship bar from user evidence, and re-run it every release.","keywords":["ai feature acceptance criteria","quality bar for ai features","ai launch readiness","go/no-go ai launch","release criteria machine learning","definition of done ai feature","precision recall user cost","ai ship decision"],"aiSummary":"Acceptance criteria for AI features must be distributional — a rate over a defined sample at a threshold set by user cost — rather than the binary per-case checks used for deterministic software. Enumerate error classes first (especially confidently-wrong-but-undetectable), set thresholds from what users lose and whether they can detect the error, write criteria in the form \"error class occurs in at most X% of sample Y measured by method Z\", and separate blocker thresholds from targets.","aiPrerequisites":["An AI feature approaching release","Access to users or research participants"],"aiLearningOutcomes":["Explain why AI acceptance criteria must be distributional rather than binary","Enumerate the five error classes and identify which offline evals under-count","Set thresholds from user cost, detectability, and the status-quo alternative","Write acceptance criteria in a measurable four-part form","Separate blocker thresholds from targets and monitors","Build the golden set and user sample that rates are measured over"],"aiDifficulty":"intermediate","aiEstimatedTime":"13 min"}],"pagination":{"total":1,"returned":1,"offset":0}}