{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-07-31T01:29:19.688Z"},"content":[{"type":"documentation","id":"b0ec6a78-460b-40c8-91f1-3f5526c8acec","slug":"ai-evaluation-dataset-golden-set","title":"Evaluation Datasets for AI Products: How to Build a Golden Set from Real User Research (2026)","url":"https://www.koji.so/docs/ai-evaluation-dataset-golden-set","summary":"A golden evaluation dataset is a product specification written as examples. Covers sizing against confidence intervals (100-500 items per bucket), the four-bucket structure of production sample, edge cases, adversarial items and failure replays, documented label-error rates in published benchmarks, an item schema with decomposed binary acceptance criteria, sourcing those criteria from real users rather than the build team, inter-rater agreement targets, and maintaining the set against distribution drift, criteria drift and overfitting.","content":"## The short answer\n\n**Your evaluation dataset is your product specification, written as examples instead of prose. Most AI evaluation programmes fail at the dataset, not the metric.** A team with a mediocre metric and an excellent golden set will catch real regressions. A team with a sophisticated LLM judge scoring 500 synthetic prompts nobody's users would ever type will catch nothing, confidently.\n\nThe practical build is: **100–500 verified examples**, drawn from four buckets (production traces, adversarial cases, edge cases, and replays of past failures), with acceptance criteria written by people who represent the users who will accept or reject the output — not by the engineers who built it. Below 100 items the confidence interval is too wide to detect the regressions you care about; above 500 the marginal example stops paying for its maintenance cost.\n\nThe part almost everyone gets wrong is the last clause. \"Ground truth\" for a subjective task is not an engineering opinion recorded quickly. It is an empirical claim about what a specific population would accept for a specific job, and establishing it is a user research problem.\n\n## Why the dataset is the hard part\n\nPublished machine-learning benchmarks — datasets built by well-resourced teams over years, and cited thousands of times — are riddled with label errors. In the canonical study, Northcutt, Athalye and Mueller found an **average of at least 3.3% errors across the test sets of ten commonly used benchmarks**, including **6% of the ImageNet validation set (2,916 mislabelled images)** and an estimated **10% (over five million items) in QuickDraw** ([Northcutt et al., 2021, arXiv:2103.14749](https://arxiv.org/abs/2103.14749)).\n\nTwo findings from that paper should change how you build your own set. First, the errors were not benign: on ImageNet with corrected labels, **ResNet-18 outperforms ResNet-50 once the prevalence of originally mislabelled test examples rises by just 6%** — meaning label noise can invert your model-selection decision. Higher-capacity models fit the systematic label errors more faithfully, so a dirty test set actively rewards the wrong model. Second, when candidate errors were flagged algorithmically and then sent to human crowdworkers for validation, **only 51% of flagged candidates were confirmed erroneous** — so automated cleaning cannot be trusted unsupervised either.\n\nIf ImageNet has a 6% error rate, your hand-assembled 200-row spreadsheet is not cleaner. Budget for adjudication from the start.\n\n## Sizing: what your n actually buys you\n\nEval set size is usually argued about aesthetically. It should be argued about statistically. If your current pass rate is around 90%, here is the 95% confidence interval on that estimate by sample size, and the smallest regression you can reliably detect:\n\n| Items in bucket | 95% CI on a 90% pass rate | Smallest detectable regression |\n|---|---|---|\n| 50 | ±8.3 pp | Only catastrophic drops |\n| 100 | ±5.9 pp | ~8–10 pp |\n| 300 | ±3.4 pp | ~5 pp |\n| 500 | ±2.6 pp | ~4 pp |\n| 1,000 | ±1.9 pp | ~3 pp |\n\nThis is why \"100–500 verified examples\" is the standard recommendation rather than a compromise: below 100 your interval swallows any regression short of a disaster, and beyond 500 you are paying maintenance for precision you will not act on. If you genuinely need to detect a 2-point regression, you need thousands of items and should be honest about that cost before promising the gate.\n\nNote also that this arithmetic applies **per bucket and per slice**. A 300-item set that reports one global number is really five 60-item sets if your users split into five meaningful segments, and none of those slices can detect anything.\n\n## The four-bucket structure\n\nA golden set that is only a production sample will tell you the average case is fine while the tail burns. A golden set that is only adversarial cases will tell you the product is broken when it is working. Build four buckets and report them separately.\n\n| Bucket | What it contains | Target share | Sourced from |\n|---|---|---|---|\n| Production sample | Real inputs, sampled to reflect real traffic | ~40% | Logs and traces, PII removed |\n| Edge cases | Rare but legitimate inputs — long inputs, unusual formats, minority segments, non-English | ~25% | Segment analysis + user interviews |\n| Adversarial | Deliberate attempts to elicit failure | ~20% | [Red-team sessions](/docs/ai-red-teaming-with-users) |\n| Failure replays | Every confirmed past failure, permanently | ~15% | Incident reports, support tickets, prior evals |\n\nTwo construction rules matter more than the exact percentages. **Oversample then downsample**: pull roughly five times your target from production traces and downsample deliberately to balance coverage, rather than taking the first 300 rows — raw traffic is dominated by the easy majority case. And **synthetic items extend a production-grounded core; they never replace it.** Synthetic expansion is legitimate for underrepresented scenarios you cannot yet observe. It is illegitimate as the base, because a model evaluated only on inputs another model invented is being graded on its own species' handwriting.\n\n## Where acceptance criteria come from\n\nHere is the step that separates an eval set that predicts user satisfaction from one that predicts nothing. Each item needs an expected behaviour, and for most generative tasks there is no single correct string — there is a space of acceptable outputs and a boundary. That boundary is an empirical fact about users, and you find it by asking them.\n\nA workable item schema:\n\n| Field | Purpose |\n|---|---|\n| `input` | The user's actual request, verbatim |\n| `context` | System state, retrieved documents, prior turns, user segment |\n| `acceptance_criteria` | 3–6 binary criteria that must all hold |\n| `unacceptable_behaviours` | Named failure modes for this item |\n| `severity` | Cost if this item fails, on a fixed scale |\n| `provenance` | Where it came from, and who verified it |\n| `version_added` | For tracking set drift over time |\n\nDecomposing a fuzzy judgement into binary criteria is the highest-leverage move available. \"Is this summary good?\" produces inter-rater agreement in the noise. \"Does it state the decision that was made? Does it name every attendee who spoke? Does it avoid asserting anything not in the transcript? Is it under 200 words?\" produces agreement you can act on, and it converts directly into an automated check. The same rubric-anchoring discipline described in [human evaluation of AI outputs](/docs/human-evaluation-ai-outputs) applies here — the difference is that you are freezing the rubric onto specific items rather than sampling fresh ones.\n\nWhere the criteria come from, in order of reliability:\n\n1. **Interviews with users doing the real job.** Show them real outputs, ask what they would accept, probe why they rejected what they rejected. The rejection reasons are your `unacceptable_behaviours` list.\n2. **Domain experts** for anything with a professional standard — clinical, legal, financial, safety.\n3. **Support tickets and complaints** — the cheapest available record of what users refuse to accept.\n4. **The build team's intuition** — useful for a first draft, never as the final authority.\n\n## The seven-step build\n\n1. **Define the unit.** One item = one input the system must handle, with its context. Ambiguity here poisons everything downstream.\n2. **Pull 5× your target from production.** Strip PII before it leaves the logging system, not after.\n3. **Stratify and downsample** by user segment, task type, input length, and difficulty. Record the strata — you will need them for slice-level reporting.\n4. **Run user sessions to write acceptance criteria.** Structured interviews on a sample of items; generalise the criteria to the rest.\n5. **Have two independent people label the full set**, and measure agreement. Kappa below 0.4 means your criteria are ambiguous — rewrite them, do not add raters. Target 0.7 or above. See [inter-rater reliability](/docs/inter-rater-reliability-qualitative-research).\n6. **Adjudicate disagreements** with a third rater, and treat every adjudication as a bug report against the criteria.\n7. **Freeze, version, and seal a holdout.** Keep 20% sealed and unused for routine gating, to detect the day your team starts optimising against the visible set.\n\n## Maintaining it: drift, decay and overfitting\n\nGolden sets rot in three distinct ways, and each has a different fix.\n\n- **Distribution drift.** Real usage moves; your production bucket stops representing traffic. Fix: re-sample the production bucket quarterly, keeping the failure-replay bucket permanent.\n- **Criteria drift.** Your product's definition of good changes. Fix: version the criteria alongside the items, and never silently rewrite a criterion — a changed criterion invalidates historical comparisons, so record it as a new version.\n- **Overfitting.** The team has seen the set enough times to fix its specific items rather than the underlying behaviour. Fix: the sealed holdout, plus a rule that any item fixed by a special case rather than a general improvement gets a sibling item added.\n\nA useful discipline: your failure-replay bucket should only ever grow. Removing a past failure from the set because \"we fixed that\" is exactly how it comes back.\n\n## Privacy: production traces are personal data\n\nReal user inputs pulled from logs are almost always personal data, and moving them into an evaluation dataset is a new purpose that needs a lawful basis. Three practical requirements: strip identifiers at the logging boundary rather than downstream; check whether your privacy notice and terms actually cover evaluation use; and make sure your deletion process reaches eval sets, which are frequently the copy everyone forgets — see [DSAR and research data](/docs/dsar-research-data). Where you recruit users to write criteria, that is research and needs [informed consent](/docs/research-ethics-guide) like any other study.\n\n## The modern approach: sourcing golden sets with Koji\n\nThe bottleneck in every golden-set project is the same: you need dozens of real users to look at real outputs and tell you, in enough detail to be actionable, what they would and would not accept. Traditionally that means recruiting, scheduling, moderating and coding — weeks of calendar time for what is conceptually an hour of judgement per person.\n\nKoji collapses that into an async study. You send a link with real outputs embedded in the questions; the AI moderator collects the judgement, probes the reasoning behind each rejection in the participant's own words, and returns analysed transcripts. Sixty domain users generating acceptance criteria is a days-long study.\n\nThe [six structured question types](/docs/structured-questions-guide) map onto golden-set construction with unusual precision:\n\n| Question type | Golden-set use |\n|---|---|\n| `yes_no` | The acceptance criteria themselves — each decomposed binary criterion is one question, and the aggregate is a pass rate |\n| `single_choice` | Failure-type classification against your taxonomy |\n| `ranking` | Pairwise and n-way preference between candidate outputs, for items with no single correct answer |\n| `scale` | Severity: what it would cost this user if this output shipped |\n| `multiple_choice` | Which acceptance criteria this specific output violated |\n| `open_ended` | *Why* they rejected it — the reasoning an automated judge structurally cannot produce, and the source of your `unacceptable_behaviours` list |\n\nBecause the answers are typed at capture rather than extracted from prose afterwards, criteria aggregate directly into the dataset schema instead of requiring a coding pass. Koji's thematic analysis clusters the free-text rejection reasons into candidate failure modes; quality scoring rates each conversation 1–5 against the research goal so a participant who clicked through without engaging is visible before their labels enter your ground truth. Voice interviews are the right modality when the output being judged is itself spoken.\n\n| | Traditional golden-set build | Koji |\n|---|---|---|\n| Who writes acceptance criteria | The build team, from intuition | Real users and domain experts |\n| Labellers | Contractors with no domain context | Screened people who do the job |\n| Time to a 300-item labelled set | 4–8 weeks | Days |\n| Rejection reasons captured | Rarely — just a label | Probed, in the participant's words |\n| Refresh cost per quarter | Repeat the whole process | Re-send the link |\n| Slice coverage | Whoever was available | Screened and quota'd by segment |\n\n## Common mistakes\n\n1. **Building the set from synthetic prompts.** Fastest route to an eval that passes while users churn.\n2. **One global pass rate.** Report by bucket and by user slice, or the tail stays invisible.\n3. **Letting the build team write ground truth alone.** They will encode the behaviour they implemented.\n4. **Never measuring label agreement.** An unmeasured set has an unknown error rate, and published benchmarks suggest it is not small.\n5. **Growing the set instead of fixing the criteria.** If raters disagree, more items will not help.\n6. **No sealed holdout.** Within two quarters the visible set becomes a training target rather than a test.\n7. **Forgetting the set in your deletion pipeline.** It is personal data, and it is the copy nobody remembers.\n\n## Frequently asked questions\n\n**How many examples does a golden dataset need?**\nFor most product evals, 100–500 verified examples per bucket. Below 100, the 95% confidence interval on a 90% pass rate is about ±6 points, which means anything short of a catastrophic regression is invisible. Above 500, the marginal example stops justifying its maintenance cost. If you must detect a 2–3 point regression, you need low thousands of items and should scope that honestly.\n\n**Can I use synthetic data to build my eval set?**\nAs an extension, yes — synthetic items are a reasonable way to cover underrepresented scenarios you cannot yet observe in production. As a base, no. A model graded only on inputs another model invented is being tested on a distribution your users do not occupy.\n\n**How often should I refresh the golden set?**\nRe-sample the production bucket quarterly, or whenever a major feature changes what users ask for. The failure-replay bucket should only grow, never shrink. Version any change to acceptance criteria explicitly, because a silent criterion rewrite invalidates every historical comparison.\n\n**What agreement level should labellers reach?**\nTarget Cohen's kappa of 0.7 or above. Between 0.4 and 0.6 the criteria are weak; below 0.4 they are ambiguous. The fix is almost always rewriting the criteria into sharper binary statements, not recruiting more raters.\n\n**Who should write the acceptance criteria?**\nPeople who represent the population that will accept or reject the output in real use — users doing the actual job, plus domain experts where a professional standard applies. The build team's intuition is a good first draft and a bad final authority, because it encodes the behaviour that was implemented rather than the behaviour that is wanted.\n\n**Can an LLM label my golden set for me?**\nIt can propose labels and flag likely errors, and that is a genuine time saver. It should not be the final authority on the set that defines correctness — the set exists precisely to check the models. Note that when label errors were flagged algorithmically in the Northcutt study, only about half were confirmed erroneous by human review.\n\n## Related resources\n\n- [Human Evaluation of AI Outputs](/docs/human-evaluation-ai-outputs) — the rubric, rater and agreement mechanics your labelling pass depends on\n- [LLM-as-a-Judge vs. Human Evaluation](/docs/llm-as-a-judge-vs-human-evaluation) — what to automate once the golden set exists\n- [AI Red Teaming with Real Users](/docs/ai-red-teaming-with-users) — where the adversarial bucket comes from\n- [Structured Questions Guide](/docs/structured-questions-guide) — the six question types that turn user judgement into a dataset schema\n- [Inter-Rater Reliability in Qualitative Research](/docs/inter-rater-reliability-qualitative-research) — measuring whether your criteria are actually unambiguous\n- [User Research for AI Products](/docs/user-research-for-ai-products) — trust calibration and failure tolerance, the qualitative half of AI product research\n- [Synthetic Users in Research](/docs/synthetic-users-research-methodology) — where AI-generated respondents are and are not valid","category":"Research Methods","lastModified":"2026-07-30T03:21:03.629815+00:00","metaTitle":"Building an AI Evaluation Dataset: The Golden Set Guide (2026)","metaDescription":"How to build and maintain an LLM evaluation golden set: sizing, confidence intervals, the four-bucket structure, and sourcing acceptance criteria from users.","keywords":["ai evaluation dataset","golden dataset llm","eval set construction","ground truth dataset ai","llm evaluation data","golden set size","ai regression testing dataset","acceptance criteria ai product","label errors benchmark","building evals from user research"],"aiSummary":"A golden evaluation dataset is a product specification written as examples. Covers sizing against confidence intervals (100-500 items per bucket), the four-bucket structure of production sample, edge cases, adversarial items and failure replays, documented label-error rates in published benchmarks, an item schema with decomposed binary acceptance criteria, sourcing those criteria from real users rather than the build team, inter-rater agreement targets, and maintaining the set against distribution drift, criteria drift and overfitting.","aiPrerequisites":["Basic understanding of how AI product evaluation works","Familiarity with confidence intervals and sampling"],"aiLearningOutcomes":["Size an evaluation set against the smallest regression you need to detect","Structure a golden set into production, edge-case, adversarial and failure-replay buckets","Decompose fuzzy quality judgements into binary acceptance criteria","Source acceptance criteria from real users instead of the build team","Measure and fix labelling disagreement","Maintain the set against distribution drift, criteria drift and overfitting"],"aiDifficulty":"intermediate","aiEstimatedTime":"14 min"}],"pagination":{"total":1,"returned":1,"offset":0}}