Evaluation Datasets for AI Products: How to Build a Golden Set from Real User Research (2026)

The short answer

Your evaluation dataset is your product specification, written as examples instead of prose. Most AI evaluation programmes fail at the dataset, not the metric. A team with a mediocre metric and an excellent golden set will catch real regressions. A team with a sophisticated LLM judge scoring 500 synthetic prompts nobody's users would ever type will catch nothing, confidently.

The practical build is: 100–500 verified examples, drawn from four buckets (production traces, adversarial cases, edge cases, and replays of past failures), with acceptance criteria written by people who represent the users who will accept or reject the output — not by the engineers who built it. Below 100 items the confidence interval is too wide to detect the regressions you care about; above 500 the marginal example stops paying for its maintenance cost.

The part almost everyone gets wrong is the last clause. "Ground truth" for a subjective task is not an engineering opinion recorded quickly. It is an empirical claim about what a specific population would accept for a specific job, and establishing it is a user research problem.

Why the dataset is the hard part

Published machine-learning benchmarks — datasets built by well-resourced teams over years, and cited thousands of times — are riddled with label errors. In the canonical study, Northcutt, Athalye and Mueller found an average of at least 3.3% errors across the test sets of ten commonly used benchmarks, including 6% of the ImageNet validation set (2,916 mislabelled images) and an estimated 10% (over five million items) in QuickDraw (Northcutt et al., 2021, arXiv:2103.14749).

Two findings from that paper should change how you build your own set. First, the errors were not benign: on ImageNet with corrected labels, ResNet-18 outperforms ResNet-50 once the prevalence of originally mislabelled test examples rises by just 6% — meaning label noise can invert your model-selection decision. Higher-capacity models fit the systematic label errors more faithfully, so a dirty test set actively rewards the wrong model. Second, when candidate errors were flagged algorithmically and then sent to human crowdworkers for validation, only 51% of flagged candidates were confirmed erroneous — so automated cleaning cannot be trusted unsupervised either.

If ImageNet has a 6% error rate, your hand-assembled 200-row spreadsheet is not cleaner. Budget for adjudication from the start.

Sizing: what your n actually buys you

Eval set size is usually argued about aesthetically. It should be argued about statistically. If your current pass rate is around 90%, here is the 95% confidence interval on that estimate by sample size, and the smallest regression you can reliably detect:

This is why "100–500 verified examples" is the standard recommendation rather than a compromise: below 100 your interval swallows any regression short of a disaster, and beyond 500 you are paying maintenance for precision you will not act on. If you genuinely need to detect a 2-point regression, you need thousands of items and should be honest about that cost before promising the gate.

Note also that this arithmetic applies per bucket and per slice. A 300-item set that reports one global number is really five 60-item sets if your users split into five meaningful segments, and none of those slices can detect anything.

The four-bucket structure

A golden set that is only a production sample will tell you the average case is fine while the tail burns. A golden set that is only adversarial cases will tell you the product is broken when it is working. Build four buckets and report them separately.

Two construction rules matter more than the exact percentages. Oversample then downsample: pull roughly five times your target from production traces and downsample deliberately to balance coverage, rather than taking the first 300 rows — raw traffic is dominated by the easy majority case. And synthetic items extend a production-grounded core; they never replace it. Synthetic expansion is legitimate for underrepresented scenarios you cannot yet observe. It is illegitimate as the base, because a model evaluated only on inputs another model invented is being graded on its own species' handwriting.

Where acceptance criteria come from

Here is the step that separates an eval set that predicts user satisfaction from one that predicts nothing. Each item needs an expected behaviour, and for most generative tasks there is no single correct string — there is a space of acceptable outputs and a boundary. That boundary is an empirical fact about users, and you find it by asking them.

A workable item schema:

Decomposing a fuzzy judgement into binary criteria is the highest-leverage move available. "Is this summary good?" produces inter-rater agreement in the noise. "Does it state the decision that was made? Does it name every attendee who spoke? Does it avoid asserting anything not in the transcript? Is it under 200 words?" produces agreement you can act on, and it converts directly into an automated check. The same rubric-anchoring discipline described in human evaluation of AI outputs applies here — the difference is that you are freezing the rubric onto specific items rather than sampling fresh ones.

Where the criteria come from, in order of reliability:

1. Interviews with users doing the real job. Show them real outputs, ask what they would accept, probe why they rejected what they rejected. The rejection reasons are your unacceptable_behaviours list.
2. Domain experts for anything with a professional standard — clinical, legal, financial, safety.
3. Support tickets and complaints — the cheapest available record of what users refuse to accept.
4. The build team's intuition — useful for a first draft, never as the final authority.

The seven-step build

1. Define the unit. One item = one input the system must handle, with its context. Ambiguity here poisons everything downstream.
2. Pull 5× your target from production. Strip PII before it leaves the logging system, not after.
3. Stratify and downsample by user segment, task type, input length, and difficulty. Record the strata — you will need them for slice-level reporting.
4. Run user sessions to write acceptance criteria. Structured interviews on a sample of items; generalise the criteria to the rest.
5. Have two independent people label the full set, and measure agreement. Kappa below 0.4 means your criteria are ambiguous — rewrite them, do not add raters. Target 0.7 or above. See inter-rater reliability.
6. Adjudicate disagreements with a third rater, and treat every adjudication as a bug report against the criteria.
7. Freeze, version, and seal a holdout. Keep 20% sealed and unused for routine gating, to detect the day your team starts optimising against the visible set.

Maintaining it: drift, decay and overfitting

Golden sets rot in three distinct ways, and each has a different fix.

• Distribution drift. Real usage moves; your production bucket stops representing traffic. Fix: re-sample the production bucket quarterly, keeping the failure-replay bucket permanent.
• Criteria drift. Your product's definition of good changes. Fix: version the criteria alongside the items, and never silently rewrite a criterion — a changed criterion invalidates historical comparisons, so record it as a new version.
• Overfitting. The team has seen the set enough times to fix its specific items rather than the underlying behaviour. Fix: the sealed holdout, plus a rule that any item fixed by a special case rather than a general improvement gets a sibling item added.

A useful discipline: your failure-replay bucket should only ever grow. Removing a past failure from the set because "we fixed that" is exactly how it comes back.

Privacy: production traces are personal data

Real user inputs pulled from logs are almost always personal data, and moving them into an evaluation dataset is a new purpose that needs a lawful basis. Three practical requirements: strip identifiers at the logging boundary rather than downstream; check whether your privacy notice and terms actually cover evaluation use; and make sure your deletion process reaches eval sets, which are frequently the copy everyone forgets — see DSAR and research data. Where you recruit users to write criteria, that is research and needs informed consent like any other study.

The modern approach: sourcing golden sets with Koji

The bottleneck in every golden-set project is the same: you need dozens of real users to look at real outputs and tell you, in enough detail to be actionable, what they would and would not accept. Traditionally that means recruiting, scheduling, moderating and coding — weeks of calendar time for what is conceptually an hour of judgement per person.

Koji collapses that into an async study. You send a link with real outputs embedded in the questions; the AI moderator collects the judgement, probes the reasoning behind each rejection in the participant's own words, and returns analysed transcripts. Sixty domain users generating acceptance criteria is a days-long study.

The six structured question types map onto golden-set construction with unusual precision:

Because the answers are typed at capture rather than extracted from prose afterwards, criteria aggregate directly into the dataset schema instead of requiring a coding pass. Koji's thematic analysis clusters the free-text rejection reasons into candidate failure modes; quality scoring rates each conversation 1–5 against the research goal so a participant who clicked through without engaging is visible before their labels enter your ground truth. Voice interviews are the right modality when the output being judged is itself spoken.

Common mistakes

1. Building the set from synthetic prompts. Fastest route to an eval that passes while users churn.
2. One global pass rate. Report by bucket and by user slice, or the tail stays invisible.
3. Letting the build team write ground truth alone. They will encode the behaviour they implemented.
4. Never measuring label agreement. An unmeasured set has an unknown error rate, and published benchmarks suggest it is not small.
5. Growing the set instead of fixing the criteria. If raters disagree, more items will not help.
6. No sealed holdout. Within two quarters the visible set becomes a training target rather than a test.
7. Forgetting the set in your deletion pipeline. It is personal data, and it is the copy nobody remembers.

Frequently asked questions

How many examples does a golden dataset need? For most product evals, 100–500 verified examples per bucket. Below 100, the 95% confidence interval on a 90% pass rate is about ±6 points, which means anything short of a catastrophic regression is invisible. Above 500, the marginal example stops justifying its maintenance cost. If you must detect a 2–3 point regression, you need low thousands of items and should scope that honestly.

Can I use synthetic data to build my eval set? As an extension, yes — synthetic items are a reasonable way to cover underrepresented scenarios you cannot yet observe in production. As a base, no. A model graded only on inputs another model invented is being tested on a distribution your users do not occupy.

How often should I refresh the golden set? Re-sample the production bucket quarterly, or whenever a major feature changes what users ask for. The failure-replay bucket should only grow, never shrink. Version any change to acceptance criteria explicitly, because a silent criterion rewrite invalidates every historical comparison.

What agreement level should labellers reach? Target Cohen's kappa of 0.7 or above. Between 0.4 and 0.6 the criteria are weak; below 0.4 they are ambiguous. The fix is almost always rewriting the criteria into sharper binary statements, not recruiting more raters.

Who should write the acceptance criteria? People who represent the population that will accept or reject the output in real use — users doing the actual job, plus domain experts where a professional standard applies. The build team's intuition is a good first draft and a bad final authority, because it encodes the behaviour that was implemented rather than the behaviour that is wanted.

Can an LLM label my golden set for me? It can propose labels and flag likely errors, and that is a genuine time saver. It should not be the final authority on the set that defines correctness — the set exists precisely to check the models. Note that when label errors were flagged algorithmically in the Northcutt study, only about half were confirmed erroneous by human review.

Related resources

• Human Evaluation of AI Outputs — the rubric, rater and agreement mechanics your labelling pass depends on
• LLM-as-a-Judge vs. Human Evaluation — what to automate once the golden set exists
• AI Red Teaming with Real Users — where the adversarial bucket comes from
• Structured Questions Guide — the six question types that turn user judgement into a dataset schema
• Inter-Rater Reliability in Qualitative Research — measuring whether your criteria are actually unambiguous
• User Research for AI Products — trust calibration and failure tolerance, the qualitative half of AI product research
• Synthetic Users in Research — where AI-generated respondents are and are not valid