Human Evaluation of AI Outputs: The Complete Guide for Product Teams (2026)

The short answer

Human evaluation is the practice of having people score AI outputs against an explicit rubric so you know whether your model is actually good enough to ship. It is the only method that produces ground truth. Automated metrics compare your output to a reference string; benchmarks tell you how a base model performs on someone else's tasks; neither tells you whether your users would accept this answer for their job.

A defensible human evaluation has five parts: a rubric with concrete anchors, a frozen sample of real inputs, at least two independent raters per item, a measured inter-rater agreement score, and blind randomised presentation. Get those right and you have a number you can defend in a launch review. Get them wrong and you have expensive opinions.

The expensive part has always been the why. A rating tells you output 47 scored 2 out of 5; it does not tell you what a rater expected instead, or which failure would have made a real user churn. That is a qualitative research problem, and it is exactly what an AI-native platform like Koji automates — running the rating and the follow-up interview in the same session, then producing thematic analysis of the reasoning across every rater without anyone reading a transcript.

---

What human evaluation is — and what it is not

Three adjacent things get confused constantly:

You need all three. Human evaluation is the narrowest and the most measurable: it takes outputs out of the product, strips the branding, and asks a rater to judge quality against criteria you wrote down in advance.

It is also the layer that most teams skip. Automated metrics are free and instant, so they get run every commit; human evaluation costs money and calendar time, so it gets deferred until a launch goes badly. Stanford-affiliated research on evaluation practice reports that systematic evaluation reduces production failures by up to 60% while enabling roughly 5x faster iteration — the cost of the eval is almost always smaller than the cost of the incident it prevents.

---

Why automated metrics are not enough

Reference-based metrics (BLEU, ROUGE, exact match) assume there is one right answer written down somewhere. For summarisation, drafting, support replies, agent trajectories, and anything conversational, there are hundreds of acceptable answers and the metric punishes the good ones for using different words.

Public benchmarks have the opposite problem: they measure general capability on tasks that are not yours. A model that tops a reasoning leaderboard can still fail badly at "write a refund email in our brand voice that does not promise anything legal will not honour."

Human evaluation is what closes the gap between capable and acceptable for our use case. Everything else is a proxy.

---

Step 1: Write the rubric before you look at any output

The rubric is the study design. Everything downstream — agreement, cost, defensibility — is determined here.

Use concrete anchors, not adjectives. "Helpfulness: 1–5" produces noise. "5 = answers the question, cites the correct policy, and requires no follow-up; 3 = answers the question but omits a condition the customer needs; 1 = wrong, or invents a policy" produces agreement. Rubrics with ambiguous anchors lose discriminative power — every rater centres on 3 and the scores stop separating good from bad.

Decompose multi-dimensional scales into binary criteria. Instead of one 1–5 "quality" score, ask five yes/no questions: is it factually correct against the source? does it follow the format? is the tone on-brand? does it refuse appropriately? is it complete? Recent rubric research finds that breaking a compound Likert item into fine-grained binary criteria materially improves inter-rater agreement, because each rater is judging one thing at a time.

Separate objective from subjective criteria. Factual accuracy and format compliance are checkable and should reach near-perfect agreement. Tone and helpfulness are judgements and will not — that is fine, as long as you report them separately instead of averaging them into one misleading number.

Freeze a calibration set. Pick 15–25 examples, agree on the "correct" score for each as a group, and use them to train every new rater before they touch live items. This is the single cheapest quality control in the whole process.

---

Step 2: Choose the scoring mode

Pairwise is more reliable when the difference is subtle — people are far better at "which of these two" than at "is this a 3 or a 4" — but pairwise results do not give you an absolute quality level, so you cannot use them alone as a ship gate. Most mature teams run pointwise continuously and pairwise at decision points.

---

Step 3: Decide who rates, and what that costs

A standard 200-example evaluation set with three raters per item lands around $300–$1,200 on a crowd panel and considerably more with experts. Building a durable custom evaluation dataset of 5,000–10,000 examples is a $20,000–$50,000 project with $5,000–$15,000 a year of maintenance as your product and model shift.

Two rules that save money: (1) never buy expert ratings for criteria a non-expert can judge — split the rubric and route each criterion to the cheapest competent pool; (2) sample your eval set from real production inputs, stratified across your actual traffic mix, rather than from examples the team wrote.

---

Step 4: Sample size and agreement

How many items? For a ship/no-ship gate on a single quality bar, 150–300 stratified real inputs is the working range: enough to detect a meaningful regression, small enough to re-run weekly. For comparing two models where you expect a small difference, you need more items, not more raters per item.

How many raters per item? Two minimum, three when the criterion is subjective. Below two you cannot compute agreement at all, and an evaluation without an agreement number is an opinion with a spreadsheet.

Agreement targets. Compute Cohen's kappa for two raters, Krippendorff's alpha for three or more. Practitioner guidance converges on these bands:

Aim for kappa ≥ 0.7 between human pairs before you trust the numbers — and before you consider handing any of the scoring to an automated judge. See inter-rater reliability for the calculation mechanics.

---

Step 5: Control the biases you introduce

• Blind the source. Raters must not know which model, prompt version, or vendor produced an output. Knowing produces the result the team hoped for.
• Randomise order per rater. Position effects are real and large in comparison tasks — the same bias that makes automated judges over-prefer whichever answer appears first.
• Counterbalance pairs. In pairwise mode, show A/B to half the raters and B/A to the other half.
• Rotate the calibration check. Insert two known-answer items into every batch to detect rater drift and fatigue.
• Re-bench quarterly. Model behaviour changes underneath you. A rubric validated in January against a frozen labelled set should be re-validated in April.

---

How to run human evaluation with Koji

Traditional tooling forces a split: a spreadsheet or annotation tool collects the scores, and a separate round of interviews — if anyone has time — collects the reasoning. Koji collapses both into one AI-moderated session, which is why the "why" stops being the part that gets cut.

Koji's six structured question types map directly onto evaluation design:

• scale — rubric criteria on a defined 1–5 or 1–7 anchor set, aggregated into distributions rather than averages that hide bimodal disagreement
• yes_no — the binary decomposed criteria that drive high agreement (factually correct? format compliant? appropriate refusal?)
• ranking — pairwise and n-way preference ordering across model or prompt variants
• single_choice — failure-type classification (hallucination / omission / tone / policy breach / formatting)
• multiple_choice — every failure mode present in one output, not just the worst one
• open_ended — the rater's reasoning, where the AI moderator automatically probes: what did you expect instead? what would a customer have done after reading this?

Then the parts that normally take a week happen automatically: every open-ended justification is thematically analysed, failure types are aggregated across raters, and the report shows the distribution per criterion with representative quotes attached. Voice mode is available when you want experts thinking out loud rather than typing, and a customisable AI consultant can carry your rubric and domain context into every session so raters are probed the way a specialist would probe them.

Two practical advantages over manual programmes: you can run 60 raters as easily as six, and re-running the identical study against a new model version is a duplicate-and-launch operation rather than a re-recruitment project. Only conversations that clear Koji's quality bar (a 3+ on its 1–5 interview quality score) consume a credit, so low-effort responses do not silently inflate your eval budget.

---

A worked example: a support-reply assistant

A B2B SaaS team wants to ship an AI draft-reply feature to their support inbox.

1. Sample. 200 real tickets, stratified: 40% billing, 30% troubleshooting, 20% account changes, 10% cancellations.
2. Rubric. Five binary criteria (factually correct against the help centre, no invented policy, correct escalation, on-brand tone, complete) plus one 1–5 overall usefulness scale with written anchors.
3. Raters. Three support agents per item for the factual criteria; two brand/marketing reviewers for tone. Cost: about 15 hours of internal time.
4. Design. Blind to prompt version, randomised order, two calibration items per batch of 25.
5. Run. Delivered as a Koji study — yes_no questions for the binary criteria, scale for usefulness, single_choice for failure type, open_ended for reasoning with automatic AI probing.
6. Result. Overall usefulness averaged 3.8, which alone would have shipped. The distribution was bimodal: 4.4 on billing, 2.6 on cancellations. Failure-type aggregation showed 71% of the low scores were "invented a policy," concentrated in cancellation tickets. Thematic analysis of the open-ended reasoning surfaced the actual cause: the retention-offer rules were not in the retrieval corpus.
7. Decision. Ship for billing and troubleshooting, block cancellations behind a human, re-run the same study in two weeks. Kappa on the factual criteria: 0.81. On tone: 0.52 — reported separately, not averaged in.

The eval took four days end to end. The version of this study that a spreadsheet produces would have reported "3.8, looks fine."

---

Common mistakes

1. Averaging everything into one number. A single quality score hides the bimodal distribution that tells you what to fix. Report per-criterion and per-segment.
2. One rater per item. No agreement measure, no defensibility.
3. Cherry-picked eval sets. Examples written by the team are easier than production traffic, every time.
4. Skipping calibration. Untrained raters disagree about the rubric, not about the outputs.
5. Scores without reasoning. A number tells you there is a problem; only the open-ended follow-up tells you what to change.
6. Never re-running it. Evaluation is a cadence, not a milestone — see research refresh cadence.

---

Frequently asked questions

How many examples do I need for a human evaluation? For a ship gate on one quality bar, 150–300 stratified real production inputs is the practical range. For detecting a small difference between two models, increase the number of items rather than the number of raters per item.

What inter-rater agreement should I target? Cohen's kappa of 0.7 or above between human pairs. Below 0.4 means the rubric is ambiguous and needs rewriting; 0.6–0.8 is acceptable; above 0.8 is strong enough that parts of the scoring can be safely automated.

Can I use an LLM to do the rating instead? Partly, and only after humans have established ground truth. Automated judges match human preferences well on some tasks and carry measurable position, verbosity, and self-preference biases on others. See LLM-as-a-judge vs. human evaluation for the decision framework.

How much does human evaluation cost? Roughly $0.50–$2.00 per rating on a crowd panel, $8–$25 for loaded internal time, and $5–$15 (or $150–$300 an hour) for subject-matter experts. A 200-item set with three raters typically lands between $300 and $1,200 on a crowd panel.

Is human evaluation the same as usability testing? No. Usability testing asks whether a person can complete a task in your interface. Human evaluation asks whether a specific output meets a written quality standard, with the product context deliberately removed.

How do I capture why raters scored something low? Pair every score with an open-ended justification and probe it. Koji's AI moderator asks the follow-up automatically on every response and thematically analyses the reasoning across all raters, so you get failure causes rather than just failure counts.

---

Related resources

• Structured Questions Guide — the six question types that make rubric-based evaluation aggregatable
• LLM-as-a-Judge vs. Human Evaluation — when automated scoring is safe and when it is not
• User Research for AI Products — trust calibration, failure tolerance, and control
• Inter-Rater Reliability in Qualitative Research — computing kappa and alpha
• Can You Trust AI Interviewers? — how Koji constrains hallucination and bias
• Preference Testing Guide — the design-choice sibling of pairwise evaluation
• Research Refresh Cadence — keeping evaluations from going stale