Reference table
Inter-annotator agreement metrics and when to use each
A comparison of five ways to measure agreement between annotators, from raw percent agreement to Krippendorff's alpha and per-label F1 against a gold set, with when each fits and what it hides. Use it to pick a metric before you report labeling quality on an evaluation or training set. The common bar for alpha is 0.800 for reliable data, with 0.667 to 0.800 for tentative conclusions only.
Where this appears
What's in this reference table
| Metric | Use when | Notes |
|---|---|---|
| Percent agreement | Never as your only number | No chance correction, inflated by class imbalance |
| Cohen's kappa | Exactly two annotators, nominal labels, every item labeled by both | The most widely reported, does not extend to more raters |
| Fleiss' kappa | Fixed number of raters per item, nominal labels | Assumes the same count of raters per item, not the same people |
| Krippendorff's alpha | Any number of raters, missing labels, nominal, ordinal or interval data | The most flexible, and the right default for real annotation pipelines |
| Per-label F1 vs gold | You have a trusted reference set | Shows which label is broken, which agreement metrics cannot |
Use this image
Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:
html
<a href="https://www.koji.so/visuals/inter-annotator-agreement-metrics-compared-table"><img src="https://www.koji.so/visuals/inter-annotator-agreement-metrics-compared-table/inter-annotator-agreement-metrics-compared-table.png" alt="Table comparing percent agreement, Cohen's kappa, Fleiss' kappa, Krippendorff's alpha and per-label F1 by when to use each." width="1200" height="823" /></a>
<p>Source: <a href="https://www.koji.so/visuals/inter-annotator-agreement-metrics-compared-table">Inter-annotator agreement metrics and when to use each</a>, Koji</p>
