Framework
The four systems of data annotation quality
Annotation quality broken into four separate systems: guidelines that settle edge cases, an agreement metric that shows whether the task is learnable, gold tasks that catch drift and fraud, and adjudication that turns disagreement into guideline changes. Each one lists the question it answers and how it looks when it fails. It is for teams building evaluation sets, safety taxonomies or training labels for AI products.
Where this appears
What's in this framework
- Guidelines: Answer: what do I do with this ambiguous item? Failing sign: agreement stays low whoever you hire.
- Agreement measurement: Answer: can people learn this task at all? Failing sign: nobody knows whether 80% is good or terrible.
- Gold tasks: Answer: is this annotator still calibrated today? Failing sign: quality decays quietly over weeks.
- Adjudication: Answer: what happens when two good annotators disagree? Failing sign: majority vote hides the hard cases.
Use this image
Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:
html
<a href="https://www.koji.so/visuals/annotation-quality-four-systems-framework"><img src="https://www.koji.so/visuals/annotation-quality-four-systems-framework/annotation-quality-four-systems-framework.png" alt="Framework of the four systems behind annotation quality: guidelines, agreement measurement, gold tasks and adjudication, each with its failing sign." width="1200" height="724" /></a>
<p>Source: <a href="https://www.koji.so/visuals/annotation-quality-four-systems-framework">The four systems of data annotation quality</a>, Koji</p>
