{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-09-24T12:17:28.099Z"},"content":[{"type":"documentation","id":"3c39b69c-ada6-42d0-8271-a21afdb65cdf","slug":"category-separation-coding-agreement","title":"Two Coders Cannot Rescue Categories That Were Too Close Together (2026)","url":"https://www.koji.so/docs/category-separation-coding-agreement","summary":"Maximum achievable coding agreement is fixed when the categories are written, not during analysis. A codes capacity is governed by minimum distance - the separation of the two closest categories - so the average clarity of a codebook is the wrong statistic. Worked case: sharpening 14 of 15 pairs raises the average from 3.80 to 4.73 and correction capacity not at all, while fixing the one tight pair from 1 to 3 raises the average only to 3.93 but takes correctable errors from 0 to 1. Agreement scores measure decoder convergence, not accuracy, because identically trained coders are correlated on exactly the boundary cases that test distance.","content":"**Short answer:** the maximum agreement your analysis can reach was decided before anyone read a transcript. It was decided when you wrote the categories. In coding theory, everything a code can repair is governed by one number - its **minimum distance**, the separation between its two closest codewords - and the governing word is *minimum*, not average. Your codebook works the same way. If two of your themes sit one judgement call apart, the clarity of the other nineteen is irrelevant, and adding a second coder cannot help: two people trained on the same definitions are not two independent readings of the world, they are two runs of the same decoder.\n\nThe first two articles in this sequence were about damage already present in your data and whether you can find it. This one is about where the capacity to find it came from in the first place, and the answer inverts the usual order of operations. Agreement is not something you achieve during analysis. It is something you spend or squander during design.\n\n### Minimum distance, and why only the minimum counts\n\nA block code is a set of permitted codewords. Its minimum distance `d` is the smallest number of positions in which any two of them differ. Everything follows from it:\n\n| Minimum distance d | Errors detectable (d - 1) | Errors correctable floor((d-1)/2) |\n| --- | --- | --- |\n| 1 | 0 | 0 |\n| 2 | 1 | 0 |\n| 3 | 2 | 1 |\n| 4 | 3 | 1 |\n| 5 | 4 | 2 |\n\nNote the d = 2 row, because it is the one every research team is already living in. A distance-2 code detects one error and corrects none. That is **exactly** what double-coding gives you: when two coders disagree you know something is wrong, and you have no principled way to say which of them is right, so you go back to the source or you convene a meeting. Double-coding is not a correction mechanism. It is a detection mechanism, and the coding-theory table said so before anyone ran it.\n\nTo get correction - a procedure that tells you the right answer without re-litigating - you need d = 3. Three well-separated categories, not two readings of two adjacent ones.\n\n### The arithmetic of the tightest pair\n\nHere is why the word *minimum* carries the entire argument.\n\nTake a codebook with 6 themes. That is 15 distinct pairs. Suppose you score the separation of each pair on a simple scale - how many independent criteria must change before an excerpt moves from one theme to the other. Say 14 of your pairs score 4, and one troublesome pair scores 1.\n\n- **Average separation: (14 x 4 + 1) / 15 = 3.80.** That reads like a healthy codebook.\n- **Minimum distance: 1.** Errors detectable: 0. Errors correctable: 0.\n\nNow run three possible interventions and watch which one matters.\n\n1. **Sharpen your 14 good pairs from 4 to 5.** New average: (14 x 5 + 1) / 15 = **4.73**, a big improvement on paper. New minimum distance: **still 1**. Correction capacity: **still 0**.\n2. **Add a third coder.** Average unchanged, minimum unchanged, capacity unchanged.\n3. **Fix only the tight pair, from 1 to 3.** New average: (14 x 4 + 3) / 15 = **3.93** - barely moved, a gain of 0.13. New minimum distance: **3**. Errors detectable: 2. Errors correctable: 1.\n\nThe intervention that barely moves the headline number is the only one that changes what your analysis can do. The intervention that most improves the average accomplishes nothing at all. This is the single most actionable consequence of the minimum-distance framing, and it is invisible to every average-based codebook quality measure.\n\n### Your categories have centres and edges\n\nThere is a complication that makes the tight pair harder to spot than the arithmetic suggests: categories are not points. They have prototypical members at the core and marginal members at the fringe. Page Laubheimer of Nielsen Norman Group, writing about category outliers in information architecture in October 2021, puts the graded structure precisely:\n\n> Noncentral category members are still relatively easily recognizable as members of their categories, but not as easily recognizable as other central members of the same category.\n\n### Distance is measured at the boundary, not between the centres\n\nThis is where most separation audits go wrong. When you ask whether two themes are distinct, you naturally compare their **definitions** - which are descriptions of their centres. Two centres can be far apart while their peripheries very nearly touch.\n\n*Pricing concerns* and *perceived value* have obviously different centres. A participant saying *it is too expensive for what it is* sits in the overlap, and it is the overlap, not the centre, that sets `d`. The correct question is never *are these two themes different?* It is: **what is the excerpt that is hardest to place between them, and how many criteria separate it from either side?** If the answer is one, your minimum distance is one, however elegant the definitions read.\n\n### What an agreement score is actually measuring\n\nOnce the categories are fixed, an agreement statistic reports something narrower than most people believe. Braun and Clarke, citing Yardley, state it plainly:\n\n> all coding agreement demonstrates is that coders have been trained to code in the same way not that coding is 'reliable' or 'accurate' with respect to the underlying phenomena that is coded and described\n\nIn the language of this sequence: an agreement score measures **decoder convergence**, not channel accuracy. Two decoders that share a decoding table will agree on a wrong codeword every single time, and their agreement will be perfect. Kappa cannot see the difference between *we are both right* and *we were both trained the same way*.\n\n### Two coders trained together are one coder\n\nThis is the same correlated-residual problem that puts a floor under double data entry, arriving from a different direction. A repetition code only buys you anything when the copies fail independently. Two coders who read the same codebook, sat in the same calibration session, and were corrected by the same lead are strongly correlated decoders. On the clear cases they agree because the cases are clear. On the boundary cases - the only ones where `d` is being tested - they have been trained into the same tie-breaking habit, so they agree there too, and the agreement is manufactured rather than earned.\n\nThe effective minimum distance of two identically trained coders on a boundary case is closer to 1 than to 2. Your kappa will not show this, because kappa is computed from exactly the agreements that the shared training produced.\n\n### The capstone: the remedy arrives after the capacity was fixed\n\nPut the sequence together and the ordering is the finding.\n\nBraun and Clarke are insistent that \"themes are constructed, created, generated rather than simply emerging\" from data. If themes are constructed, then the distance between them is **a parameter you chose**, not a property of the world you discovered. And the standard remedy for poor agreement - more training, more coders, another calibration round - is applied *after* that parameter has been set, to a quantity it cannot move.\n\nThe reflex when agreement comes back low is to work on the coders. The minimum-distance framing says the coders were the last honest measurement you had: low agreement on a boundary pair is your instrument correctly reporting that `d = 1`. Training the disagreement away does not raise the distance. It raises the agreement while leaving the distance where it was, which converts a visible problem into an invisible one - a detectable error turned back into an undetectable one, which is precisely the wrong direction.\n\n### How to widen the tightest pair\n\nFour moves, in rough order of how often they are the right one.\n\n1. **Merge.** If you cannot construct an excerpt that clearly belongs to A and not B, they are one theme wearing two names. Merging costs you a row in the report and buys you correction capacity everywhere. In Koji, merging two themes and re-running the analysis is a single operation across the whole corpus.\n2. **Add a criterion.** Introduce an independent dimension on which the two differ - who is speaking, what triggered it, what outcome is being described - so that moving between them requires two changes rather than one.\n3. **Split off the boundary.** Give the contested overlap its own named theme. This is the honest option when the overlap is substantively interesting, and it converts a silent misclassification into a visible category.\n4. **Define by decision rule, not by description.** Replace *this theme covers concerns about cost* with an ordered test: *does the participant name a number? If yes, pricing. If no, and they compare to an alternative, value.* Decision rules are testable at the boundary; descriptions are not.\n\n### A worked separation audit\n\nRun this before coding, not after. It takes under an hour on a 6-theme codebook.\n\n- List all pairs. For 6 themes that is 15; for 8 it is 28.\n- For each pair, write the single hardest excerpt you can imagine that might belong to either.\n- Count the criteria that would have to change for it to move. That count is the pair distance.\n- **Take the minimum across all pairs. That number, not the average, is your codebook.**\n- If the minimum is 1 or 2, you have no correction capacity, and any agreement statistic you compute afterwards will be reporting your training, not your accuracy. Run the audit again after any codebook revision, which in a Koji project is cheap enough to do more than once.\n\n## How Koji handles this\n\nThe reason separation audits are rare is not that researchers disagree with them. It is that acting on one has historically meant re-coding everything by hand, and nobody re-codes 60 transcripts to widen a theme pair.\n\n- **Re-coding is cheap, so the audit becomes actionable.** Koji applies a revised theme definition across the entire corpus in minutes. The intervention that actually raises minimum distance - changing the categories and starting over - stops being the expensive option.\n- **One definition applied identically to transcript 1 and transcript 60.** This removes drift, which is a real and separate error class, and it is worth having on its own terms.\n- **Grounded items make the boundary visible.** Because every item stays attached to its source span, you can pull the excerpts nearest a contested boundary and read them together, which is exactly the input a separation audit needs.\n- **Structured questions raise distance by construction.** A `single_choice` or `ranking` response has no boundary to misjudge - the participant did the classifying. Moving a decision from open-ended interpretation into one of the six types in the [structured questions guide](/docs/structured-questions-guide) is the most reliable way to make a tight pair stop mattering.\n- **Quality scoring separates thin evidence from ambiguous evidence.** A 1-5 score across relevance, depth, coverage and completion tells you whether a hard-to-place excerpt is hard because the categories overlap or because the interview was thin, and those two problems have different fixes.\n\nThe honest caveat matters here, and it is the same one that applies to any single decoder: running Koji twice on the same genuinely ambiguous excerpt will tend to give the same answer twice, just as two identically trained humans do. Automated consistency does not raise minimum distance and should never be reported as if it did. What automation actually changes is the **cost of the one intervention that does** raise it. Consistency is the cheap part. Being able to afford to change your categories and re-run is the valuable part.\n\n## Common mistakes to avoid\n\n- **Reporting average codebook clarity.** The average is the wrong statistic. Your codebook is exactly as good as its worst pair, and a strong average can conceal a distance of 1.\n- **Responding to low agreement with more training.** Low agreement on a boundary pair is a correct measurement. Train it away and you have hidden the defect rather than fixed it.\n- **Treating two coders as two independent readings.** Identically trained coders are correlated decoders, and they are most correlated on exactly the cases that test your minimum distance.\n- **Auditing separation between definitions instead of between edge cases.** Definitions describe centres. Distance lives at the boundary, so the audit must be built from the hardest excerpt, not the clearest one.\n- **Deferring the audit until after coding.** By then the capacity is spent. Run the separation pass while the codebook is still cheap to change, which in a Koji project is any time before you rely on the report.\n\n## Frequently asked questions\n\n### What is minimum distance in the context of a codebook?\n\nIt is the separation between your two closest categories - the number of independent criteria that must change before a borderline excerpt moves from one to the other. Coding theory shows that a code with minimum distance d can detect d - 1 errors and correct floor((d-1)/2), so this single number determines what your analysis can catch. The word minimum is load-bearing: your worst pair sets it, regardless of how clear the rest are.\n\n### Why does adding a second coder not fix low agreement?\n\nBecause two coders is a distance-2 arrangement, which detects one error and corrects none. When they disagree you learn that something is wrong but not which reading is right. Worse, two coders trained on the same codebook by the same lead are correlated rather than independent, and they are most correlated on the boundary cases where the distance is actually being tested.\n\n### Should I improve my clearest categories or my worst pair?\n\nYour worst pair, always. Sharpening fourteen already-good pairs from 4 to 5 raises the average from 3.80 to 4.73 and changes correction capacity not at all. Fixing the single tight pair from 1 to 3 raises the average only to 3.93 but takes detectable errors from 0 to 2 and correctable errors from 0 to 1.\n\n### What does an inter-rater reliability score actually measure?\n\nBraun and Clarke, citing Yardley, argue that coding agreement demonstrates that coders have been trained to code in the same way, not that the coding is reliable or accurate with respect to the phenomena being described. It is a measure of decoder convergence rather than of accuracy, which is why a high score is compatible with both coders being consistently wrong.\n\n### How do I widen two themes that keep getting confused?\n\nFour options, roughly in order of frequency: merge them if you cannot construct an excerpt that clearly belongs to only one; add an independent criterion so that moving between them requires two changes; split the contested overlap into its own named theme; or replace descriptive definitions with an ordered decision rule that is testable at the boundary rather than at the centre.\n\n### Does AI analysis solve the category separation problem?\n\nNo, and it is worth being precise about why. An AI analyst is a single decoder, so running it twice on a genuinely ambiguous excerpt tends to reproduce the same answer, exactly as two identically trained humans do. What Koji changes is the cost of the only intervention that does raise minimum distance: revising the categories and re-coding the whole corpus, which takes minutes rather than weeks.\n\n## Related Resources\n\n- [Inter-Rater Reliability in Qualitative Research](/docs/inter-rater-reliability-qualitative-research) - how agreement is calculated and interpreted, downstream of the design choice described here.\n- [How to Build a Qualitative Research Codebook](/docs/qualitative-research-codebook) - the practical build process for the categories whose separation this article audits.\n- [Your Totals Still Add Up](/docs/offsetting-errors-aggregate-checks-research) - the correlated-residual problem arriving from the data-checking side.\n- [Missing Answers vs. Wrong Answers](/docs/missing-answers-vs-wrong-answers-research) - the erasure and error distinction that the same minimum-distance bound governs.\n- [Thematic Analysis Guide](/docs/thematic-analysis-guide) - the full analysis workflow these categories sit inside.\n- [Structured Questions Guide](/docs/structured-questions-guide) - the six question types, and how structured answers remove a boundary judgement entirely.","category":"Analysis & Synthesis","lastModified":"2026-09-24T03:24:53.203191+00:00","metaTitle":"Category Separation: The Cap on Your Coding Agreement (2026)","metaDescription":"Coding agreement is capped by your two closest categories. Why training coders cannot raise it, and the audit that can.","keywords":["category separation coding","overlapping themes","codebook design","coding agreement cap","minimum distance codebook","inter-rater reliability limits","thematic analysis categories","mutually exclusive themes"],"aiSummary":"Maximum achievable coding agreement is fixed when the categories are written, not during analysis. A codes capacity is governed by minimum distance - the separation of the two closest categories - so the average clarity of a codebook is the wrong statistic. Worked case: sharpening 14 of 15 pairs raises the average from 3.80 to 4.73 and correction capacity not at all, while fixing the one tight pair from 1 to 3 raises the average only to 3.93 but takes correctable errors from 0 to 1. Agreement scores measure decoder convergence, not accuracy, because identically trained coders are correlated on exactly the boundary cases that test distance.","aiPrerequisites":["Experience building or applying a qualitative codebook","Familiarity with inter-rater reliability measures such as kappa"],"aiLearningOutcomes":["Compute the minimum distance of a codebook rather than its average clarity","Explain why a second coder detects but cannot correct","Audit category separation at the boundary instead of between definitions","Choose between merging, adding a criterion, splitting, or writing decision rules"],"aiDifficulty":"advanced","aiEstimatedTime":"14 min"}],"pagination":{"total":1,"returned":1,"offset":0}}