Two Coders Cannot Rescue Categories That Were Too Close Together (2026)

Short answer: the maximum agreement your analysis can reach was decided before anyone read a transcript. It was decided when you wrote the categories. In coding theory, everything a code can repair is governed by one number - its minimum distance, the separation between its two closest codewords - and the governing word is minimum, not average. Your codebook works the same way. If two of your themes sit one judgement call apart, the clarity of the other nineteen is irrelevant, and adding a second coder cannot help: two people trained on the same definitions are not two independent readings of the world, they are two runs of the same decoder.

The first two articles in this sequence were about damage already present in your data and whether you can find it. This one is about where the capacity to find it came from in the first place, and the answer inverts the usual order of operations. Agreement is not something you achieve during analysis. It is something you spend or squander during design.

Minimum distance, and why only the minimum counts

A block code is a set of permitted codewords. Its minimum distance d is the smallest number of positions in which any two of them differ. Everything follows from it:

Note the d = 2 row, because it is the one every research team is already living in. A distance-2 code detects one error and corrects none. That is exactly what double-coding gives you: when two coders disagree you know something is wrong, and you have no principled way to say which of them is right, so you go back to the source or you convene a meeting. Double-coding is not a correction mechanism. It is a detection mechanism, and the coding-theory table said so before anyone ran it.

To get correction - a procedure that tells you the right answer without re-litigating - you need d = 3. Three well-separated categories, not two readings of two adjacent ones.

The arithmetic of the tightest pair

Here is why the word minimum carries the entire argument.

Take a codebook with 6 themes. That is 15 distinct pairs. Suppose you score the separation of each pair on a simple scale - how many independent criteria must change before an excerpt moves from one theme to the other. Say 14 of your pairs score 4, and one troublesome pair scores 1.

• Average separation: (14 x 4 + 1) / 15 = 3.80. That reads like a healthy codebook.
• Minimum distance: 1. Errors detectable: 0. Errors correctable: 0.

Now run three possible interventions and watch which one matters.

1. Sharpen your 14 good pairs from 4 to 5. New average: (14 x 5 + 1) / 15 = 4.73, a big improvement on paper. New minimum distance: still 1. Correction capacity: still 0.
2. Add a third coder. Average unchanged, minimum unchanged, capacity unchanged.
3. Fix only the tight pair, from 1 to 3. New average: (14 x 4 + 3) / 15 = 3.93 - barely moved, a gain of 0.13. New minimum distance: 3. Errors detectable: 2. Errors correctable: 1.

The intervention that barely moves the headline number is the only one that changes what your analysis can do. The intervention that most improves the average accomplishes nothing at all. This is the single most actionable consequence of the minimum-distance framing, and it is invisible to every average-based codebook quality measure.

Your categories have centres and edges

There is a complication that makes the tight pair harder to spot than the arithmetic suggests: categories are not points. They have prototypical members at the core and marginal members at the fringe. Page Laubheimer of Nielsen Norman Group, writing about category outliers in information architecture in October 2021, puts the graded structure precisely:

Noncentral category members are still relatively easily recognizable as members of their categories, but not as easily recognizable as other central members of the same category.

Distance is measured at the boundary, not between the centres

This is where most separation audits go wrong. When you ask whether two themes are distinct, you naturally compare their definitions - which are descriptions of their centres. Two centres can be far apart while their peripheries very nearly touch.

Pricing concerns and perceived value have obviously different centres. A participant saying it is too expensive for what it is sits in the overlap, and it is the overlap, not the centre, that sets d. The correct question is never are these two themes different? It is: what is the excerpt that is hardest to place between them, and how many criteria separate it from either side? If the answer is one, your minimum distance is one, however elegant the definitions read.

What an agreement score is actually measuring

Once the categories are fixed, an agreement statistic reports something narrower than most people believe. Braun and Clarke, citing Yardley, state it plainly:

all coding agreement demonstrates is that coders have been trained to code in the same way not that coding is 'reliable' or 'accurate' with respect to the underlying phenomena that is coded and described

In the language of this sequence: an agreement score measures decoder convergence, not channel accuracy. Two decoders that share a decoding table will agree on a wrong codeword every single time, and their agreement will be perfect. Kappa cannot see the difference between we are both right and we were both trained the same way.

Two coders trained together are one coder

This is the same correlated-residual problem that puts a floor under double data entry, arriving from a different direction. A repetition code only buys you anything when the copies fail independently. Two coders who read the same codebook, sat in the same calibration session, and were corrected by the same lead are strongly correlated decoders. On the clear cases they agree because the cases are clear. On the boundary cases - the only ones where d is being tested - they have been trained into the same tie-breaking habit, so they agree there too, and the agreement is manufactured rather than earned.

The effective minimum distance of two identically trained coders on a boundary case is closer to 1 than to 2. Your kappa will not show this, because kappa is computed from exactly the agreements that the shared training produced.

The capstone: the remedy arrives after the capacity was fixed

Put the sequence together and the ordering is the finding.

Braun and Clarke are insistent that "themes are constructed, created, generated rather than simply emerging" from data. If themes are constructed, then the distance between them is a parameter you chose, not a property of the world you discovered. And the standard remedy for poor agreement - more training, more coders, another calibration round - is applied after that parameter has been set, to a quantity it cannot move.

The reflex when agreement comes back low is to work on the coders. The minimum-distance framing says the coders were the last honest measurement you had: low agreement on a boundary pair is your instrument correctly reporting that d = 1. Training the disagreement away does not raise the distance. It raises the agreement while leaving the distance where it was, which converts a visible problem into an invisible one - a detectable error turned back into an undetectable one, which is precisely the wrong direction.

How to widen the tightest pair

Four moves, in rough order of how often they are the right one.

1. Merge. If you cannot construct an excerpt that clearly belongs to A and not B, they are one theme wearing two names. Merging costs you a row in the report and buys you correction capacity everywhere. In Koji, merging two themes and re-running the analysis is a single operation across the whole corpus.
2. Add a criterion. Introduce an independent dimension on which the two differ - who is speaking, what triggered it, what outcome is being described - so that moving between them requires two changes rather than one.
3. Split off the boundary. Give the contested overlap its own named theme. This is the honest option when the overlap is substantively interesting, and it converts a silent misclassification into a visible category.
4. Define by decision rule, not by description. Replace this theme covers concerns about cost with an ordered test: does the participant name a number? If yes, pricing. If no, and they compare to an alternative, value. Decision rules are testable at the boundary; descriptions are not.

A worked separation audit

Run this before coding, not after. It takes under an hour on a 6-theme codebook.

• List all pairs. For 6 themes that is 15; for 8 it is 28.
• For each pair, write the single hardest excerpt you can imagine that might belong to either.
• Count the criteria that would have to change for it to move. That count is the pair distance.
• Take the minimum across all pairs. That number, not the average, is your codebook.
• If the minimum is 1 or 2, you have no correction capacity, and any agreement statistic you compute afterwards will be reporting your training, not your accuracy. Run the audit again after any codebook revision, which in a Koji project is cheap enough to do more than once.

How Koji handles this

The reason separation audits are rare is not that researchers disagree with them. It is that acting on one has historically meant re-coding everything by hand, and nobody re-codes 60 transcripts to widen a theme pair.

• Re-coding is cheap, so the audit becomes actionable. Koji applies a revised theme definition across the entire corpus in minutes. The intervention that actually raises minimum distance - changing the categories and starting over - stops being the expensive option.
• One definition applied identically to transcript 1 and transcript 60. This removes drift, which is a real and separate error class, and it is worth having on its own terms.
• Grounded items make the boundary visible. Because every item stays attached to its source span, you can pull the excerpts nearest a contested boundary and read them together, which is exactly the input a separation audit needs.
• Structured questions raise distance by construction. A single_choice or ranking response has no boundary to misjudge - the participant did the classifying. Moving a decision from open-ended interpretation into one of the six types in the structured questions guide is the most reliable way to make a tight pair stop mattering.
• Quality scoring separates thin evidence from ambiguous evidence. A 1-5 score across relevance, depth, coverage and completion tells you whether a hard-to-place excerpt is hard because the categories overlap or because the interview was thin, and those two problems have different fixes.

The honest caveat matters here, and it is the same one that applies to any single decoder: running Koji twice on the same genuinely ambiguous excerpt will tend to give the same answer twice, just as two identically trained humans do. Automated consistency does not raise minimum distance and should never be reported as if it did. What automation actually changes is the cost of the one intervention that does raise it. Consistency is the cheap part. Being able to afford to change your categories and re-run is the valuable part.

Common mistakes to avoid

• Reporting average codebook clarity. The average is the wrong statistic. Your codebook is exactly as good as its worst pair, and a strong average can conceal a distance of 1.
• Responding to low agreement with more training. Low agreement on a boundary pair is a correct measurement. Train it away and you have hidden the defect rather than fixed it.
• Treating two coders as two independent readings. Identically trained coders are correlated decoders, and they are most correlated on exactly the cases that test your minimum distance.
• Auditing separation between definitions instead of between edge cases. Definitions describe centres. Distance lives at the boundary, so the audit must be built from the hardest excerpt, not the clearest one.
• Deferring the audit until after coding. By then the capacity is spent. Run the separation pass while the codebook is still cheap to change, which in a Koji project is any time before you rely on the report.

Frequently asked questions

What is minimum distance in the context of a codebook?

It is the separation between your two closest categories - the number of independent criteria that must change before a borderline excerpt moves from one to the other. Coding theory shows that a code with minimum distance d can detect d - 1 errors and correct floor((d-1)/2), so this single number determines what your analysis can catch. The word minimum is load-bearing: your worst pair sets it, regardless of how clear the rest are.

Why does adding a second coder not fix low agreement?

Because two coders is a distance-2 arrangement, which detects one error and corrects none. When they disagree you learn that something is wrong but not which reading is right. Worse, two coders trained on the same codebook by the same lead are correlated rather than independent, and they are most correlated on the boundary cases where the distance is actually being tested.

Should I improve my clearest categories or my worst pair?

Your worst pair, always. Sharpening fourteen already-good pairs from 4 to 5 raises the average from 3.80 to 4.73 and changes correction capacity not at all. Fixing the single tight pair from 1 to 3 raises the average only to 3.93 but takes detectable errors from 0 to 2 and correctable errors from 0 to 1.

What does an inter-rater reliability score actually measure?

Braun and Clarke, citing Yardley, argue that coding agreement demonstrates that coders have been trained to code in the same way, not that the coding is reliable or accurate with respect to the phenomena being described. It is a measure of decoder convergence rather than of accuracy, which is why a high score is compatible with both coders being consistently wrong.

How do I widen two themes that keep getting confused?

Four options, roughly in order of frequency: merge them if you cannot construct an excerpt that clearly belongs to only one; add an independent criterion so that moving between them requires two changes; split the contested overlap into its own named theme; or replace descriptive definitions with an ordered decision rule that is testable at the boundary rather than at the centre.

Does AI analysis solve the category separation problem?

No, and it is worth being precise about why. An AI analyst is a single decoder, so running it twice on a genuinely ambiguous excerpt tends to reproduce the same answer, exactly as two identically trained humans do. What Koji changes is the cost of the only intervention that does raise minimum distance: revising the categories and re-coding the whole corpus, which takes minutes rather than weeks.

Related Resources

• Inter-Rater Reliability in Qualitative Research - how agreement is calculated and interpreted, downstream of the design choice described here.
• How to Build a Qualitative Research Codebook - the practical build process for the categories whose separation this article audits.
• Your Totals Still Add Up - the correlated-residual problem arriving from the data-checking side.
• Missing Answers vs. Wrong Answers - the erasure and error distinction that the same minimum-distance bound governs.
• Thematic Analysis Guide - the full analysis workflow these categories sit inside.
• Structured Questions Guide - the six question types, and how structured answers remove a boundary judgement entirely.