You Cannot Spot-Check Your Way to Data Quality: The All-or-None Rule for Research QA

Bottom line up front: The reflexive fix for a research process that keeps letting bad data through is a spot-check: review a handful of transcripts, sample a few coded excerpts, sign off. Quality engineering has known since the 1950s that the sample review is the one option that is almost never the right one. A spot-check of ten items detects a 5 percent defect rate only 40.1 percent of the time. Deming's all-or-none rule says the cost-minimising policy is a corner solution: inspect nothing, or inspect everything, depending on whether your defect rate is above or below the ratio of the cost of one check to the cost of one escape. Sampling sits at the optimum essentially never.

The move everyone makes, and the number that kills it

A team measures its research pipeline yield, finds it uncomfortable, and does the obvious thing: adds a review gate. Someone will check a sample of transcripts. Someone will spot-check the coding. The reasoning feels unimpeachable. Checking some is better than checking none, and checking all is too expensive.

Both halves of that sentence are wrong in a way that is exactly computable.

Here is what a single-sample plan actually detects. Each cell is the probability that the plan accepts the batch, given the true defect rate. A plan of n=10, c=0 means "review ten items and reject the batch if you find one or more defects" - the strictest spot-check most teams would ever run.

Computed from the binomial distribution; each cell is the probability of observing at most c defects in n draws at the stated defect rate.

Read the top row. A batch of interviews that is 5 percent junk sails through your strict ten-item review 59.9 percent of the time. A batch that is 10 percent junk sails through 34.9 percent of the time. The review is not catching a 10 percent defect rate; it is catching a coin flip's worth of a 10 percent defect rate, and every time it passes, someone writes "reviewed" in a status column and the batch is treated as clean.

Turn the same arithmetic the other way: with a 5 percent defect rate, a ten-item spot-check finds at least one defect only 40.1 percent of the time. At 10 percent it finds one 65.1 percent of the time. You would need to be at a 20 percent defect rate before a ten-item check is more likely than not to be genuinely informative, and at 20 percent you did not need a check to know you had a problem.

The vocabulary that makes this precise

The field that owns this problem is acceptance sampling, and its terms are worth borrowing because they name things research teams argue about without labels. The NIST Engineering Statistics Handbook defines them cleanly:

• Operating characteristic curve. "This curve plots the probability of accepting the lot (Y-axis) versus the lot fraction or percent defectives (X-axis). The OC curve is the primary tool for displaying and investigating the properties of a LASP." Every review policy has an OC curve whether or not anyone has drawn it. The table above is one.
• Acceptable quality level. "The AQL is a percent defective that is the base line requirement for the quality of the producer's product." Your research team has an implicit AQL. It has probably never been said out loud.
• Lot tolerance percent defective. "The LTPD is a designated high defect level that would be unacceptable to the consumer."
• Producer's risk. "the probability, for a given (n,c) sampling plan, of rejecting a lot that has a defect level equal to the AQL." This is the cost nobody counts: the good batch you throw away and re-field.
• Consumer's risk. "the probability, for a given (n,c) sampling plan, of accepting a lot with a defect level equal to the LTPD."

The two risks are the point. A review gate has a false-reject rate as well as a false-accept rate, and the false rejects are expensive in research because re-fielding a study costs weeks. A review step is not a free filter bolted onto the side of the process. It is another stage in the series, with its own yield, and it can lower the total.

Deming's all-or-none rule

W. Edwards Deming derived the condition under which inspecting at all is worth it, in Chapter 15 of Out of the Crisis (MIT CAES, 1986). It is usually called the kp rule or the all-or-none rule, and it is startlingly simple.

Let k1 be the cost of inspecting one item and k2 be the cost of letting one defective item through to be discovered downstream. Let p be the incoming fraction defective. Then:

• If p is less than k1/k2, minimum average cost occurs with no inspection.
• If p is greater than k1/k2, minimum average cost occurs with 100 percent inspection.

There is no middle. Sampling inspection is optimal only at the exact break-even point, which is a measure-zero coincidence. Every intermediate policy - review a fifth, review a sample, review the ones that look odd - is dominated by one of the two corners.

For a research team, the ratio is easy to estimate:

Break-even is k1/k2 by construction.

Now put real research numbers in. Reading one transcript carefully costs, say, fifteen minutes. A bad interview that reaches synthesis and lands in a readout costs a wrong feature decision, or at minimum an analyst's day plus a credibility hit. If you put k2 at fifty times k1 - conservative for anything feeding a roadmap decision - your break-even defect rate is 2 percent. Almost no research pipeline runs below a 2 percent defect rate at the elicitation stage. Which means the rule's answer for most teams is not "sample more carefully." It is inspect everything, and if you cannot afford to inspect everything, that is a statement about your process cost, not a licence to sample.

Deming's own position on what inspection buys you was blunter still. From Out of the Crisis, page 29: "Inspection does not improve the quality, nor guarantee quality. Inspection is too late." On the same page he quotes Harold F. Dodge, the Bell Labs statistician who invented acceptance sampling in the first place: "You can not inspect quality into a product." And on page 227: "Quality can not be inspected into a product or service; it must be built into it." The third of Deming's fourteen points is the instruction that follows: "Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place."

The reviewer has an error rate too, and it gets worse as things get worse

Even 100 percent review is not 100 percent detection, and the way detection degrades is counterintuitive enough that it deserves its own number.

Gordon and colleagues, in Health Expectations 27(3), 2024, make the point with a worked Bayesian example while analysing fraudulent survey respondents. Suppose a fraud-detection strategy has 90 percent sensitivity and 90 percent specificity, which is far better than most manual review. Their result, stated in the paper:

• At 50 percent fraud prevalence, "approximately 90% of responses we determine to be authentic will truly be authentic."
• At 80 percent prevalence, "only approximately 69% of responses classified as authentic would be truly authentic."
• At 90 percent prevalence, "the proportion of truly authentic responses decreases to 50%."

The detector did not get worse. The base rate did. The same review procedure that is trustworthy on a clean batch becomes a coin flip on a dirty one, which is precisely the situation in which teams lean hardest on it. This is the deep reason inspection cannot substitute for process control: inspection quality is a function of the incoming quality it is supposed to be protecting you from.

The same arithmetic explains why the two Gordon numbers from the previous article matter so much. Their suspected-fraud rate was 17.4 percent from the initial channel and 83.1 percent after the study was posted to social media. A review policy calibrated on the first channel is nearly useless on the second, and nothing in the review policy itself signals that it has stopped working.

The sign inversion

The previous article in this series said: every stage you add lowers the yield of the chain, because stages multiply. The natural inference is that you should add a checking stage to catch what the other stages drop.

That inference is backwards, and the reason is the whole point of this article. A checking stage is not an exception to the multiplication rule; it is subject to it. It adds its own false-reject rate to the chain, its own delay, and its own escape rate. Adding inspection to a low-yield process gives you a low-yield process that also takes longer and occasionally throws away good work.

The correct move has the opposite shape. In the yield article, the lever was measure each stage and fund the minimum. Here, the lever is stop measuring harder and change the stage that is producing defects, because the measurement is itself a sample with an OC curve and its power is worse than your intuition. More checking is not a weaker version of better process. It is a different, dominated policy.

What to do instead

1. Compute your break-even. Estimate k1 (cost of one check) and k2 (cost of one escape) for each stage. The ratio is your threshold defect rate. It takes ten minutes and it usually surprises people.
2. Estimate p per stage, once, properly. Not a spot-check. Take one batch and inspect all of it. You are not doing quality control; you are measuring the process so you never have to guess again.
3. If p is above the threshold, do not sample. Fix or automate. The two ways out of expensive 100 percent inspection are to reduce the defect rate at source, or to make inspection so cheap that k1 collapses and the break-even moves out of reach.
4. Mistake-proof at capture, not at review. A screener question that a fraudulent respondent cannot answer beats any amount of downstream transcript reading, because it moves the defect out of the batch instead of finding it in the batch. See survey fraud and respondent quality for the specific mechanisms.
5. Keep review for what it is genuinely good at: learning, not filtering. Deming's objection is to inspection as a quality strategy, not to reading your own data. A structured peer review of a study design catches whole classes of error that no sampling plan addresses, and it happens before the defects are created. The research peer review QA gate is the right shape for this: a gate on the design, not a filter on the output.
6. Say the OC curve out loud when someone proposes a sample review. "We will check ten" is a policy with a known false-accept rate. Write it in the doc.

When 100 percent inspection is right

The rule cuts both ways, and the "inspect nothing" corner is real. If your incoming defect rate is genuinely below k1/k2 - a small internal panel of known customers, a study run with an established screener on a channel you have measured - then reviewing is a net cost. The honest version of that policy is to say so, rather than performing a review that has a 90.4 percent chance of accepting a 1-percent-defective batch and calling it assurance.

Between the corners, the thing that changes the answer is k1. If checking one item costs almost nothing, the break-even defect rate falls toward zero and 100 percent inspection wins for every realistic p. That is the lever worth pulling, and it is a tooling question rather than a process question.

How Koji flips the corner solution

Koji's design attacks k1 rather than the sampling plan, which is the only move the rule licenses.

• Every interview is scored, not a sample. Koji generates a quality score from 1 to 5 for each completed interview against the study's research goals as it finishes. The inspection rate is 100 percent by construction, so consumer's risk from sampling is zero, and no batch is ever "reviewed" in a way that means "ten of them were."
• The check is not a separate stage. Because the score is produced from the same analysis pass that produces the themes, review does not add a handoff to the chain. It costs no additional elapsed time, which is what pushes k1 low enough for the all-or-none rule to land on the "inspect everything" corner.
• Defects get prevented at capture. Koji's six structured question types - open_ended, scale, single_choice, multiple_choice, ranking, and yes_no - constrain the answer space where constraint is appropriate, so a whole class of unusable response never enters the batch. See the structured questions guide for how the types map to analysis and reporting.
• The AI moderator re-probes instead of accepting a thin answer. A traditional interview's defects are created live and can only be found later. An AI moderator that follows up on a non-answer removes the defect at the moment it would otherwise be baked in, which is Deming's point about building quality in rather than inspecting it in, implemented literally.
• You can still read everything. Full transcripts remain available, so the 100 percent inspection corner is genuinely available to a human when the decision warrants it, rather than being priced out.

Legacy survey tooling gives you the opposite economics. Fielding is cheap, review is expensive and manual, so every team lands on sampling - the one policy the mathematics rules out.

Honest objections

"Deming's rule assumes a stable process, and ours is not." Correct, and Deming said so: the kp rule applies to a process in statistical control. For an unstable process, the honest reading is worse for sampling, not better, because an unstable p means your sampling plan's OC curve is calibrated to a defect rate that is no longer current. This is exactly the Gordon 17.4 to 83.1 percent case.

"We cannot inspect everything, so we have to sample." That is a real constraint, but it should be recorded as a known unmanaged risk rather than as assurance. If you sample ten and accept, write down the probability that you have just accepted a 10 percent defective batch. It is 34.9 percent.

"Acceptance sampling is standard practice across whole industries." It was, and the profession has been arguing about it since Mood's theorem and Deming's critique. Acceptance sampling answers "should I accept this lot at a stated risk," which is a supplier-relations question. It does not answer "how good is my process," which is the question research teams are actually asking when they spot-check.

"Our reviewers are better than 90 percent sensitivity." Possibly, on clean batches. Ask what their sensitivity is on a batch that is 80 percent bad, then re-read the Gordon numbers. Reviewer performance is not a constant.

Frequently asked questions

How many transcripts should we spot-check?

The honest answer from the mathematics is: none, or all of them. Compute k1/k2 - the cost of one check divided by the cost of one defect escaping - and compare it to your actual defect rate. If your defect rate is higher, review everything. If it is lower, reviewing is a net cost. A partial review is dominated by one of those two policies at essentially every defect rate.

What does a spot-check of ten actually detect?

Not much at the rates that matter. Reviewing ten items and rejecting on any defect accepts a 5-percent-defective batch 59.9 percent of the time and a 10-percent-defective batch 34.9 percent of the time. Put the other way, at a 5 percent defect rate a ten-item check finds at least one defect only 40.1 percent of the time.

What is Deming's all-or-none rule?

It is the cost-minimising inspection policy derived in Chapter 15 of Out of the Crisis. With k1 the cost of inspecting one item and k2 the cost of a defect escaping, inspect nothing if the incoming fraction defective is below k1/k2 and inspect everything if it is above. Sampling is optimal only exactly at the break-even point, so in practice it is never the right policy.

Does that mean peer review of research is a waste of time?

No, and this is the important distinction. Deming's objection is to inspection as a substitute for process quality. Reviewing a study design before fielding prevents defects rather than filtering them, which is the thing he was arguing for. Reviewing a sample of outputs after fielding is the thing he was arguing against.

Why does a reviewer's accuracy fall when data quality falls?

Because predictive value depends on base rate, not just on sensitivity and specificity. Gordon and colleagues show that a detector with 90 percent sensitivity and specificity leaves about 90 percent of "authentic" classifications truly authentic at 50 percent fraud prevalence, about 69 percent at 80 percent prevalence, and 50 percent at 90 percent prevalence. Your review gets least trustworthy exactly when you need it most.

What is the producer's risk in a research review gate?

It is the chance of rejecting and re-fielding a batch that was actually fine. Research teams almost never count this cost, but re-fielding a study is weeks of elapsed time and a delayed decision. A review policy has to be judged on both risks, and tightening the acceptance number to reduce escapes always raises false rejects.

Related Resources

• Structured Questions Guide - constraining the answer space so defects are never created
• Research Pipeline Yield - the series-model arithmetic that makes an extra review stage costly
• When No Study Was Wrong - the failures that no amount of inspection can catch
• The Research Peer Review QA Gate - review applied to the design, where it prevents rather than filters
• Survey Fraud and Respondent Quality - moving defects out of the batch at the screener
• Inter-Rater Reliability in Qualitative Research - measuring the coding stage rather than sampling it
• Research Calibration and Brier Scores - scoring judgement quality once you have stopped sampling it