{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-09-24T10:37:01.817Z"},"content":[{"type":"documentation","id":"1b0db8a7-0fd4-49ac-aecc-1123dbd53aad","slug":"reliability-growth-pilot-testing-rounds","title":"When to Stop Piloting: The Defect-Decay Stopping Rule for Interview Guides","url":"https://www.koji.so/docs/reliability-growth-pilot-testing-rounds","summary":"Applies Duane reliability growth modelling to interview-guide piloting. The Duane postulate (cumulative MTBF vs cumulative time is a straight line on log-log paper, MTBFc = bT^a, with growth rates observed between 0.24 and 0.65) maps onto sessions-per-defect vs cumulative pilot sessions. Because the growth rate a is always below 1, cumulative defects grow as T^(1-a) and never converge, which proves \"pilot until nothing new comes up\" cannot be a stopping rule. Replaces it with an economic rule: stop when expected new defects per round times the cost of an escaped defect falls below the cost of a round. Worked five-round example yields a = 0.35, and the arithmetic shows most teams stop piloting far too early because recruited pilot participants cost nearly as much as real ones. Cheap AI pilot conversations move the stopping point far down the curve.","content":"\"Pilot until nothing new comes up\" is not a stopping rule. It is a promise to pilot forever, because the rate at which a pilot finds new defects decays toward zero without ever arriving there. Reliability engineers worked this out in the 1960s and gave it a model. Research teams are still using a feeling.\n\nThe rule that replaces it is economic, not empirical: stop piloting when the expected cost of the defects the next round would find drops below the cost of running that round. That threshold is computable from your own first three rounds, and for most teams it sits much later than where they actually stop.\n\n## Why \"nothing new came up\" fails as a stopping rule\n\nThree problems, and they compound:\n\n- It is unfalsifiable in a small sample. One quiet round proves nothing when your per-round defect count is already down to two or three.\n- It ignores severity. A round that surfaces one catastrophic defect beats a round that surfaces four cosmetic ones, and \"nothing new\" treats them identically.\n- It has no relationship to cost. It never asks what the next round is worth, which is the only question a stopping rule should answer.\n\n### The Duane postulate\n\nIn the 1960s J. T. Duane observed something regular about systems under an improvement programme. ReliaSoft's reference documentation states the observation directly: \"the cumulative MTBF versus cumulative operating time followed a straight line when plotted on log-log paper.\"\n\nThe relationship is written as MTBFc = bT^a, where T is cumulative test time, b is the initial parameter, and a is the growth rate, constrained so that 0 <= a <= 1. The documentation notes: \"The lower values reflect slow early growth and the higher values reflect fast early growth.\" Observed values across equipment types fall in a band between 0.24 and 0.65.\n\nThe NIST/SEMATECH e-Handbook makes the same point about what this kind of testing is for, and gives it a name: \"Another name for reliability improvement testing is TAAF testing, standing for Test, Analyze And Fix.\" That is precisely what a pilot round is.\n\n### Translating the model to an interview guide\n\nThe mapping is direct once you pick the right units:\n\n| Reliability engineering | Interview guide piloting |\n| --- | --- |\n| Cumulative test time T | Cumulative pilot sessions run |\n| Failure | A defect found in the guide |\n| Cumulative MTBF | Sessions per defect found so far |\n| Fix between rounds | Rewriting the question that broke |\n| Growth rate a | How fast your pilot is exhausting defects |\n\nThe quantity you plot is cumulative sessions divided by cumulative defects. It rises as the guide improves, because it takes more and more sessions to shake out one more problem.\n\n### A worked example: five pilot rounds of two sessions each\n\n| Round | Sessions this round | Defects found | Cumulative sessions T | Cumulative defects N | Sessions per defect (T/N) |\n| --- | --- | --- | --- | --- | --- |\n| 1 | 2 | 6 | 2 | 6 | 0.33 |\n| 2 | 2 | 4 | 4 | 10 | 0.40 |\n| 3 | 2 | 3 | 6 | 13 | 0.46 |\n| 4 | 2 | 2 | 8 | 15 | 0.53 |\n| 5 | 2 | 2 | 10 | 17 | 0.59 |\n\nPlot the last column against T on log-log axes and you get a near straight line. The slope is the growth rate:\n\na = (ln 0.59 - ln 0.33) / (ln 10 - ln 2) = 0.568 / 1.609 = 0.35\n\nAn a of 0.35 sits squarely inside the 0.24 to 0.65 band, which is the first useful signal: this pilot is behaving normally. A much lower slope would say your fixes are not sticking. A much higher one would say you front-loaded the obvious defects and are about to hit a wall.\n\n### Why the curve never reaches zero\n\nRearranging the model gives the cumulative defect count directly:\n\nN = T / MTBFc = T / (b x T^a) = T^(1-a) / b\n\nBecause a is always less than 1, the exponent (1 - a) is always positive. N grows without bound as T grows. It grows slowly - here as T^0.65, which is sublinear - but it never flattens to a horizontal line and never stops producing new defects.\n\nThis is the mathematical statement of why \"pilot until nothing new comes up\" cannot work. There is always something new. You just have to wait longer and longer for it.\n\n### What the model predicts for the next rounds\n\nWith b = 0.26 and a = 0.35, projecting forward from round 5:\n\n| Cumulative sessions T | Predicted cumulative defects N | New defects since previous row |\n| --- | --- | --- |\n| 10 | 17.1 | - |\n| 12 | 19.2 | 2.2 |\n| 14 | 21.3 | 2.0 |\n| 20 | 26.8 | 5.5 (over three rounds) |\n\nDoubling the pilot from 10 sessions to 20 buys about 10 more defects. That is not nothing, and it is the number the stopping rule needs.\n\n### The economic stopping rule\n\nStop when:\n\n(expected new defects per round) x (expected cost of one escaped defect) < (cost of one pilot round)\n\nWork it with real figures. Say a recruited pilot participant costs 150 to source and incentivise, so a two-session round costs 300. Say an escaped defect costs you roughly one usable session out of your real study, and your real participants cost 250 each.\n\nAt round 6, the model predicts 2.2 new defects. Expected value: 2.2 x 250 = 550. The round costs 300. Keep piloting.\n\nPush it out to T = 20, where the marginal rate has fallen to about 1.7 defects per round. Expected value: 1.7 x 250 = 425, still above 300. Keep piloting.\n\n### The uncomfortable conclusion: almost everyone under-pilots\n\nRun that arithmetic honestly and the stopping point lands far to the right of where teams actually stop. Most guides get two pilot sessions and a thumbs-up. The model says that at a realistic growth rate, round 6 and round 10 are still positive-expected-value, because each escaped defect quietly consumes a paid participant for the entire remaining study.\n\nThe reason teams stop early is not that they disagree with the arithmetic. It is that recruiting a pilot participant costs nearly as much as recruiting a real one, so every pilot round feels like it is eating the study. That constraint is real, and it is the thing worth attacking.\n\n### Recalibrating when a pilot round gets cheap\n\nThe stopping rule is a ratio, so it moves when either side moves. Cut the cost of a pilot round by an order of magnitude and the optimal stopping point slides far down the curve - you can afford to keep riding the decay long past the point where a recruited-panel pilot would have to quit.\n\nThis is the single highest-leverage change available, and it is structural rather than methodological. Nothing about the Duane curve changes. The economics on the right-hand side of the inequality do.\n\n## How Koji handles this\n\n- Koji collapses the cost side of the stopping rule. A pilot conversation costs credits rather than a recruiting fee - text conversations cost 1 credit and voice conversations cost 3 - which moves the economic stopping point far to the right of where a traditional pilot has to stop.\n- Koji's quality gate means only conversations scoring 3 or above consume credits, so pilot sessions that break badly enough to be unusable are not billed as data.\n- Koji's quality scoring gives you the defect signal directly. Each conversation is scored 1-5 across relevance, depth, and coverage, and a question that scores consistently low on coverage is a located defect rather than a vague sense that something is off.\n- Koji's structured questions eliminate an entire defect class before round 1. The six types - open_ended, scale, single_choice, multiple_choice, ranking, and yes_no - remove the wording ambiguity and dead-end branching that dominate early pilot rounds.\n- Koji's AI follow-up questions surface dead ends faster, because the AI probes every branch rather than the one the moderator expected.\n\nThe honest caveat: cheap pilot rounds do not fix a guide by themselves. Riding the curve further only pays if you actually analyse and fix between rounds. Piloting without the Analyze and Fix half of TAAF produces a flat line, not growth.\n\n## Common mistakes\n\n- Treating one quiet round as proof of convergence. At two defects per round, a zero-defect round is well within noise.\n- Fitting the curve without fixing between rounds. If you do not repair the defect, you are not on a growth curve at all - you are just re-measuring the same broken guide.\n- Ignoring severity. Weight defects by what they cost downstream before you run the arithmetic, or one cosmetic typo counts the same as a question nobody can answer.\n- Piloting with the friendliest available participant, which suppresses the defect rate and makes the curve look better than the guide is.\n- Assuming the growth rate from someone else's study. Compute your own from your first three rounds; it takes two minutes and it is the only number that describes your guide.\n\n## Frequently asked questions\n\n### How many pilot interviews should I run before launching a study?\n\nThere is no fixed number, and any article that gives you one is guessing. Run three rounds, compute your own growth rate from the sessions-per-defect column, then apply the economic rule: keep going while the expected new defects per round times the cost of an escaped defect exceeds the cost of a round. For most teams on a recruited panel that lands somewhere past round 5, which is considerably further than they currently go.\n\n### What is a normal growth rate for an interview guide pilot?\n\nReliability engineering observes growth rates between 0.24 and 0.65 across equipment types, and an interview guide pilot behaving normally tends to land in the same band. The absolute value matters less than what an outlier tells you. A very low slope means your fixes are not sticking between rounds. A very high slope means you cleared the obvious defects fast and the remaining ones will be expensive to find.\n\n### Does this replace data saturation?\n\nNo, they measure different things and both apply. Saturation is about the findings: you stop collecting when new participants stop producing new themes. Defect decay is about the instrument: you stop piloting when the guide stops producing new problems worth finding. A study can be nowhere near thematic saturation while its guide is thoroughly debugged, and the reverse happens too.\n\n### Can I count severity instead of just counting defects?\n\nYes, and you should. Assign each defect a cost in units of lost usable sessions, then plot cumulative sessions against cumulative severity-weighted defects instead of a raw count. The model is indifferent to what you are counting as long as you count it consistently. Raw counts are only the default because they are easy, not because they are right.\n\n### Why does the defect curve never reach zero?\n\nBecause the growth rate is always below 1, which makes cumulative defects grow as T raised to a positive power. Growth is sublinear, so the discoveries get further and further apart, but the total never converges to a ceiling. That is the formal reason a stopping rule has to be economic: there is no natural endpoint where the guide is finished, only a point where looking harder stops being worth the money.\n\n### How does an AI interviewer change the stopping point?\n\nIt changes the cost side of the inequality, not the curve. With Koji, a pilot conversation costs 1 credit for text or 3 for voice instead of a recruiting fee, so a round that used to cost hundreds costs a rounding error. The expected-value threshold is then crossed much later, which means you can debug a guide far more thoroughly before a single real participant ever sees it.\n\n## Related Resources\n\n- [Structured Questions in AI Interviews](/docs/structured-questions-guide) - removing the defect classes that dominate early pilot rounds\n- [Pilot Study in User Research](/docs/pilot-study-user-research-guide) - how to run the rounds this model measures\n- [Why Your First Three Interviews Always Go Worst](/docs/interview-guide-failure-rate-curve) - what happens to the defects a pilot does not catch\n- [Data Saturation in Qualitative Research](/docs/data-saturation-qualitative-research) - the other stopping question, and why it is a different one\n- [Interim Analysis and Stopping Rules](/docs/interim-analysis-stopping-rules-research) - stopping data collection rather than stopping piloting\n- [Understanding Quality Scores](/docs/understanding-quality-scores) - turning a vague sense of trouble into a located defect","category":"Research Methods","lastModified":"2026-09-24T03:31:52.173222+00:00","metaTitle":"When to Stop Piloting: The Defect-Decay Rule (2026)","metaDescription":"The defect curve never reaches zero, so \"pilot until nothing new comes up\" fails. Use the economic stopping rule instead, with a worked example.","keywords":["when to stop piloting","how many pilot interviews","reliability growth research","duane plot","defect decay interview guide","pilot testing rounds"],"aiSummary":"Applies Duane reliability growth modelling to interview-guide piloting. The Duane postulate (cumulative MTBF vs cumulative time is a straight line on log-log paper, MTBFc = bT^a, with growth rates observed between 0.24 and 0.65) maps onto sessions-per-defect vs cumulative pilot sessions. Because the growth rate a is always below 1, cumulative defects grow as T^(1-a) and never converge, which proves \"pilot until nothing new comes up\" cannot be a stopping rule. Replaces it with an economic rule: stop when expected new defects per round times the cost of an escaped defect falls below the cost of a round. Worked five-round example yields a = 0.35, and the arithmetic shows most teams stop piloting far too early because recruited pilot participants cost nearly as much as real ones. Cheap AI pilot conversations move the stopping point far down the curve.","aiDifficulty":"advanced","aiEstimatedTime":"10 min"}],"pagination":{"total":1,"returned":1,"offset":0}}