Interviewer Variance: Every Participant Answered a Slightly Different Question (2026)

Here is a failure mode with no villain. Nobody lied. No response was malformed. The sample was drawn properly, the guide was well written, the analysis was competent, and the arithmetic is right. And the study is still wrong, because the one thing you assumed was held constant -- the question itself -- was not. Each moderator delivered it a little differently, each participant therefore answered a slightly different question, and the transcript records a single constant question for all of them.

Survey methodologists have measured this for sixty years and given it a number. The headline result is that a 500-interview study can be worth 125 interviews, and nothing in the dataset will tell you. It is also the clearest quantitative case for AI-moderated research, which is why the honest accounting of what a platform like Koji does and does not fix appears further down rather than at the top.

The measurement: intra-interviewer correlation

The quantity is called the intra-interviewer correlation, written rho. It is the ratio of the variance between interviewers in the mean of an item to the total variance in that item. If every moderator gets the same average answer, rho is zero. If moderators differ systematically -- one warms people up, one rushes, one adds an example to question four -- rho climbs.

Interviewer variance inflates the uncertainty of your estimates, in some cases more than the clustering induced by geography does. It behaves exactly like cluster sampling, because it is cluster sampling: each moderator's caseload is a cluster. The inflation factor is the design effect

deff = 1 + rho x (m - 1)

where m is the average number of interviews per moderator. Divide your nominal sample by deff and you get the effective sample size: the number of independent observations your study is actually worth.

The important property of that formula is that rho is multiplied by the workload. Even a small correlation gets amplified when each moderator runs a lot of interviews -- which is precisely what an efficient, well-run study does.

What rho is in practice

The classic estimates come from Groves and Magilavy and from Groves and Kahn. Across face-to-face surveys, rho ranged from 0.005 to 0.102. In centralized telephone surveys, where interviewers are closely monitored and supervised, the range was 0.0018 to 0.0184 -- an order of magnitude lower. More recent work treats rho of about 0.12 as large.

Those numbers look reassuringly small. Run them through the formula with a realistic workload and they stop looking small.

At the top of the published face-to-face range, with a workload most teams would consider modest, a 500-interview study carries the precision of 125 interviews. You paid for 500. You can report 500. Your confidence intervals, if you computed them the ordinary way, are roughly half as wide as they should be.

The result that should change how you think about it

The single most useful finding in this literature is that interviewer variance is not a property of the question.

In one mixed-mode study, the item satisfaction with healthcare -- the same question, the same instrument, the same population -- produced rho of 0.125 when asked face to face and 0.029 when asked by telephone. A factor of 4.3, from nothing but the delivery channel and the supervision that came with it.

Carry that through the formula at a workload of 25. The face-to-face version of that question is worth 125 effective participants out of 500. The telephone version of the same question is worth 295. Same words, same people, 170 effective participants of difference created entirely by how the question was delivered.

This is why we wrote a good guide is not a defence. The guide was identical in both arms.

The workload lever, which runs against instinct

Because deff depends on the product of rho and workload, the number of moderators is a design parameter with real statistical consequences -- and it points the opposite way from operational efficiency.

Take 500 interviews at rho = 0.05 and vary only how many moderators share the work:

Consolidating a study from 20 moderators onto 5 -- fewer people to brief, fewer contracts, better utilisation, every instinct a research ops lead has -- costs 143 effective participants while changing neither the sample nor the guide nor the budget for fieldwork. The efficient staffing plan is the imprecise one.

There is a second, harder requirement lurking here. To separate a genuine interviewer effect from the simple fact that different moderators got different kinds of participant, you need an interpenetrated design: participants randomly assigned to moderators. Almost nobody does this, which means most reported rho values are partly confounded with caseload composition, and most studies have no way to estimate their own rho at all.

Where this sits among research failures

It is worth being precise about what kind of failure this is, because it does not resemble the others.

• It is not a sampling failure. The right people were recruited.
• It is not a response failure. Participants answered honestly and fully.
• It is not an analysis failure. The statistics were computed correctly on the data collected.
• It is not a leading question failure. The written question may be flawless.

It is a failure of stimulus constancy: the input was supposed to be held fixed across participants and was not, and the record of the study shows the fixed version. There is no defect to find inside any single interview. Each transcript is fine. The damage lives only in the comparison between them -- which is the one place nobody looks, because comparison is what the analysis is for, not something the analysis examines.

The other three articles in this series describe the specific mechanisms by which the stimulus drifts. A moderator adds a premise that was not in the stem, covered in presupposition in interview questions. A moderator fills a pause or appends a qualifier after the question has ended, covered in response timing and hedged answers. A moderator improvises an answer when a participant asks what a term means, covered in repair and clarification in research interviews. Each of those is a small, reasonable, well-intentioned act. Rho is what they add up to.

The honest version of the AI argument

Koji's AI moderator delivers the same question, with the same wording, in the same order, with the same scripted clarifications, to every participant in a study -- whether that is 20 people or 2,000. Structurally, that drives the between-moderator variance component toward zero, because there is no second moderator to vary from. The deff inflation from rho_int does not get reduced so much as removed from the design.

That is a genuine and substantial gain, and it is worth being exact about what it does and does not buy.

What it buys: precision and comparability. Your effective sample size approaches your nominal sample size. Cross-segment comparisons are not contaminated by which moderator happened to run which segment. A study fielded in March is comparable to one fielded in September, because the instrument did not drift in between. And the entire trend visible in the published data -- centralized, supervised telephone interviewing showing rho an order of magnitude below field interviewing -- points in exactly this direction. Consistent, monitored delivery is the mechanism, and an AI moderator is the limiting case of it.

What it does not buy: validity. Removing variance between moderators converts variable error into constant error. A question with a premise buried in it is now delivered with that premise, identically, to all 2,000 participants. The error no longer shows up as noise, and -- this is the uncomfortable part -- it no longer shows up as a moderator discrepancy either, because there is no contrast left to reveal it. A study with three moderators has at least the possibility that someone notices question four behaving oddly in one caseload.

So consistency raises the stakes on question design rather than lowering them. The defences are the ones in this series: audit the premises before launch, script the clarifications, and put a known-null item in the instrument so the process has something to fail on, as described in negative controls in user research. Consistency multiplies whatever your instrument does. Make sure it is doing the right thing first.

In Koji that discipline is supported by the study brief and by the six structured question types -- open_ended, scale, single_choice, multiple_choice, ranking, and yes_no -- which force the answer space to be declared explicitly rather than improvised at the moment of asking, as laid out in the structured questions guide. Every turn of every interview, including clarifications and follow-ups, is logged in the transcript, so the instrument as delivered is auditable after the fact rather than reconstructed from memory. Koji also scores each conversation for quality on a 1-to-5 scale automatically, with only conversations scoring 3 or above consuming a credit, and can run voice or text interviews from the same study definition -- which means the delivery channel, the variable that moved rho by a factor of 4.3 in the study above, is a setting rather than a staffing decision.

What to do on your next study

1. Record who moderated each interview. You cannot estimate rho without it, and most qualitative studies do not capture it at all.
2. Compute the design effect before you report a confidence interval. Use deff = 1 + rho x (m - 1). If you have no estimate of rho, assume 0.02 for a tightly supervised study and 0.05 otherwise, and say so.
3. Report effective sample size next to nominal sample size. A study described as n = 500 (effective n = 227 at rho = 0.05) is a study whose reader can calibrate.
4. Randomise participants to moderators if you have more than one. Without interpenetration you cannot tell a moderator effect from a caseload effect.
5. Prefer more moderators with smaller caseloads, or one consistent AI moderator, over a small number of heavily loaded humans. Koji removes the trade-off entirely at this step, because adding participants does not add moderators.
6. Never pool across a mid-study wording change. A guide edited in week two creates two instruments, and pooling them is the same error as pooling two moderators.

Frequently asked questions

What is interviewer variance?

It is the portion of variation in your results caused by differences between the people asking the questions rather than differences between the people answering. It is measured as the intra-interviewer correlation, rho -- the ratio of between-interviewer variance to total variance -- and it inflates the uncertainty in your estimates the same way cluster sampling does.

How much does interviewer variance actually cost?

The design effect is 1 + rho x (m - 1), where m is interviews per moderator. At rho of 0.125, near the top of the published face-to-face range, with 25 interviews per moderator, the design effect is 4.0 -- so a 500-interview study carries the precision of 125 interviews.

Is interviewer variance a property of the question?

No, and this is the most useful finding in the literature. The same item -- satisfaction with healthcare -- produced rho of 0.125 face to face and 0.029 by telephone in the same study. How a question is delivered and supervised matters as much as how it is worded.

Does using fewer moderators make a study more consistent?

The opposite, statistically. Because the design effect multiplies rho by the average workload, concentrating 500 interviews onto 5 moderators instead of 20 raises the design effect from 2.20 to 5.95 and drops effective sample size from 227 to 84. Fewer moderators means bigger clusters, not less variance.

Does an AI moderator eliminate interviewer variance?

It removes the between-moderator component, because there is only one moderator delivering identical wording to everyone -- which is why effective sample size approaches nominal sample size with Koji. But it converts variable error into constant error: a badly framed question is now uniformly badly framed, with no moderator contrast to reveal it. Consistency buys precision, not validity.

How do I estimate rho for my own study?

You need the moderator recorded for every interview, and ideally participants randomly assigned to moderators so that moderator effects are not confounded with caseload composition. Then fit a model with moderator as a random effect and take the ratio of between-moderator variance to total variance. Without random assignment, treat any estimate as an upper bound.

Related Resources

• Structured questions guide -- declaring the answer space explicitly with the six question types instead of improvising it at the moment of asking.
• Presupposition in interview questions -- one specific mechanism by which a delivered question drifts from the written one.
• Repair and clarification in research interviews -- scripting the clarifications that otherwise get improvised differently by every moderator.
• Response timing and hedged answers -- why filling a pause changes the stimulus.
• Negative controls in user research -- putting a known-null item in the instrument so a consistent process still has something to fail on.
• AI-moderated interviews -- how automated moderation works in practice, and where it compares to human moderation.