The AI Interviewer House Effect: When One Interviewer Turns Variance Into Bias

Short answer: an AI interviewer does not remove the interviewer effect. It removes interviewer variance and converts what is left into bias. With a panel of human moderators, the asker's influence shows up as extra spread in your data — measurable, correctable, and visible in your error bars. With a single AI interviewer running every conversation, there is no second asker to compare against, so the asker's influence becomes a constant offset applied to every answer in the study. Constants do not widen confidence intervals. Your interval gets tighter and your number can still be shifted. The fix is not to go back to humans: it is to treat the interviewer as an instrument with a version number and to randomise respondents across two interviewer configurations at least once per study design.

This article explains the arithmetic survey statisticians have used since 1962, shows what changes when the interviewer count drops to one, and gives you a protocol for measuring your own house effect.

The formula every survey team knows and most product teams have never seen

When several interviewers each handle a batch of respondents, answers collected by the same interviewer resemble each other more than they resemble answers collected by a different interviewer. That clustering inflates the variance of whatever you estimate. The multiplier is the interviewer design effect, from Kish (1962):

deff(int) = 1 + (m - 1) x rho(int)

• rho(int) (the intra-interviewer correlation) is the share of total variance in an answer attributable to differences between interviewers.
• m is the average interviewer workload — how many interviews each interviewer conducts.

Two things about this formula matter more than the formula itself.

First, ρ is always small. Groves (1989) compiled estimates across many published studies and found that for telephone surveys the value was seldom higher than 0.02. Work on the European Social Survey reports that intraclass correlations found throughout the literature range roughly between 0.00 and 0.05, sometimes increasing to 0.10, with some outliers exceeding 0.10. Analysis of the first ESS round across more than 20 countries, covering 105 survey items, found intra-interviewer correlation varying from about 0.05 in the Scandinavian countries to 0.15–0.20 in the Southern European countries.

Second, small ρ still does enormous damage, because it is multiplied by workload. The standard worked example: if interviewer workloads are 40 to 50 interviews and ρ is 0.02, the variance of the sample mean is increased by 80 to 100 percent. A correlation of two percent nearly doubles your variance. That is the same as throwing away roughly half your sample.

This is the real, unglamorous case against a panel of human moderators, and it is stronger than the usual argument about consistency or scheduling. A study run by twelve freelancers is not a study with n=240. It is a study with an effective sample size closer to 130, and nothing on the invoice says so.

What actually changes when the interviewer count drops to one

Set m equal to your entire sample and the formula stops being useful, for a reason that is easy to miss.

With one interviewer, there is no between-interviewer contrast left in the data. You cannot estimate ρ, because estimating it requires comparing groups of respondents handled by different askers. There is only one group. The interviewer term is perfectly confounded with the study mean.

That is not the same as the interviewer effect being zero. It means the effect has moved. Specifically:

Read that table honestly and it is a strong argument for an AI interviewer on five of six rows. Perfect consistency and reproducibility are real methodological wins, and no human panel can offer them. But row two is the trap. The bias does not announce itself. A study with one asker produces a narrower interval around a number that may be systematically shifted — the most dangerous combination in applied research, because narrow intervals are what stakeholders read as confidence.

Our own guide to interviewer bias makes the case that automating the moderator eliminates interviewer variance. That is correct as far as it goes, and this article is the second half of the sentence: eliminating the variance is the easy part, and it leaves a residue that has to be measured separately.

Which questions carry the most house effect

Interviewer effects are not spread evenly across a questionnaire. They concentrate where the asker has the most room to shape the exchange:

• Open-ended and attitudinal questions — highest exposure. Probing decisions, follow-up depth, and what counts as "enough" of an answer are all interviewer choices.
• Sensitive or socially loaded questions — high exposure, because the asker's perceived judgement changes what people admit. See social desirability bias.
• Ambiguous questions requiring clarification — high exposure, because clarification is improvised.
• Closed, fully specified questions — lowest exposure. The response options do the work, not the asker.

This gives you a practical design rule that costs nothing. Anything you intend to report as a number should be asked with a structured question type; open-ended questions should carry the mechanism, not the metric. Koji ships six structured types — open_ended, scale, single_choice, multiple_choice, ranking, and yes_no — and five of the six are closed formats where the AI reads fixed options and records a fixed answer. Put your tracked metric in a scale or single_choice question, let the AI probe conversationally around it with open_ended, and you have quarantined your headline number from the part of the interview where the house effect lives.

The interviewer A/B: a split ballot for the asker

Here is the method that makes the whole problem tractable, and it is only available to teams whose interviewer is software.

A split-ballot experiment randomises respondents across two versions of a question to measure how much of an answer is the wording. An interviewer A/B randomises respondents across two versions of the interviewer to measure how much of the answer is the asker.

How to run one:

1. Build two interviewer configurations that differ in exactly one dimension — persona warmth, probing aggressiveness, whether company context is loaded, or voice versus text delivery.
2. Randomly assign incoming participants to configuration A or configuration B. Do not let them choose; self-selection creates a mode effect on top of the interviewer effect and you will not be able to separate them.
3. Field both arms simultaneously with the same recruitment source and the same screener.
4. Compare your headline metric across arms. The difference is your house effect for that dimension.
5. Report it. A single line — "configuration A produced a mean satisfaction of 7.1 and configuration B produced 6.6; we report A and note a 0.5-point interviewer sensitivity" — is worth more than another hundred interviews.

Sizing: the same rule of thumb as any two-arm comparison applies. Roughly 200 completions per arm lets you detect a difference of about 10 percentage points on a proportion. If you cannot afford 400 interviews, run the A/B once on your most important recurring study rather than a small underpowered version on every study.

Frequency: you do not need this on every project. You need it once per interviewer configuration you rely on, and again whenever that configuration changes.

Version-pinning: treat your interviewer as an instrument with a serial number

A laboratory does not swap the spectrometer mid-experiment and call the readings comparable. The same discipline applies here, and almost no one applies it.

The rule: for any tracking study, the interviewer configuration is part of the instrument. Freeze it, record it, and treat a change to it as a series break.

Concretely, record alongside every wave:

• The persona and tone configuration in use
• Whether company context documents were loaded, and which version
• The probing and follow-up settings
• The delivery mode offered, and the realised mode mix
• The date of the last configuration change

When you do change the configuration — and you should, because tuning improves data quality — run one wave with both the old and new configurations in parallel. That overlap wave is what lets you state whether a movement in your metric is your customers changing or your instrument changing. Without it, every tuning improvement silently contaminates your trend line.

This is the single biggest advantage of an AI interviewer over a human panel, and it is rarely claimed: a configuration is an artifact that can be versioned, diffed, re-run, and audited. A person cannot be. You can never ask a human moderator to reproduce exactly the interview they conducted eight months ago. You can ask software to.

How Koji makes the house effect measurable

Koji is built so that all four of the ingredients above exist as real objects rather than institutional memory:

• The interviewer is configurable and inspectable. Custom personas and interviewer tuning are settings on the study, not habits in a contractor's head, so "which interviewer ran this?" has an exact answer.
• Every interview is captured in full. Transcripts, question sequence, and follow-up behaviour are all retained, so you can check after the fact whether one arm was probed harder than the other — see paradata for how to read follow-up depth as a diagnostic.
• Structured questions isolate the metric. The six question types produce the same coded output regardless of how the surrounding conversation went.
• Randomised assignment is a study setting, not a spreadsheet. Two studies with the same brief, the same recruitment link strategy, and different interviewer configurations is a complete interviewer A/B.
• Analysis is automatic on both arms. Because reports generate without manual coding, running two arms costs roughly what running one arm costs — which is why this experiment is practical here and was never practical with human moderators.

A five-step house-effect protocol

1. Declare your headline metric and move it into a closed structured question type.
2. Pin the interviewer configuration and record it in the study record.
3. Run one interviewer A/B on the dimension you are least sure about, with ~200 per arm.
4. Publish the sensitivity next to the number, the way a good methods note publishes a margin of error.
5. Re-run the A/B after any configuration change, or overlap one wave with both versions.

Teams that do this end up with something almost no research function has: a documented, quantified estimate of how much of their number belongs to the person — or the system — that asked the question.

Frequently asked questions

Does an AI interviewer eliminate interviewer bias?

It eliminates interviewer variance — the inconsistency between different askers — which is a genuine and substantial win, because that variance can nearly double the variance of your estimates at typical human workloads. It does not eliminate interviewer effects. With one asker, the effect becomes a constant offset shared by every interview in the study, which means it no longer widens your confidence interval and no longer shows up in any standard diagnostic. You have to test for it deliberately with an interviewer A/B.

What is a realistic intra-interviewer correlation?

Published estimates cluster low. Groves (1989) found that for telephone surveys the value was seldom higher than 0.02, and European Social Survey work reports intraclass correlations across the literature ranging roughly between 0.00 and 0.05, sometimes increasing to 0.10, with outliers above that. ESS analysis across 20-plus countries and 105 items found values from about 0.05 in Scandinavia to 0.15–0.20 in Southern Europe. The important point is that these small numbers get multiplied by interviewer workload, so a ρ of 0.02 at workloads of 40 to 50 interviews increases the variance of the sample mean by 80 to 100 percent.

How do I detect a house effect if I only have one interviewer?

You cannot detect it from a single-arm study, and that is the core problem. Detection requires a contrast: run two interviewer configurations against randomly assigned respondents in the same fielding window, then compare your headline metric between arms. The alternative is an external contrast — comparing your result against a differently collected benchmark — but that confounds interviewer effects with every other difference between the two studies.

Should I go back to human moderators to avoid this?

No. A human panel replaces one unmeasured constant with many unmeasured and uncontrollable differences, and it costs you the ability to reproduce or audit anything. The comparison that matters is not "biased versus unbiased" — it is "a stable, versioned, testable offset" versus "an unstable offset you cannot reconstruct." The first is a research instrument. The second is a collection of anecdotes gathered under a shared brief.

Does this affect qualitative studies too?

Yes, but differently. In a qualitative study you are not defending a point estimate, so the variance arithmetic does not apply. What does apply is the shaping of which topics get explored: an interviewer that probes hard on pricing and lightly on onboarding produces a corpus where pricing themes look more prevalent. Reviewing follow-up depth by question is the practical check.

How often should I re-run an interviewer A/B?

Once per configuration, not once per study. Re-run it whenever you change the persona, change the context documents the AI works from, change probing behaviour, or change the delivery mode mix — and always before comparing a new wave of a tracking study against historical waves collected under a different configuration.

Related Resources

• Interviewer Bias — the variance problem this article picks up where it ends
• Split-Ballot Question Wording Experiments — the same experimental logic applied to the question instead of the asker
• Mode Effects in Mixed-Mode Research — why letting participants choose voice or text confounds an interviewer A/B
• Total Survey Error — where interviewer effects sit in the full error budget
• AI Interviewer Tuning — the settings that define a configuration worth pinning
• Structured Questions Guide — the six question types that quarantine your headline metric