AI Interview Best Practices: 14 Rules for Running High-Quality AI-Moderated Customer Research

AI-moderated interviews are not the same as surveys, and they are not the same as moderated user interviews. They are a third research method with their own design patterns, failure modes, and best practices. Teams that get this right run 50 high-quality customer interviews in a week with a report ready by the following Monday. Teams that don't get this right ship a "set and forget" interview that produces shallow data and stale insights.

This guide is the practical playbook for running AI interviews that match or exceed the quality of moderated research. Fourteen rules, each with a specific recommendation and the reasoning behind it. Some are about study design, some are about question writing, some are about mode selection and recruiting. All of them are battle-tested by teams running customer research at scale on Koji.

If you're evaluating AI customer research tools or you've just shipped your first study and want to do the next one better, this is the document for you.

Rule 1: Write Like You're Briefing a Senior Researcher

The single highest-leverage thing you can do for AI interview quality is write a clear research brief. The brief is what the AI moderator uses to decide when to probe deeper, when to move on, and how to interpret ambiguous answers.

Good briefs are specific about:

• The research goal: What decision will this research inform?
• The hypothesis: What do you believe, that this research might confirm or refute?
• The target audience: Who should be in this study, and what background do they bring?
• What "good" looks like: What kinds of answers would be especially useful to capture verbatim?
• What's out of scope: Topics not to probe on, even if participants mention them.

If your brief reads like a survey title ("Customer Satisfaction Q4"), the AI has no anchor for judgment. If it reads like an internal Slack thread to a senior researcher ("We're trying to figure out why churn spiked among Pro-tier customers in September — specifically whether the pricing change or the UI redesign drove it"), the AI moderator has the context to interview like a senior researcher.

Rule 2: Use the Right Question Type for Each Job

Koji supports six question types — open_ended, scale, single_choice, multiple_choice, ranking, yes_no — and each does one job well. Picking the wrong type at design time means worse data at analysis time.

The matching rules:

• Scale for numeric ratings you'll track over time (NPS, CSAT, satisfaction)
• Single_choice for mutually exclusive categorical answers (role, primary use case, journey stage)
• Multiple_choice for non-exclusive selections (features used, competitors considered)
• Ranking for explicit preference ordering
• Yes/no for binary checkpoints (have they used X, did they complete Y)
• Open-ended for everything else — discovery, "why," capturing customer language

The most common mistake is overusing open-ended when a structured type would produce cleaner data, or overusing structured types when an open-ended would surface unexpected insights. A well-designed study uses three to five different types across 8-15 questions.

Rule 3: Sequence Questions From Open to Specific

The order of questions in an AI interview matters a lot. The pattern that works:

1. Warm-up: An open-ended question that lets the participant orient themselves and the AI ("Walk me through your role and what you do day to day").
2. Discovery: Open-ended questions about behavior, workflow, or experience ("How do you currently handle X?").
3. Structured anchors: Scale or single_choice questions that capture comparable data.
4. Targeted probes: Specific yes/no, multiple_choice, or scale questions that test hypotheses.
5. Catch-all close: A final open-ended question that lets the participant raise anything important you didn't ask about ("What's the most important thing about this topic that I didn't ask?").

This sequence avoids priming early answers with structured options, and it gives the catch-all close — often where the best insights surface.

Rule 4: Set Probing Depth Deliberately

Every Koji question has a maxFollowUps setting. Defaults are reasonable but not optimal for every question.

Guidelines:

• 0 follow-ups: For demographic questions or fast pulse-checks where the answer is the answer.
• 1 follow-up (default): For most structured questions where you want a qualitative why.
• 2-3 follow-ups: For the two or three most important questions in the study — the ones whose answers determine the research outcome.

Don't set everything to 3. Interviews that probe too deeply on every question feel like interrogations and drop completion rates. Reserve depth for the questions where it actually matters.

Rule 5: Choose Mode by Audience, Not by Researcher Preference

Voice mode and text mode produce different kinds of data. Voice mode tends to produce longer, more reflective open-ended answers and feels more like a moderated interview. Text mode is faster, more comfortable for many B2B professionals, and produces cleaner data when participants want time to compose answers carefully.

The right mode depends on the audience, not the researcher:

• Consumer audiences, narrative topics: Voice mode usually wins.
• B2B professionals, technical topics, work-hours research: Text mode usually wins.
• International audiences with English-as-second-language: Text mode is gentler.
• Sensitive topics: Anonymous text mode often gets more honest answers.

If you're not sure, run a small pilot in both modes and compare the depth and completion rates before committing.

Rule 6: Recruit Like the Quality of the Answers Depends on It

It does. The single biggest determinant of insight quality in an AI interview is not the AI — it is the participant. Five carefully recruited customers will produce better research than fifty random panel respondents.

For B2B research:

• Use targeted recruitment from your CRM, customer list, or LinkedIn outreach.
• Pay incentives that match the audience (executives need $200+ to participate; power users might accept a $50 gift card).
• Pre-screen with a one-question screener to confirm relevance before booking the interview.

For consumer research:

• Recruit from your existing user base when possible — they have context the AI can work with.
• Use a participant panel only when targeting an audience you can't access directly.
• Always screen for relevance before paying the incentive.

Good recruiting is the foundation. With tools like Koji, AI moderation removes the bottleneck on running interviews — but recruitment is still where most of the work happens.

Rule 7: Use the Quality Gate, Don't Override It

Koji's analysis pipeline scores every interview on quality dimensions: completeness, depth, coherence, and engagement. Low-quality interviews are flagged automatically before they pollute your aggregated report.

Resist the temptation to override the quality gate. If 10 of your 50 interviews scored low, those 10 are giving you signal — usually one of three things:

• The screener missed a participant pool that doesn't match the audience
• The brief was unclear about what depth of answer counted as complete
• The interview length was too long for the actual content of the study

Understanding why interviews failed quality is often more valuable than rerunning the study. The quality gate is built to protect your final report from noise — let it do its job, and use the failed interviews as a research signal about your study design.

Rule 8: Pilot Every Study Before Full Launch

The single best way to catch design mistakes is to run a tiny pilot — three to five interviews — before launching the full study. Pilots catch:

• Questions that participants consistently misunderstand
• Options that don't cover the answer space (lots of "Other" responses)
• Probing instructions that produce off-tone follow-ups
• Study length that feels too long or too rushed
• Brief language that the AI is interpreting differently than you intended

After the pilot, read the transcripts (not just the aggregated themes) and adjust the brief, questions, or probing instructions before launching to the full sample.

Rule 9: Don't Pre-Code Open-Ended Questions

One of the most common mistakes in open-ended question design is hiding a structured question inside an open-ended one. "Tell me about your biggest pain point — is it pricing, ease of use, or integration?" is not an open-ended question. It is a multiple_choice question with a misleading prompt.

If you want frequency data on a known set of pain points, use multiple_choice. If you want to discover the pain points participants generate spontaneously, use open_ended with no leading options. Mixing the two corrupts both.

Rule 10: Use Scale Questions With Anchor Probing

A scale question without follow-up is just a number. A scale question with anchor: true enabled is a research artifact — the AI follows up with "What would change that?" or "Walk me through what would have made that a 9 or 10?", capturing the qualitative why behind the score.

This is the single most powerful pattern in Koji: scale + anchor probing. It compresses what would otherwise be two separate questions ("Rate it" + "Why?") into a single conversational moment, and it produces a structured number plus a synthesized qualitative summary at report time.

Default to enabling anchor probing on every scale question that matters.

Rule 11: Avoid Leading and Loaded Questions

The AI moderator follows the script you write. If your questions lead participants toward a desired answer, the data will reflect that bias. Watch for:

• Presupposing answers: "What did you love about onboarding?" presupposes love. "Walk me through your onboarding experience" is neutral.
• Loaded terms: "How frustrating was the slow loading?" is loaded. "How did the loading experience compare to what you expected?" is neutral.
• Confirmatory probing instructions: Instructions like "Probe for the pricing concern" can bias the AI toward fishing for that specific concern. Instead, write "Probe for the most important factor in their decision."

Research bias is a deep topic — Koji's separate guide on research bias covers this in more detail. The short version: write questions like you're a journalist, not a marketer.

Rule 12: Run Studies in Waves, Not Months-Long Continuous Modes

The biggest mistake teams make with always-on AI interviews is letting them run indefinitely. The data drifts: your product changes, your customer base evolves, the language they use shifts. After three months, the early responses and the late responses are answering subtly different studies.

The better pattern:

• Run studies in time-boxed waves (1-2 weeks each).
• Refresh the brief between waves based on what you learned.
• Tag each wave with a date so you can track how answers change over time.
• For continuous discovery: think of it as weekly waves rather than always-on, with the brief refreshed weekly.

This discipline produces longitudinal data you can actually trust.

Rule 13: Chat With Your Report Before You Present It

Koji generates a structured research report from every study — but the most valuable feature is the ability to chat with your report after it's generated. This is where prep for stakeholder presentations happens.

Before presenting:

• Ask the report the questions stakeholders are likely to ask: "What did Enterprise customers say about pricing?" "Did anyone mention competitor X?"
• Probe for counter-evidence: "Are there participants who disagreed with this theme?"
• Find your strongest verbatim quotes: "Give me the most compelling quote about feature Y."
• Stress-test your headline insight: "How many participants does this finding actually rely on?"

Going into a stakeholder meeting having already chatted with the data means you'll handle every "but what about..." question with evidence — not vibes.

Rule 14: Close the Loop With Participants and Stakeholders

The last 5% of value in any research study is closing the loop. Skip this and the next study will have a harder time recruiting and getting stakeholder buy-in.

With participants:

• Send a brief thank-you message summarizing what you learned at a high level (without breaking anonymity).
• For interview participants who shared specific feature requests, share when those features ship.
• Build a participant panel of customers who've done research with you — they'll respond faster and give better answers the second time.

With stakeholders:

• Tag every research finding to the decision it informed.
• Track which findings led to shipped changes and which didn't.
• Send a quarterly "what research said and what we shipped" recap to maintain stakeholder buy-in for the research program.

This is research ops at its best — not a separate role, but a discipline every team can practice.

What These Rules Add Up To

Follow these 14 rules and your AI interviews will produce research-grade insights at a pace traditional methods can't match: 30 interviews in a week, a report ready the next day, and confidence that the findings will hold up to stakeholder scrutiny.

Skip these rules and your AI interviews will produce noisy, shallow data that confirms whatever bias you walked in with. The technology is the same in both cases — the difference is the rigor you bring to the design.

Platforms like Koji handle the heavy lifting of moderation, transcription, coding, and synthesis. Your job is to design the studies that make that automation produce trustworthy research. These 14 rules are the foundation.

Related Resources

• Structured Questions Guide: All 6 Koji Question Types
• How to Write a Research Brief
• How Koji's AI Probing Works
• Understanding Quality Scores
• Voice vs Text Interview: When to Use Each Mode
• Pilot Study in User Research
• Research Bias: The Complete Guide