{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-06-01T07:52:01.678Z"},"content":[{"type":"documentation","id":"76eb2621-af32-4743-8718-34247d0a25c7","slug":"ai-interview-best-practices","title":"AI Interview Best Practices: 14 Rules for Running High-Quality AI-Moderated Customer Research","url":"https://www.koji.so/docs/ai-interview-best-practices","summary":"14 best practices for running high-quality AI-moderated customer interviews on platforms like Koji: write the brief like you're briefing a senior researcher, match each question type to the right job, sequence questions open-to-specific, set probing depth deliberately, choose mode by audience, recruit carefully, trust the quality gate, pilot before launch, avoid pre-coded open-ended questions, use scale + anchor probing, avoid leading questions, run time-boxed waves, chat with your report before presenting, and close the loop with participants and stakeholders.","content":"\n# AI Interview Best Practices: 14 Rules for Running High-Quality AI-Moderated Customer Research\n\nAI-moderated interviews are not the same as surveys, and they are not the same as moderated user interviews. They are a third research method with their own design patterns, failure modes, and best practices. Teams that get this right run 50 high-quality customer interviews in a week with a report ready by the following Monday. Teams that don't get this right ship a \"set and forget\" interview that produces shallow data and stale insights.\n\nThis guide is the practical playbook for running AI interviews that match or exceed the quality of moderated research. Fourteen rules, each with a specific recommendation and the reasoning behind it. Some are about study design, some are about question writing, some are about mode selection and recruiting. All of them are battle-tested by teams running customer research at scale on Koji.\n\nIf you're evaluating AI customer research tools or you've just shipped your first study and want to do the next one better, this is the document for you.\n\n## Rule 1: Write Like You're Briefing a Senior Researcher\n\nThe single highest-leverage thing you can do for AI interview quality is write a clear research brief. The brief is what the AI moderator uses to decide when to probe deeper, when to move on, and how to interpret ambiguous answers.\n\nGood briefs are specific about:\n- **The research goal**: What decision will this research inform?\n- **The hypothesis**: What do you believe, that this research might confirm or refute?\n- **The target audience**: Who should be in this study, and what background do they bring?\n- **What \"good\" looks like**: What kinds of answers would be especially useful to capture verbatim?\n- **What's out of scope**: Topics not to probe on, even if participants mention them.\n\nIf your brief reads like a survey title (\"Customer Satisfaction Q4\"), the AI has no anchor for judgment. If it reads like an internal Slack thread to a senior researcher (\"We're trying to figure out why churn spiked among Pro-tier customers in September — specifically whether the pricing change or the UI redesign drove it\"), the AI moderator has the context to interview like a senior researcher.\n\n## Rule 2: Use the Right Question Type for Each Job\n\nKoji supports six question types — `open_ended`, `scale`, `single_choice`, `multiple_choice`, `ranking`, `yes_no` — and each does one job well. Picking the wrong type at design time means worse data at analysis time.\n\nThe matching rules:\n- **Scale** for numeric ratings you'll track over time (NPS, CSAT, satisfaction)\n- **Single_choice** for mutually exclusive categorical answers (role, primary use case, journey stage)\n- **Multiple_choice** for non-exclusive selections (features used, competitors considered)\n- **Ranking** for explicit preference ordering\n- **Yes/no** for binary checkpoints (have they used X, did they complete Y)\n- **Open-ended** for everything else — discovery, \"why,\" capturing customer language\n\nThe most common mistake is overusing open-ended when a structured type would produce cleaner data, or overusing structured types when an open-ended would surface unexpected insights. A well-designed study uses three to five different types across 8-15 questions.\n\n## Rule 3: Sequence Questions From Open to Specific\n\nThe order of questions in an AI interview matters a lot. The pattern that works:\n\n1. **Warm-up**: An open-ended question that lets the participant orient themselves and the AI (\"Walk me through your role and what you do day to day\").\n2. **Discovery**: Open-ended questions about behavior, workflow, or experience (\"How do you currently handle X?\").\n3. **Structured anchors**: Scale or single_choice questions that capture comparable data.\n4. **Targeted probes**: Specific yes/no, multiple_choice, or scale questions that test hypotheses.\n5. **Catch-all close**: A final open-ended question that lets the participant raise anything important you didn't ask about (\"What's the most important thing about this topic that I didn't ask?\").\n\nThis sequence avoids priming early answers with structured options, and it gives the catch-all close — often where the best insights surface.\n\n## Rule 4: Set Probing Depth Deliberately\n\nEvery Koji question has a `maxFollowUps` setting. Defaults are reasonable but not optimal for every question.\n\nGuidelines:\n- **0 follow-ups**: For demographic questions or fast pulse-checks where the answer is the answer.\n- **1 follow-up (default)**: For most structured questions where you want a qualitative why.\n- **2-3 follow-ups**: For the two or three most important questions in the study — the ones whose answers determine the research outcome.\n\nDon't set everything to 3. Interviews that probe too deeply on every question feel like interrogations and drop completion rates. Reserve depth for the questions where it actually matters.\n\n## Rule 5: Choose Mode by Audience, Not by Researcher Preference\n\nVoice mode and text mode produce different kinds of data. Voice mode tends to produce longer, more reflective open-ended answers and feels more like a moderated interview. Text mode is faster, more comfortable for many B2B professionals, and produces cleaner data when participants want time to compose answers carefully.\n\nThe right mode depends on the audience, not the researcher:\n- **Consumer audiences, narrative topics**: Voice mode usually wins.\n- **B2B professionals, technical topics, work-hours research**: Text mode usually wins.\n- **International audiences with English-as-second-language**: Text mode is gentler.\n- **Sensitive topics**: Anonymous text mode often gets more honest answers.\n\nIf you're not sure, run a small pilot in both modes and compare the depth and completion rates before committing.\n\n## Rule 6: Recruit Like the Quality of the Answers Depends on It\n\nIt does. The single biggest determinant of insight quality in an AI interview is not the AI — it is the participant. Five carefully recruited customers will produce better research than fifty random panel respondents.\n\nFor B2B research:\n- Use targeted recruitment from your CRM, customer list, or LinkedIn outreach.\n- Pay incentives that match the audience (executives need $200+ to participate; power users might accept a $50 gift card).\n- Pre-screen with a one-question screener to confirm relevance before booking the interview.\n\nFor consumer research:\n- Recruit from your existing user base when possible — they have context the AI can work with.\n- Use a participant panel only when targeting an audience you can't access directly.\n- Always screen for relevance before paying the incentive.\n\nGood recruiting is the foundation. With tools like Koji, AI moderation removes the bottleneck on running interviews — but recruitment is still where most of the work happens.\n\n## Rule 7: Use the Quality Gate, Don't Override It\n\nKoji's analysis pipeline scores every interview on quality dimensions: completeness, depth, coherence, and engagement. Low-quality interviews are flagged automatically before they pollute your aggregated report.\n\nResist the temptation to override the quality gate. If 10 of your 50 interviews scored low, those 10 are giving you signal — usually one of three things:\n- The screener missed a participant pool that doesn't match the audience\n- The brief was unclear about what depth of answer counted as complete\n- The interview length was too long for the actual content of the study\n\nUnderstanding why interviews failed quality is often more valuable than rerunning the study. The quality gate is built to protect your final report from noise — let it do its job, and use the failed interviews as a research signal about your study design.\n\n## Rule 8: Pilot Every Study Before Full Launch\n\nThe single best way to catch design mistakes is to run a tiny pilot — three to five interviews — before launching the full study. Pilots catch:\n- Questions that participants consistently misunderstand\n- Options that don't cover the answer space (lots of \"Other\" responses)\n- Probing instructions that produce off-tone follow-ups\n- Study length that feels too long or too rushed\n- Brief language that the AI is interpreting differently than you intended\n\nAfter the pilot, read the transcripts (not just the aggregated themes) and adjust the brief, questions, or probing instructions before launching to the full sample.\n\n## Rule 9: Don't Pre-Code Open-Ended Questions\n\nOne of the most common mistakes in open-ended question design is hiding a structured question inside an open-ended one. \"Tell me about your biggest pain point — is it pricing, ease of use, or integration?\" is not an open-ended question. It is a multiple_choice question with a misleading prompt.\n\nIf you want frequency data on a known set of pain points, use `multiple_choice`. If you want to discover the pain points participants generate spontaneously, use `open_ended` with no leading options. Mixing the two corrupts both.\n\n## Rule 10: Use Scale Questions With Anchor Probing\n\nA scale question without follow-up is just a number. A scale question with `anchor: true` enabled is a research artifact — the AI follows up with \"What would change that?\" or \"Walk me through what would have made that a 9 or 10?\", capturing the qualitative why behind the score.\n\nThis is the single most powerful pattern in Koji: scale + anchor probing. It compresses what would otherwise be two separate questions (\"Rate it\" + \"Why?\") into a single conversational moment, and it produces a structured number plus a synthesized qualitative summary at report time.\n\nDefault to enabling anchor probing on every scale question that matters.\n\n## Rule 11: Avoid Leading and Loaded Questions\n\nThe AI moderator follows the script you write. If your questions lead participants toward a desired answer, the data will reflect that bias. Watch for:\n\n- **Presupposing answers**: \"What did you love about onboarding?\" presupposes love. \"Walk me through your onboarding experience\" is neutral.\n- **Loaded terms**: \"How frustrating was the slow loading?\" is loaded. \"How did the loading experience compare to what you expected?\" is neutral.\n- **Confirmatory probing instructions**: Instructions like \"Probe for the pricing concern\" can bias the AI toward fishing for that specific concern. Instead, write \"Probe for the most important factor in their decision.\"\n\nResearch bias is a deep topic — Koji's separate guide on research bias covers this in more detail. The short version: write questions like you're a journalist, not a marketer.\n\n## Rule 12: Run Studies in Waves, Not Months-Long Continuous Modes\n\nThe biggest mistake teams make with always-on AI interviews is letting them run indefinitely. The data drifts: your product changes, your customer base evolves, the language they use shifts. After three months, the early responses and the late responses are answering subtly different studies.\n\nThe better pattern:\n- Run studies in **time-boxed waves** (1-2 weeks each).\n- Refresh the brief between waves based on what you learned.\n- Tag each wave with a date so you can track how answers change over time.\n- For continuous discovery: think of it as weekly waves rather than always-on, with the brief refreshed weekly.\n\nThis discipline produces longitudinal data you can actually trust.\n\n## Rule 13: Chat With Your Report Before You Present It\n\nKoji generates a structured research report from every study — but the most valuable feature is the ability to chat with your report after it's generated. This is where prep for stakeholder presentations happens.\n\nBefore presenting:\n- Ask the report the questions stakeholders are likely to ask: \"What did Enterprise customers say about pricing?\" \"Did anyone mention competitor X?\"\n- Probe for counter-evidence: \"Are there participants who disagreed with this theme?\"\n- Find your strongest verbatim quotes: \"Give me the most compelling quote about feature Y.\"\n- Stress-test your headline insight: \"How many participants does this finding actually rely on?\"\n\nGoing into a stakeholder meeting having already chatted with the data means you'll handle every \"but what about...\" question with evidence — not vibes.\n\n## Rule 14: Close the Loop With Participants and Stakeholders\n\nThe last 5% of value in any research study is closing the loop. Skip this and the next study will have a harder time recruiting and getting stakeholder buy-in.\n\nWith participants:\n- Send a brief thank-you message summarizing what you learned at a high level (without breaking anonymity).\n- For interview participants who shared specific feature requests, share when those features ship.\n- Build a participant panel of customers who've done research with you — they'll respond faster and give better answers the second time.\n\nWith stakeholders:\n- Tag every research finding to the decision it informed.\n- Track which findings led to shipped changes and which didn't.\n- Send a quarterly \"what research said and what we shipped\" recap to maintain stakeholder buy-in for the research program.\n\nThis is research ops at its best — not a separate role, but a discipline every team can practice.\n\n## What These Rules Add Up To\n\nFollow these 14 rules and your AI interviews will produce research-grade insights at a pace traditional methods can't match: 30 interviews in a week, a report ready the next day, and confidence that the findings will hold up to stakeholder scrutiny.\n\nSkip these rules and your AI interviews will produce noisy, shallow data that confirms whatever bias you walked in with. The technology is the same in both cases — the difference is the rigor you bring to the design.\n\nPlatforms like Koji handle the heavy lifting of moderation, transcription, coding, and synthesis. Your job is to design the studies that make that automation produce trustworthy research. These 14 rules are the foundation.\n\n## Related Resources\n\n- [Structured Questions Guide: All 6 Koji Question Types](/docs/structured-questions-guide)\n- [How to Write a Research Brief](/docs/how-to-write-research-brief)\n- [How Koji's AI Probing Works](/docs/ai-probing-guide)\n- [Understanding Quality Scores](/docs/understanding-quality-scores)\n- [Voice vs Text Interview: When to Use Each Mode](/docs/voice-vs-text-interviews)\n- [Pilot Study in User Research](/docs/pilot-study-user-research-guide)\n- [Research Bias: The Complete Guide](/docs/research-bias-guide)\n","category":"Interview Techniques","lastModified":"2026-06-01T03:28:37.341491+00:00","metaTitle":"AI Interview Best Practices: 14 Rules for Quality Research | Koji","metaDescription":"A practical playbook for running AI-moderated customer interviews that produce research-grade insights. 14 rules covering brief design, question types, probing depth, mode selection, recruiting, quality gates, and stakeholder buy-in.","keywords":["ai interview best practices","ai customer interview tips","how to run ai interviews","ai moderator best practices","ai interview design","quality ai research"],"aiSummary":"14 best practices for running high-quality AI-moderated customer interviews on platforms like Koji: write the brief like you're briefing a senior researcher, match each question type to the right job, sequence questions open-to-specific, set probing depth deliberately, choose mode by audience, recruit carefully, trust the quality gate, pilot before launch, avoid pre-coded open-ended questions, use scale + anchor probing, avoid leading questions, run time-boxed waves, chat with your report before presenting, and close the loop with participants and stakeholders.","aiPrerequisites":["Basic familiarity with AI-moderated interview platforms","Understanding of qualitative vs quantitative research"],"aiLearningOutcomes":["Design AI interview studies that produce research-grade insights","Pick the right Koji question type for each research goal","Configure AI probing depth deliberately rather than using defaults","Choose voice vs text mode based on audience characteristics","Use Koji's quality gate as a research signal rather than overriding it","Run AI interview studies in time-boxed waves for trustworthy longitudinal data"],"aiDifficulty":"intermediate","aiEstimatedTime":"14 min read"}],"pagination":{"total":1,"returned":1,"offset":0}}