Piloting an AI Moderator: A UX Research Lead Playbook for Testing It Against Your Own Moderators

An AI moderator pilot is a small, scored, side-by-side test of an AI interviewer against your own moderators, run on one real question before you hand it a study that matters. Done as a four-week plan with a written scorecard, it replaces the argument about whether AI can probe like a trained researcher with a result you can show your team. By the end of this playbook you can run the pilot, apply the decision rules and write the one-page memo that says where AI moderation earns a role in your practice and where it does not.

Why piloting an AI moderator matters now

Your stakeholders are already moving. Maze's 2026 report, a survey of nearly 500 researchers, designers and product professionals, found that insights are increasingly produced by product managers (39%), market researchers (35%) and marketers (23%), with quality and consistency gaps flagged along the way (Maze, Future of User Research 2026). If a PM can already launch an AI interview tool without asking you, the question for a research lead is no longer whether AI moderation shows up in your org. It is whether you set the standard it is held to.

The evidence is better than the hype and better than the skeptics. A pre-registered, between-subjects study from Columbia Business School and Wharton (317 participants: 139 AI-moderated, 24 human-moderated, 154 static interviews) reports that AI moderation matched human moderation on depth and covered more themes, while participants sounded more emotionally engaged with a live human (Deng, Liu, Toubia and Jain, arXiv, 2026). Note the small human arm of 24. Earlier work points the same direction: in a field study of 381 AI-led interviews, Chopra and Haaland found that AI interviews reached thematic saturation in small samples comparable to human-led studies, and that a majority of respondents strictly preferred AI-assisted interviews to human-led ones (Chopra and Haaland, CESifo working paper, 2023).

Then the cautions, which matter just as much. In a randomized experiment with 3,160 respondents, AI-moderated interviews produced 4.8 times as many words as standard written open-ends, but completion fell from 99.4% to 40.5%, with drop-off concentrated among people less comfortable with AI tools (Verasight, August 2026). And Nielsen Norman Group's write-up of its test of two AI interviewers concludes that AI moderation supplements human-led interviews rather than replacing them. It works for structured feedback at scale, and participants felt heard largely because the AI summarised well, although neither tool let them confirm those summaries (NN/g, AI-Moderated Interviews: If, When, and How to Use Them).

Read together, these say the same thing: the answer depends on the question you ask and the people you ask it of. A pilot on your own question with your own participants is the only way to know. The cost of skipping it is predictable. Either you ban a method your stakeholders will use anyway, or you adopt it on a vendor's demo and discover the gaps in front of a VP.

What good looks like in a pilot

Treat it like a study with a hypothesis, not a tool trial. Four things separate a pilot people believe from one they argue about.

One real question, not a demo topic. Use a study you were going to run anyway: next quarter's discovery round, a post-launch round, a churn question. A pilot on a toy topic tells you nothing about the topic that matters.

The same brief and the same guide in both arms. The only thing that differs is who asks. If your moderators improvise and the AI follows a script, you are comparing guides, not moderators.

Criteria and thresholds written before fieldwork. Pick the scorecard and the pass marks in week one. Writing them after you have read the transcripts is how a pilot becomes a confirmation exercise.

Blind scoring. Strip the arm labels from transcripts and have two people score them. If one of the two reviewers is your most skeptical moderator, even better.

Here is the plan I would run, counted back from the Monday your next generative round starts:

• T minus 4 weeks: choose the question, write the one-line brief and the interview guide, and write the scorecard with pass marks.
• T minus 3 weeks: build the study, preview it, recruit from one source.
• T minus 2 weeks: field both arms in the same five days. Aim for 12 AI interviews and 6 human interviews.
• T minus 1 week: score blind, trace findings to quotes, read the drop-offs.
• T minus 0: write the one-page memo and decide, per question type, who moderates.

Why 12 and 6? NN/g notes that there is no golden number for interview studies and that five participants are often too few for exploratory work; the stopping rule is saturation, the point where new interviews add little (NN/g, How Many Participants for a UX Interview?). Six human interviews are enough to see the main themes your own moderators produce, and twelve AI interviews are cheap enough to give the AI arm room to show or lose its themes. See also data saturation in qualitative research.

The scorecard. Five criteria, each with a pass mark. These thresholds are Koji's rules of thumb, compiled from the studies above and our reading of how research teams judge moderation. They are not a published standard, so change them if your team has stricter ones, but write them down before you start.

1. Theme coverage. Code the human arm into themes first. The AI arm passes if at least 80% of those themes also appear in it. Also list the AI-only themes: that is where the extra reach shows up.
2. Probe depth. On three core questions, count the transcripts where a vague first answer got a follow-up that produced a concrete example. The AI arm passes if its rate is within 20 percentage points of the human arm.
3. The surprise test. Did either arm surface something you did not expect? If only the human arm did, keep humans for exploratory work.
4. Traceability. Pick 10 findings from the AI report and trace each to the quotes behind it. Pass is 10 of 10. One miss means you fix the guide or the brief before using it.
5. Participant experience. Compare completion between arms and add one closing question about how the conversation felt. A completion rate far below the human arm is a finding, not noise. The Verasight result above is why.

Common mistakes:

• Piloting on a topic nobody will act on.
• Letting the human moderators improvise while the AI follows a script, or the reverse.
• Judging transcripts when you know which arm they came from.
• Comparing speed and cost only. Cheap, wrong answers are still wrong.
• Running 40 AI interviews against 3 human ones and calling the difference a result.
• Skipping the participant side: who dropped out, and where.

How to set up the AI arm in Koji

You need a plan that fits the pilot. Text interviews work on every plan. Voice interviews need the Interviews plan or Enterprise. A 12-interview text pilot costs 12 credits and a 12-interview voice pilot costs 36 (text is 1 credit and voice is 3 per completed interview), and conversations scoring below 3 out of 5 on quality are not charged (how the quality gate works). The Insights plan includes 29 credits a month, which covers a text pilot with room to spare.

1. Start a study from the template that matches your real question. Open the template gallery in the dashboard, filter to UX Researchers, and pick the closest fit (for example Evaluative Post-Test Interviews for a usability debrief, or a discovery template for generative work). Or paste the one-line brief below into the box that asks what you want to research.
2. Shape the research brief with the same guide your moderators use. Put your decision, your hypothesis and your participant description in the brief, then paste in the question list. Add guardrails for anything the interviewer must not say, for example your roadmap.
3. Add structured questions where you want a number. Use a 1 to 10 scale and a single choice question where your human guide has one, each with an open follow-up, so both arms produce comparable numbers.
4. Choose text or voice and run a preview. Pick the mode your human arm uses. A preview interview is saved with its own analysis but is not counted or charged, so read it as a participant and fix awkward questions before you invite anyone.
5. Invite from the same source and cap the arm. Send the interview link to the participants you assigned to the AI arm, alternating invitations between arms so assignment is not left to chance. Set a response limit of 12.
6. Read the report, then the transcripts. Each interview gets a quality score from 1 to 5 with relevance, depth and coverage, and the report shows themes and findings with the supporting quotes. Export responses to CSV on any plan to score blind in your own spreadsheet.

Copy this interview guide

The guide below works for most discovery pilots. Replace the bracketed parts with your topic. Paste it into any tool, including your human moderators' notes.

Opening line: "Thanks for doing this. I'm interested in how you actually handle [topic], so there are no right answers. Tell me about real situations rather than what you think should happen."

1. "Tell me about the last time you [did the task]. Walk me through it from the start." (open) Probe: "What happened right before that?"
2. "How often do you do this?" (single choice: daily, weekly, monthly, rarely) Probe: "What makes it more or less frequent?"
3. "What was the hardest part of that time?" (open) Probe: "Can you give me a specific example?"
4. "What did you try to work around it?" (open) Probe: "How well did that work?"
5. "How satisfied were you with how it turned out?" (1 to 10) Probe: "What would have moved that number up by two?"
6. "Who else was involved, and what did they need from you?" (open) Probe: "What did they say when it went wrong?"
7. "If you could change one thing about how this works, what would it be?" (open) Probe: "What would that make possible?"
8. "Is there anything I should have asked and did not?" (open) Probe: "Say more about that."

Closing question (pilot only): "How did this conversation compare with talking to a person about the same thing?" (open)

One-line brief to paste into Koji: "Find out how [audience] handle [task] today, what slows them down, and what workarounds they use, so we can decide [decision]."

In a pilot, the closing question is your data on the participant side. Read those answers next to the completion numbers.

What you get

For the AI arm: a quality score and analysis per interview, themes across interviews, high-impact findings with the quotes behind them, and CSV export of the responses. For the pilot as a whole: a scorecard filled in and a memo.

How a research lead uses it the next day:

• In the team meeting: put the scorecard on screen. The conversation moves from "I don't trust it" to "it passed four of five, and here is what it missed."
• In the memo: state per question type who moderates. A common outcome is AI for structured, repeatable rounds such as post-launch feedback and screening conversations, and humans for open exploration and sensitive topics.
• With your PMs: hand them a guide template that has passed your own pilot. That is the practical answer to the democratisation worry in the Maze data above. Read more in research democratization.

The same playbook elsewhere

Any team adopting a new method can run this pilot. HR teams test an AI exit interviewer against their HRBPs before rolling it out, and customer success teams test it on a renewal-risk conversation against their CSMs. A neighbouring researcher playbook, validating synthetic user findings with real participants, uses the same side-by-side logic for a different question: does an AI persona agree with real people? All researcher playbooks are on the use-case library page for researchers.

Why Koji for an AI moderator pilot

The table compares the three ways a research lead can collect the "why" at this scale. Human moderators are still the right choice for many studies, and written open-ends are still the cheapest way to count things.

Reasons that hold up in the product:

• A study you can test before you pay. Previews are saved but not counted or charged, so you can refine the guide before the pilot starts.
• Control over the interviewer. Guardrails and the research brief define what the interviewer covers and avoids. That makes the pilot a test of your guide, not of a black box.
• Comparable numbers inside conversations. Scale, choice and yes or no questions sit next to open ones, with a follow-up setting for each.
• Evidence you can audit. Every finding opens to the supporting quotes, and every interview has a quality score with its parts, so the traceability check takes minutes.
• Quality gate on cost. Conversations scoring below 3 out of 5 are not charged, so a bad interview does not cost you.

For a broader look at the tool market see AI-moderated interview platforms compared, and for the decision framework behind this playbook see AI vs human moderators.

Questions you might have about AI interviews

Can an AI really probe like a trained moderator? Sometimes, and that is what the pilot measures. The academic work above reports depth and theme coverage comparable to human moderation in structured studies, and NN/g found the limits show up in open exploration. Koji's interviewer follows up on your questions according to the follow-up setting on each one and the guardrails you set. Score probe depth on your own transcripts instead of taking anyone's word for it.

Will participants talk to it, and will they be candid? Some will not, and the Verasight data above shows completion can fall sharply. Offer text or voice, say in the invitation how long it takes and what happens to the data, and compare completion between arms. Chopra and Haaland found a majority of respondents preferred AI-assisted interviews in their setting, so measure it in yours.

Where does participant data go? Koji states EU data residency. If your team needs control of the model keys, Bring Your Own Key is part of the Enterprise plan (bring your own key). Check your own policy on recording consent before any study.

When Koji may not be the right fit

If your question is exploratory and emotionally loaded, such as grief, health or workplace conflict, keep human moderators and use the pilot to confirm that the line is where you think it is. If you need to watch behaviour, in-person observation or a shared screen with a facilitator, Koji is not a replacement. And if your team cannot spare 18 interviews and two reviewers for a fortnight, finish a smaller pilot with six AI interviews and three human ones rather than skipping the scorecard.

Metrics to track

• Completion rate in each arm, and the step where people leave
• Share of the human arm's themes found in the AI arm
• AI-only themes, counted separately
• Probe depth rate on three core questions
• Findings traced to quotes out of 10
• Quality score distribution and the share below 3
• Participant rating of the conversation from the closing question

Frequently asked questions

How many interviews does the pilot need? Twelve AI and six human interviews on one question is enough to see the main themes in each arm. Use saturation as your guide and add more if new themes keep appearing.

Should the pilot use text or voice? Use the mode your human arm uses. If your moderators run video calls, try voice for the AI arm. Voice needs the Interviews plan or Enterprise, and costs 3 credits per completed interview instead of 1.

Do I need a separate guide for the AI arm? No. Use the same guide so the only difference is who asks. Add guardrails if the interviewer must avoid anything your human moderators know to avoid.

What if the AI arm fails a criterion? Fix the cause first. Often it is the guide, such as a vague question. Re-run the failed criterion on a new batch of six. If it still fails, write that into the memo and keep humans for that question type.

Can I run the pilot without a research budget? Yes. A text pilot costs 12 credits, and the Insights plan includes 29 a month.

Run your pilot this month

Open the template gallery in your Koji dashboard, pick the closest UX Researchers template or paste the one-line brief above, and preview it before you invite anyone. For more on how researchers use Koji, see the researchers page, and for the method behind the decision see AI vs human moderators and understanding quality scores.