All frameworks and cheat sheets

Process

How to find your AI moderator's effective session length

Process in five steps for finding an AI moderator's effective session length, from one probe placed at four depths to a session cap below the knee
Measure it again whenever your platform changes models, because the knee moves.

A five-step test for finding how many turns an AI interviewer handles before its follow-ups start to slip. You place the same checkable probe at four depths, score whether the moderator used the earlier answer correctly, plot accuracy by turn and cap sessions 20% below the drop. It takes about two hours and works on any AI-moderated interview platform, Koji included.

Where this appears

What's in this process

  1. Write one position-matched probe: A factual, checkable question with one correct follow-up. Keep it identical every time.
  2. Run it at four depths: Place it at turns 3, 10, 25 and 40 with neutral filler, at least 15 sessions per depth.
  3. Score correctness, not plausibility: Did it use the earlier answer correctly? A vague recovery counts as a miss.
  4. Plot accuracy against turn depth: Look for the knee. It nearly always comes earlier than teams expect.
  5. Cap sessions 20% below the knee: Anything past that point becomes a second session, not a longer first one.

Use this image

Free to use in your docs, decks and posts under CC BY 4.0, as long as you credit Koji with a link to this page. To embed it, paste this into your page:

html
<a href="https://www.koji.so/visuals/test-ai-moderator-effective-session-length-steps"><img src="https://www.koji.so/visuals/test-ai-moderator-effective-session-length-steps/test-ai-moderator-effective-session-length-steps.png" alt="Process in five steps for finding an AI moderator's effective session length, from one probe placed at four depths to a session cap below the knee" width="1200" height="792" /></a>
<p>Source: <a href="https://www.koji.so/visuals/test-ai-moderator-effective-session-length-steps">How to find your AI moderator's effective session length</a>, Koji</p>