{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-08-06T08:57:41.957Z"},"content":[{"type":"documentation","id":"fde0fa1e-9b5f-4986-818b-b3fe53b6f072","slug":"ai-conversation-memory-long-interviews","title":"Conversation Memory and Long-Session Degradation: Why AI Interviews Get Worse After Turn 20 (2026)","url":"https://www.koji.so/docs/ai-conversation-memory-long-interviews","summary":"Long AI-moderated interviews degrade measurably: models lose an average 39% of single-turn performance in multi-turn conversation, attend worst to the middle of a context, and do not recover from early errors. This guide gives the evidence base, a session budget calculation, the four observable symptoms, a position-matched probe test to find your effective session length, and the design patterns that mitigate it.","content":"**Short answer: an AI moderator is not equally good at turn 40 as it was at turn 4, and the degradation is measurable, predictable, and mostly designable-around.** Across the published evidence, large language models lose an average of 39% of their single-turn performance once a task is spread over a multi-turn conversation, most of that loss coming from a collapse in reliability rather than in raw capability. They attend well to the beginning and the end of a long context and poorly to the middle. And once they take a wrong turn, they do not recover on their own. For research, that means your interview guide - warm-up first, hard questions in the middle, wrap-up last - puts your most valuable questions in the model attention trough. This guide covers what the evidence actually shows, the four symptoms you can watch for in real transcripts, a session budget you can compute for your own study, and the test protocol that tells you where your moderator starts to slip.\n\n## The problem nobody instruments\n\nTeams evaluate AI moderators the way they evaluate a chatbot demo: they ask it three questions, the answers are good, and they ship it. Then they field a 25-minute interview with 12 questions and adaptive follow-ups, and somewhere around the two-thirds mark the moderator starts asking things the participant already answered, drops a required probe, or quietly stops enforcing the study goal.\n\nNobody catches it, because the two places researchers actually read transcripts are the opening (does it introduce itself properly?) and the closing (did it wrap up?). Those are precisely the two regions where the model performs best. The failure lives where nobody looks.\n\nThis is not a vendor problem or a prompt problem. It is a well-documented property of how current language models use long contexts, and it has been measured in three independent lines of research.\n\n## What the evidence actually shows\n\n### Finding 1: performance is U-shaped across the context\n\nIn *Lost in the Middle: How Language Models Use Long Contexts* (Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang, *Transactions of the Association for Computational Linguistics*, vol. 12, pp. 157-173, 2024), the authors placed a single relevant document at different positions inside a long input and measured whether the model could use it.\n\nThe result was a U-shaped curve: performance is highest when the relevant information sits at the very beginning of the context (primacy bias) or the very end (recency bias), and degrades sharply in the middle. The magnitude is not subtle. The paper reports that multi-document question-answering performance for one model **dropped by more than 20% depending only on where the answer was placed**, and that in the worst case the 20- and 30-document settings scored *lower than giving the model no documents at all* - below the closed-book baseline of 56.1%.\n\nTwo secondary findings matter just as much for research design. First, extended-context models were not reliably better at using their context; the authors found models often had identical performance to their own extended-context counterparts. Second, adding more material has sharply diminishing returns: going from 20 retrieved documents to 50 improved performance by only about 1.5%. The authors conclude that current models **cannot effectively reason over their entire context window when prompted for downstream tasks**.\n\n### Finding 2: the drop is bigger in conversation than in one long prompt\n\n*LLMs Get Lost In Multi-Turn Conversation* (Philippe Laban and colleagues, 2025) took the same underlying tasks and split them two ways: once as a single fully-specified instruction, and once as an underspecified conversation where the requirements arrive in pieces across turns - exactly how a real interview works.\n\nAnalysing over 200,000 simulated conversations across six generation tasks and every top open- and closed-weight model they tested, they found **an average performance drop of 39% in the multi-turn setting**. Decomposing it produced the finding that should change how you evaluate a moderator: the loss was only a minor decline in aptitude, but a large increase in *unreliability*. The model does not get much dumber. It gets much more variable.\n\nThe mechanism they describe is the one you will recognise from bad transcripts: models make assumptions in early turns, prematurely commit to an answer, and then over-rely on it. In their words, **when LLMs take a wrong turn in a conversation, they get lost and do not recover**.\n\n### Finding 3: the effective context is far shorter than the advertised one\n\n*NoLiMa: Long-Context Evaluation Beyond Literal Matching* (Adobe Research, 2025) rebuilt the standard needle-in-a-haystack test so that the question and the target share almost no literal words, forcing the model to infer the connection rather than pattern-match a keyword. Evaluating 13 models that all claim support for at least 128K tokens, **11 of them fell below 50% of their own short-context baseline at just 32K tokens**. One widely used 70B model scored 94.3% at short length and 42.7% at 32K, giving it an effective length of roughly 2K.\n\nThat gap between advertised context and effective context is the single most important number for anyone running long AI-moderated sessions. A model advertised at 200K tokens may hold a research brief reliably in mind for a small fraction of that.\n\n| Evidence | What was measured | Headline result | Consequence for interviews |\n| --- | --- | --- | --- |\n| Liu et al., TACL 2024 | Position of relevant info in a long context | U-shaped curve; >20% swing; worst case below the 56.1% no-document baseline | Mid-guide questions are served worst |\n| Laban et al., 2025 | Single-turn vs multi-turn, 200,000+ conversations | 39% average drop, driven by unreliability | Variance, not incompetence, is the failure mode |\n| NoLiMa, 2025 | Long-context retrieval without lexical overlap | 11 of 13 models below 50% of baseline at 32K | Effective context is much shorter than advertised |\n\n## Why research interviews are the worst case\n\nMost long-context benchmarks are retrieval tasks: the answer is somewhere in the input, go find it. An AI-moderated interview is harder in four specific ways.\n\n**The instructions are at the front and never repeat.** Your research brief, your question list, your do-not-do rules and your tone guidance are loaded once, at position zero. Everything the participant says lands after them. As the transcript grows, the brief recedes further into the region the model handles worst.\n\n**The task is underspecified by design.** Good qualitative research does not fully specify the path in advance; it follows the interesting thread. That is precisely the setting Laban et al. found produces the 39% drop.\n\n**Errors are unobserved and uncorrected.** In a coding benchmark, a wrong answer fails a test. In an interview, a badly framed probe just produces a slightly worse answer, and the moderator keeps going. There is no signal to correct on.\n\n**The output is the input.** Every answer the participant gives is appended to the context, so the interview is a system that degrades its own operating conditions with each turn. A long, rich, high-quality interview creates exactly the conditions under which the moderator performs worst. That is the uncomfortable core of the problem: **depth and reliability trade off against each other inside a single session.**\n\n## The session budget\n\nHere is a piece of arithmetic worth doing for your own study before you field it.\n\nAn AI moderator context contains four things. Three are fixed; one grows.\n\n| Component | Typical size | Behaviour |\n| --- | --- | --- |\n| System prompt and moderator persona | 600-1,200 tokens | Fixed |\n| Research brief, goals, do-not-ask rules | 500-1,500 tokens | Fixed |\n| Question list and structured question definitions | 400-1,200 tokens | Fixed |\n| Accumulated interview transcript | 0 to 15,000+ tokens | Grows every turn |\n\nCall the fixed portion roughly 2,500 tokens. A conversational turn pair - one probe plus one substantive answer - runs about 150 to 250 tokens in text and considerably more in voice, where people speak in longer, less edited sentences.\n\nAt turn 5, the transcript is around 1,000 tokens and your brief is about 70% of everything the model is looking at. At turn 20, the transcript is around 4,000 tokens and the brief is down to 38%. At turn 40 the transcript is 8,000 tokens and the brief is 24%. By turn 60 it is under 17%.\n\n**The moderator is paying proportionally less attention to your research goals with every single answer it receives, and the decline is steepest in exactly the sessions you would call successful.** Nothing in the product tells you this is happening. The transcript looks fine. The interview completes. The report generates.\n\nThis is not an argument for short interviews. It is an argument for knowing your number, and for designing the session so that the questions you care about most are not the ones being asked at 17%.\n\n## The middle-of-the-interview problem\n\nNow combine the session budget with the U-shape, and you get the finding that should change your interview guides.\n\nStandard interview craft, taught everywhere and correct for human moderators, says: open with easy warm-up questions to build rapport, put your hardest and most important questions in the middle once the participant is comfortable, and close with a wrap-up and a catch-all. It is good advice about human attention and human rapport.\n\nApplied to an AI moderator, it systematically places your highest-value questions in the model attention trough.\n\nThe first two or three questions get primacy. The final wrap-up gets recency, and it is also the shortest distance from the current turn. Questions four through nine of a twelve-question guide sit in the middle of a context that is by then dominated by transcript - and those are the questions the study was commissioned to answer.\n\n**The fix is not to abandon warm-ups. It is to stop assuming that guide position is neutral.** Three practical corrections:\n\n1. **Re-anchor the goal, do not just state it once.** A moderator that reminds itself of the study objective before the mid-guide questions is re-establishing recency for the instruction that matters. This is a system-design property, not something you can prompt your way out of from the participant side.\n2. **Front-load the single question you would keep if you could only ask one.** If a study lives or dies on one question, question two is a better position for it than question six. This directly contradicts conventional guide structure, and the evidence supports it.\n3. **Make mid-guide questions structurally explicit rather than conversational.** A scale or single-choice question does not depend on the model synthesising the brief from a distant context. It depends on the model asking a defined question and recording a defined answer. Structure is the cheapest defence against attention decay.\n\n## The contamination window\n\nLaban et al.s finding that models *do not recover* has a specific and expensive consequence for research: interview quality is not independent across turns.\n\nIf the moderator misreads the participant at turn 8 - decides they are a churned customer when they are an active one, or locks onto the wrong feature - that misreading is now in the transcript, and the transcript is now the context. Every subsequent turn conditions on the error. The probes narrow around a false premise. The participant, being cooperative, answers the questions they are asked.\n\nCall the region from the first uncorrected error to the end of the session the **contamination window**. It is the correct unit of analysis, and it means:\n\n- **Per-turn quality scores understate the damage.** Each individual probe after the error may look reasonable in isolation. The problem is the premise they all share.\n- **The metric you want is conditional.** Not \"what fraction of probes were good\" but \"given an error at turn N, what fraction of the remaining turns were on-brief?\"\n- **Early errors are worth far more than late ones.** An error at turn 5 of a 40-turn interview contaminates 88% of the session. An error at turn 35 contaminates 12%. Weight your quality review accordingly - read the first third of your transcripts most carefully, not the last.\n\n## The four symptoms, and how to spot them\n\nYou do not need instrumentation to detect degradation. You need to know what to search your transcripts for.\n\n| Symptom | What it looks like in the transcript | Where it clusters |\n| --- | --- | --- |\n| Repetition | The moderator asks something the participant already answered, sometimes verbatim | Late-session, after long answers |\n| Brief drift | Probes wander into topics the study explicitly excluded, or chase a tangent for 5+ turns | Mid-to-late, after a rich digression |\n| Coverage failure | A required question from the guide is never asked, or is asked so obliquely it does not return an answer | Mid-guide, questions 4-9 |\n| Premise lock | Every probe from some point on assumes something the participant did not say | Anywhere; contaminates everything after |\n\n**Repetition is the cheap canary.** It is trivially detectable with string similarity across moderator turns, it correlates with the other three, and it is the one symptom participants will complain about unprompted. If your completion comments include any version of \"it asked me the same thing twice\", you have a degradation problem and the other three symptoms are almost certainly present too.\n\n## How to test your own moderator\n\nThis is a two-hour test that will tell you more about your platform than any vendor benchmark.\n\n**Step 1. Build a position-matched probe.** Write one factual, checkable question - something with an unambiguously correct follow-up, such as \"which of the three plans did you say you were on?\" Do not change the question.\n\n**Step 2. Run it at four depths.** Field the same study four times with the probe placed at turn 3, turn 10, turn 25 and turn 40, padding with neutral filler questions so that only position varies. Run at least 15 sessions per depth.\n\n**Step 3. Score correctness, not plausibility.** Did the moderator use the earlier answer correctly, yes or no? Plausible-sounding recovery (\"tell me more about your plan\") scores as a miss - it is the model routing around a failure it cannot see.\n\n**Step 4. Plot accuracy against turn depth.** You are looking for the knee. Most teams find one, and it is almost always earlier than they expected.\n\n**Step 5. Set your session cap 20% below the knee** and design anything past that point as a second session rather than a longer first one.\n\nThe number this produces is your **effective session length**, and it is a property of your platform, your prompt and your study - not a number a model card can give you. It is also the number to re-measure every time your platform changes models, because the knee moves.\n\n## Design patterns that actually help\n\n**Split long studies into linked short sessions.** Two 12-minute sessions with the same participant produce two short contexts instead of one long one. You pay a small re-recruitment cost and buy back the entire degradation curve. For longitudinal work this is free - you were running multiple waves anyway.\n\n**Use structured questions for anything load-bearing.** An open-ended question asked at turn 30 depends on the model correctly synthesising a distant brief. A scale question asked at turn 30 depends on it asking a defined question and recording a number. The second is dramatically more robust to context depth, and it is the reason the most reliable AI research designs are hybrids rather than pure conversation.\n\n**Cap adaptive follow-ups per question.** Unbounded probing is how a 12-question guide becomes a 60-turn session. A limit of two follow-ups per question keeps depth where you asked for it instead of wherever the conversation happened to get interesting.\n\n**Treat coverage as a hard gate, not an aspiration.** The system should know which questions have been answered and refuse to close the session while required ones are outstanding. This is the difference between a moderator that hopes to cover the guide and one that is structurally incapable of skipping it.\n\n**Never evaluate a moderator on a three-question demo.** The demo tests the region where every model performs best.\n\n## How Koji handles this\n\nKoji is built around the assumption that conversational depth and instrument reliability have to be engineered against each other, not traded off by accident.\n\n**Structured questions carry the load.** Koji supports six first-class question types, and each one has a defined answer shape that does not depend on the model reconstructing your intent from a distant brief:\n\n| Type | What it captures | Why it resists context decay |\n| --- | --- | --- |\n| `open_ended` | Free-form qualitative answer with AI follow-up probing | Most expressive, most context-dependent - use where depth matters most |\n| `scale` | Numeric rating such as 1-10 satisfaction or NPS | Defined answer space; correctness is checkable |\n| `single_choice` | One option from a list | Options travel with the question, not the brief |\n| `multiple_choice` | One or more options from a list | Same, with multi-select |\n| `ranking` | Items ordered by preference | Fully specified task; no synthesis required |\n| `yes_no` | Binary answer | Maximum robustness at maximum depth |\n\nEvery question carries a stable ID that follows it from interview plan to AI interviewer to analysis to report aggregation, which is what makes coverage checkable rather than assumed. A question that was never asked is visible as a gap in the data, not invisible as an absence in a transcript.\n\n**Coverage is enforced, not hoped for.** Because the study questions are first-class objects rather than lines in a prompt, the moderator knows what remains outstanding. Adaptive probing happens inside a guide it cannot silently abandon.\n\n**Quality scoring is per interview, not per platform.** Koji scores each completed interview on a 1-5 scale against your research goals with a written rationale, so a session where the moderator drifted shows up as a low-scoring interview you can read, rather than an average that quietly absorbs it. Sort your completed interviews by score and read the bottom decile: that is your degradation sample, delivered without you having to build the instrumentation.\n\n**Sessions are cheap enough to split.** The reason teams run one long interview instead of two short ones is that human moderator time makes the second session expensive. With AI moderation running in parallel and around the clock, the marginal cost of a second session is close to zero, which makes the single best defence against long-session degradation also the cheapest one available.\n\nTraditional survey tools sidestep this problem by never having a conversation at all - a SurveyMonkey form has no context to lose, and also no ability to probe. Human-moderated research sidesteps it too, at roughly 60 to 90 minutes of skilled labour per session plus scheduling. The AI-native position is the interesting one: conversational depth *and* a structural guarantee that the guide gets covered, which is only achievable if the platform treats questions as data rather than as text in a prompt.\n\n## What to measure going forward\n\nAdd four numbers to your research operations dashboard. None of them require new tooling.\n\n1. **Effective session length.** The turn depth at which position-matched probe accuracy falls off. Re-measure after every platform or model change.\n2. **Repetition rate.** Share of sessions containing a near-duplicate moderator turn. Cheap to compute, strongly correlated with everything else.\n3. **Coverage rate.** Share of required questions actually answered, by guide position. If questions 4-9 underperform questions 1-3 and 10-12, you are seeing the U-shape in your own data.\n4. **Conditional quality after first error.** For the sessions your reviewers flag, what share of subsequent turns stayed on-brief. This is the contamination window, measured.\n\nIf you only adopt one: measure coverage by guide position. It costs nothing, it uses data you already have, and it will show you the trough.\n\n## Frequently asked questions\n\n### Does a bigger context window fix long-session degradation?\n\nNo, and the evidence on this is unusually clear. Liu et al. found that models often had identical performance to their own extended-context counterparts, meaning a larger window did not translate into better use of that window. NoLiMa went further: 11 of 13 models that advertise at least 128K tokens of context fell below half their short-context baseline at 32K. Advertised context is a capacity limit, not a performance guarantee. What matters for research design is effective context, and you have to measure that yourself.\n\n### How long is too long for an AI-moderated interview?\n\nThere is no universal number, because it depends on your platform, your prompt size and how verbose your participants are - voice interviews consume context far faster than text. The practical answer is to measure your own knee using the position-matched probe test in this guide, then set your cap 20% below it. As a starting hypothesis before you have measured, most teams find that text sessions hold up well to roughly 20-25 substantive turns and that quality becomes noticeably variable past 40.\n\n### Why does the middle of my interview guide underperform?\n\nBecause language models attend best to the beginning and the end of their context and worst to the middle, a pattern Liu et al. documented as a U-shaped performance curve. Conventional interview craft puts warm-ups first, the important questions in the middle and wrap-up last, which places your highest-value questions in the weakest attention region. Either move your single most important question earlier, or make mid-guide questions structurally explicit so they do not depend on the model synthesising a distant brief.\n\n### Is this a prompt engineering problem I can fix with better instructions?\n\nOnly partially. Better instructions help at turn 3 and matter less at turn 40, because the problem is not the quality of the instruction but its position and its shrinking share of the context. The durable fixes are structural: re-anchoring the goal mid-session, capping adaptive follow-ups, enforcing coverage as a hard gate, and splitting long studies into linked short sessions. Prompt quality is necessary and not sufficient.\n\n### What is the difference between this and the model just hallucinating?\n\nThey are different failure modes with different fixes. A hallucination is the model asserting something false. Long-session degradation is the model losing track of instructions and prior turns - asking a question twice, dropping a required probe, or locking onto a premise the participant never stated. Degradation is more dangerous in research precisely because it produces no obviously wrong statement to catch. The transcript reads fine; the study is just answering a slightly different question than the one you commissioned.\n\n### How do I know if degradation affected a study I already ran?\n\nRun three checks on the existing transcripts. First, string-similarity across moderator turns to find repetition. Second, coverage by guide position - tabulate which required questions actually returned an answer, grouped by where they sat in the guide. Third, read the lowest-scoring 10% of interviews in full, since a moderator that drifted usually produces a session a reviewer can recognise immediately. If coverage is materially worse for mid-guide questions, the effect is present in your data and the affected questions need re-fielding rather than re-analysis.\n\n### Do voice interviews degrade faster than text?\n\nGenerally yes, for a mechanical reason: spoken answers are longer and less edited than typed ones, so each turn consumes more context. A voice session can reach the same context depth in roughly half the turns of a text session. If you run both modalities, measure the knee separately for each rather than assuming a single session cap covers both, and expect the voice cap to be the tighter of the two.\n\n## Related Resources\n\n- [Structured Questions in AI Interviews](/docs/structured-questions-guide) - the six question types and when each one is the right instrument\n- [AI Interview Best Practices](/docs/ai-interview-best-practices) - 14 rules for running high-quality AI-moderated research\n- [Evaluation Datasets for AI Products](/docs/ai-evaluation-dataset-golden-set) - building a golden set you can regression-test against\n- [Graceful Degradation for AI Features](/docs/ai-graceful-degradation-fallback-research) - what users should experience when the model cannot deliver\n- [Thumbs Up, Thumbs Down](/docs/ai-feedback-signals-thumbs-research) - what in-product AI feedback signals actually tell you\n- [How to Analyze Interview Transcripts with AI](/docs/ai-transcript-analysis-guide) - from raw conversations to actionable insights\n- [Reliability vs. Validity in Research](/docs/reliability-vs-validity-research) - the measurement vocabulary this guide depends on\n\n---\n\n**Run your first study free.** Koji gives you 10 free credits when you sign up - enough to field a real study, read real transcripts, and run the position-matched probe test on your own guide before you commit to a fielding plan.","category":"Research Methods","lastModified":"2026-08-06T03:18:23.871924+00:00","metaTitle":"Conversation Memory and Long-Session Degradation in AI Interviews (2026)","metaDescription":"AI moderators lose 39% of their performance in multi-turn conversation. The evidence, the four degradation symptoms, the session budget, and how to find your own effective session length.","keywords":["ai conversation memory","long session degradation","context window research","multi-turn ai interview","lost in the middle","ai moderator quality","effective context length","ai interview length"],"aiSummary":"Long AI-moderated interviews degrade measurably: models lose an average 39% of single-turn performance in multi-turn conversation, attend worst to the middle of a context, and do not recover from early errors. This guide gives the evidence base, a session budget calculation, the four observable symptoms, a position-matched probe test to find your effective session length, and the design patterns that mitigate it.","aiPrerequisites":["Familiarity with AI-moderated interviews","Basic research design vocabulary"],"aiLearningOutcomes":["Explain why AI moderator quality declines across a long session","Compute a session budget for your own study","Detect the four degradation symptoms in existing transcripts","Run a position-matched probe test to find your effective session length","Redesign interview guides so high-value questions avoid the attention trough"],"aiDifficulty":"intermediate","aiEstimatedTime":"14 min"}],"pagination":{"total":1,"returned":1,"offset":0}}