{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-09-20T04:23:16.715Z"},"content":[{"type":"documentation","id":"1391a448-20a9-4f3d-8573-a3ee292aa848","slug":"participant-confidence-vs-accuracy","title":"Participant Confidence Is Not Accuracy: How Interviewer Feedback Inflates Certainty (2026)","url":"https://www.koji.so/docs/participant-confidence-vs-accuracy","summary":"Participant confidence indicates accuracy only at the first telling, before any feedback. Wells and Bradfield (1998) showed confirming feedback inflated reported certainty, view and attention among 352 witnesses who had all chosen wrongly; Douglass and Steblay (2006) replicated large effects across 20 tests and more than 2,400 witnesses. Wixted and Wells (2017) found initial confidence can be informative under clean conditions. Researchers should keep moderators blind, use neutral acknowledgements, capture confidence immediately and weight low initial confidence.","content":"**Answer first:** how confident a participant sounds tells you something about accuracy only at the **first telling**, before anyone has reacted to their answer. Once an interviewer affirms an account - \"great, that is really helpful\", \"yes, exactly\", \"a lot of people say that\" - participants become more certain, and they also start remembering that they saw more clearly and paid more attention than they did. In the classic experiment, every one of 352 witnesses had picked the wrong person, and a single sentence of confirming feedback still inflated their reported certainty. So **capture confidence at the first telling, keep your reactions neutral, and give low initial confidence far more weight than later certainty.**\n\nThis matters because research teams routinely use confidence as a filter. The customer who says \"I am absolutely sure that is why we churned\" gets quoted in the readout; the one who says \"I think it might have been the pricing, but I am not certain\" gets softened into a maybe. That ranking is only as good as the conditions under which the confidence was formed.\n\n## The experiment: \"Good, you identified the suspect\"\n\nGary Wells and Amy Bradfield's 1998 study (\"Good, you identified the suspect: Feedback to eyewitnesses distorts their reports of the witnessing experience\", *Journal of Applied Psychology* 83, 360-376) was designed to isolate the effect of a single remark.\n\n- Participants watched a security video and tried to identify the gunman from a photospread.\n- The actual gunman was **not** in the photospread, so every witness who made an identification was wrong. The analysis covered **352** of them.\n- After identifying someone, each witness received confirming feedback (\"Good, you identified the actual suspect\"), disconfirming feedback, or no feedback.\n- Witnesses were then asked about their experience of witnessing.\n\nConfirming feedback strongly inflated how certain witnesses said they had been, and it also changed their reports of the witnessing itself: how good their view had been, how much attention they had paid, how clear their memory was, and how quickly they had made the identification. None of those things had changed. Only the feedback had.\n\nRead that list again with a research interview in mind. *How certain am I? How well did I see it? How closely was I paying attention?* Those are exactly the judgements a researcher uses to decide which participant's account to trust.\n\n## It is not a one-off\n\nThe effect has been replicated many times. Douglass and Steblay's meta-analysis (\"Memory distortion in eyewitnesses: a meta-analysis of the post-identification feedback effect\", *Applied Cognitive Psychology* 20, 859-869, 2006) pooled **20 experimental tests with more than 2,400 participant-witnesses**. It found large effects of confirming feedback on retrospective certainty, view and attention, and smaller effects on so-called objective measures such as how long the person was in view.\n\nThe pattern of effect sizes is itself useful. Feedback distorts **subjective, self-assessed** judgements most. Those are also the judgements a participant makes about their own account in an interview: how sure they are, how well they remember, how representative their experience was.\n\n## The twist: initial confidence is informative\n\nIt would be easy to conclude that participant confidence is worthless. The more recent evidence says otherwise, and the difference is timing.\n\nJohn Wixted and Gary Wells reviewed the field in \"The Relationship Between Eyewitness Confidence and Identification Accuracy: A New Synthesis\" (*Psychological Science in the Public Interest* 18(1), 2017). Their conclusion was that confidence can be a useful indicator of accuracy when it is recorded under clean conditions at the moment of first identification - and loses that value once the witness has been exposed to repeated questioning, coaching or confirming feedback. The conditions they recommend include making sure the person administering the procedure does not know which answer is the expected one. On low confidence, they were emphatic:\n\n> \"Instead of being ignored, an initial expression of low confidence should take center stage -- overshadowing all other considerations -- when a jury's goal is to evaluate the reliability of a suspect ID.\" - John T. Wixted and Gary L. Wells, 2017\n\nFor a research team, the translation is direct. **First-telling confidence is data. Later confidence is partly a record of how you reacted.** And a participant who was unsure the first time and is certain by the end of the session has told you something about your interview, not about their memory.\n\n## Where confirming feedback enters a research interview\n\nModerators give confirming feedback all the time, mostly to build rapport. Common forms:\n\n| What the moderator does | What the participant hears |\n|---|---|\n| \"That is really helpful.\" | That answer is the kind we want. |\n| \"Yes, exactly.\" / \"Right.\" | You got it right. |\n| \"A lot of people have said that.\" | Your memory matches everyone else's. |\n| Visibly taking notes after some answers and not others | Those answers mattered. |\n| Moving straight to a detailed follow-up on one theme | That theme is the important one. |\n| Showing the fix: \"We are actually building that.\" | Your diagnosis was correct. |\n\nEach of these is a small \"good, you identified the suspect\". Their effect accumulates: by the end of a session, the participant's confidence in the themes you reacted to has risen, and so has their sense of how clearly they remember them. If you then ask a `scale` question - \"How confident are you that this was the main reason?\" - you are measuring partly their memory and partly your own reactions.\n\nRapport is still important; see [building rapport in interviews](/docs/building-rapport-interviews). The goal is rapport that acknowledges the person without evaluating the content: \"Thank you, that is clear\" rather than \"That is exactly what we needed.\"\n\n## A protocol for capturing confidence before you contaminate it\n\n1. **Keep the moderator blind to the hypothesis where you can.** Wixted and Wells recommend that the person administering the procedure should not know the expected answer. In research terms: the interviewer should know the topic, not the answer you hope for. See [blind analysis](/docs/blind-analysis-research) for the analysis-side version of the same idea.\n2. **Ask for confidence right after the first free-recall answer.** \"How sure are you about that?\" or a short rating scale, asked immediately, before any follow-up that signals interest.\n3. **Use neutral acknowledgements.** Replace evaluative feedback (\"great\", \"exactly\") with receipt-only feedback (\"thank you\", \"understood\", \"go on\"), used evenly across every answer.\n4. **Never tell participants how their answer compares.** No \"others said the same\", no \"that matches what we are seeing\".\n5. **Weight low first-telling confidence heavily.** If a participant hedged at first telling, report the finding with the hedge, even if they sounded certain by the end.\n6. **Do not re-ask confidence after stimulus.** Once you have shown a concept or a fix, later confidence ratings reflect the stimulus.\n\n### A worked example\n\nA product team interviews 12 churned customers. The moderator, who believes onboarding was the problem, responds warmly whenever onboarding comes up. At the end of each session participants rate \"How confident are you that onboarding was a major reason you left?\" The average is high, and the readout says churned customers are confident onboarding drove churn.\n\nRerun it with the protocol: a moderator who does not know the hypothesis, confidence captured immediately after each participant's first unprompted reason, and neutral acknowledgements throughout. Suppose now only a minority name onboarding first, and several of those hedge when asked how sure they are. The data has not got worse. The first study was measuring the moderator's enthusiasm. (This example is illustrative; the mechanism is the one measured in the studies above.)\n\n## How Koji helps\n\nHuman moderators cannot fully control their own feedback - they nod, smile, lean in and say \"great\" without noticing, and they do it differently in every session. Legacy survey tools avoid the problem only by removing the conversation entirely, which loses the reasoning behind every rating. Koji's AI-moderated interviews keep the conversation and standardise the feedback.\n\n- **Consistent, neutral acknowledgement.** Koji's interviewer is a customizable AI consultant. You can instruct it to acknowledge without evaluating, and it applies that instruction identically in the first interview and the five-hundredth. There is no warm reaction to the answers you were hoping for.\n- **Blind by design.** You decide what goes into the interviewer's instructions. Give it the topic and the questions, and keep your hypothesis out, and the interviewer genuinely does not know which answer you expect.\n- **Confidence captured at first telling.** Place a `scale` question directly after an `open_ended` recall question in the study flow, and every participant rates their certainty at the same point, before any follow-up. Koji supports six structured question types - `open_ended`, `scale`, `single_choice`, `multiple_choice`, `ranking` and `yes_no` - covered in the [structured questions guide](/docs/structured-questions-guide).\n- **Hedges survive into the analysis.** Koji's automatic thematic analysis links each theme to the verbatim quotes behind it, so \"I think it might have been pricing, but I am not sure\" reaches your report with its hedge intact instead of being summarised into a confident bullet.\n- **Quality scoring.** Every Koji interview receives a 1-5 quality score, so thin or low-effort conversations are visible before anyone quotes them.\n\nYou do not need a background in eyewitness psychology to get this right. You need an interviewer that reacts the same way to every answer - which is much easier to guarantee in software than in people.\n\n## Frequently asked questions\n\n### Is a confident participant more likely to be right?\n\nOnly when confidence is recorded at the first telling under clean conditions. Wixted and Wells (2017) concluded that initial confidence can indicate accuracy, but confidence expressed after repeated questioning, coaching or confirming feedback is much less informative.\n\n### What is the post-identification feedback effect?\n\nIt is the finding that telling witnesses their answer was correct inflates their later reports of how certain they were, how good their view was and how much attention they paid. Wells and Bradfield (1998) demonstrated it with 352 witnesses who had all made a wrong identification.\n\n### How big is the effect?\n\nLarge. Douglass and Steblay's 2006 meta-analysis pooled 20 experimental tests and more than 2,400 witnesses and found large effects on retrospective certainty, view and attention, with smaller effects on more objective measures.\n\n### Does this mean I should stop building rapport?\n\nNo. Rapport keeps participants talking openly. The change is to acknowledge the person rather than evaluate the content: say thank you and ask them to go on, rather than telling them an answer is exactly what you needed.\n\n### When should I ask participants how confident they are?\n\nImmediately after their first unprompted answer, before follow-up questions, stimuli or any sign of which answers interest you. Confidence asked at the end of a session partly reflects how the interviewer reacted during it.\n\n### What should I do with low-confidence answers?\n\nReport them with the hedge intact and give them weight. Wixted and Wells argue that an initial expression of low confidence should take center stage when judging reliability. A participant who was unsure at first and certain by the end has usually been influenced by the session.\n\n## Related Resources\n\n- [The Misinformation Effect](/docs/misinformation-effect-user-interviews) - how researcher words become part of participant memory\n- [Memory Conformity in Group Research](/docs/memory-conformity-group-research) - why the confident co-witness usually wins\n- [Interviewer Bias](/docs/interviewer-bias) - the wider set of ways the moderator shapes the data\n- [Demand Characteristics](/docs/demand-characteristics) - participants answering what they think you want\n- [Blind Analysis](/docs/blind-analysis-research) - keeping the expected answer away from the people handling the data\n- [Structured Questions Guide](/docs/structured-questions-guide) - capturing confidence with a scale question at the right moment\n","category":"Research Methods","lastModified":"2026-09-19T03:20:54.397985+00:00","metaTitle":"Participant Confidence vs Accuracy in Interviews","metaDescription":"One line of confirming feedback inflated certainty in 352 wrong witnesses. Why first-telling confidence matters and how to capture it cleanly.","keywords":["participant confidence vs accuracy","confidence accuracy relationship","post-identification feedback effect","interviewer feedback bias","confirming feedback","wells and bradfield","neutral acknowledgement interviews"],"aiSummary":"Participant confidence indicates accuracy only at the first telling, before any feedback. Wells and Bradfield (1998) showed confirming feedback inflated reported certainty, view and attention among 352 witnesses who had all chosen wrongly; Douglass and Steblay (2006) replicated large effects across 20 tests and more than 2,400 witnesses. Wixted and Wells (2017) found initial confidence can be informative under clean conditions. Researchers should keep moderators blind, use neutral acknowledgements, capture confidence immediately and weight low initial confidence.","aiPrerequisites":["Experience moderating or reviewing research interviews","A study that asks participants to explain causes or past events"],"aiLearningOutcomes":["Explain the post-identification feedback effect","Recognise confirming feedback in moderator behaviour","Capture first-telling confidence in an interview flow","Weight hedged and confident answers appropriately"],"aiDifficulty":"intermediate","aiEstimatedTime":"12 min"}],"pagination":{"total":1,"returned":1,"offset":0}}