{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-11-16T00:29:14.237Z"},"content":[{"type":"documentation","id":"867eea81-89a2-47d6-8a36-aa5822c4ab58","slug":"understanding-quality-scores","title":"Understanding Quality Scores","url":"https://www.koji.so/docs/understanding-quality-scores","summary":"Koji automatically scores every interview from 1 to 5 across five dimensions: relevance, depth, coverage, completion, and structured quality. Only interviews scoring 3 or above count toward billing and are included in research reports. Analysis is auto-triggered on completion.","content":"Every completed or partial interview in Koji receives a quality score from 1 to 5. The score tells you how useful the interview is for your research, and it decides two things: only interviews scoring 3 or above use credits, and only those interviews go into your research report. You can see how a whole study's scores spread across four levels, and how many interviews made the report, on the **Quality Evaluation** card in your study's results, and each response's level (Shallow, Fair, Good or Excellent) in the **Evaluation** column of the results tables.\n\n## How Quality Scoring Works\n\nAfter each interview ends, whether the participant finished or stopped partway, Koji's AI automatically evaluates the conversation across multiple dimensions to produce a single quality score. The evaluation happens immediately upon interview completion and takes just a few seconds. There is no manual trigger required.\n\nThe score reflects the overall research value of the interview: not the participant's intelligence or effort, but how much actionable data the conversation produced. A high-quality interview is one that gives you clear, detailed, and relevant information you can use to make decisions.\n\n### Scores follow what the participant actually said\n\nSince September 15, 2026, the score can't be higher than the substance of the participant's own answers allows. Before that change, someone could click through a set of multiple-choice questions, answer every follow-up with a few dots, and still receive a top score. Now thin or nonsense answers to open questions and follow-ups bring the score down, so those interviews stay out of your report and you aren't charged for them.\n\nThis applies to new interviews only. Scores you already have stay the same. Studies made up entirely of structured questions are scored on their structured answers, since there is no free text to judge.\n\n### Short, relevant answers still count\n\nSince October 19, 2026, Koji also checks every interview question on its own and marks it **answered**, **off topic** or **not answered**. Follow-ups are judged together with the question they belong to.\n\n- **Answered** means the reply relates to the question and gives a real experience, preference or opinion. Length doesn't matter: \"Netflix\", \"twice a week\", a single rating or a chosen option all count.\n- **Off topic** means they replied but said nothing that answers it, such as \"ok\", \"idk\" or filler text.\n- **Not answered** means the question was never asked or never replied to.\n\nIf most of the interview questions got a real, relevant answer, and the interview as a whole stayed on topic, the score is at least 3. That interview is billed and goes into your report. Depth then decides whether it scores 3, 4 or 5. This floor stops short-answer studies from losing real respondents just because people answered in a few words, while rushed or nonsense interviews still score low.\n\nIf the analysis of an interview fails and never produces a score, that interview stays out of your report and isn't charged.\n\n## The 1-5 Scale Explained\n\nQuality scores range from 1 to 5. Here's what each score means in practice, with the label shown in an interview's analysis drawer:\n\n| Score | Label | What It Means |\n|-------|--------|---------------|\n| **1** | Poor | The interview produced very little usable data. The participant may have given one-word answers, gone off-topic, or disengaged early. |\n| **2** | Below Average | Some relevant information was shared, but responses lacked depth. There may be a few useful data points, but not enough for confident analysis. |\n| **3** | Average | A solid interview with meaningful responses. The participant engaged with most questions and provided enough detail to identify themes and insights. This is the threshold for credits and report inclusion. |\n| **4** | Good | A strong interview with detailed, thoughtful responses. The participant shared personal experiences, examples, and reasoning behind their opinions. |\n| **5** | Excellent | An exceptional interview with rich, nuanced data. The participant went deep on multiple topics, provided vivid examples, and offered perspectives you didn't anticipate. |\n\nThe Quality Evaluation card and the results tables group these five scores into four levels: **Shallow** (1-2), **Fair** (3), **Good** (4) and **Excellent** (5). Fair and above is billed and goes into your report. Shallow is free and stays out of it. So an interview the drawer calls Average appears as Fair in the tables.\n\nThe cut-off is the same everywhere: a score has to be 3 or higher to count. A score in between, such as 2.5, is Shallow on the Quality Evaluation card, in the tables, in billing and in the report alike.\n\n## The Five Quality Dimensions\n\nKoji evaluates each interview across five specific dimensions that combine into the overall score:\n\n### Relevance\nHow well the participant's responses addressed the research questions defined in your study brief. On-topic, focused answers score higher than tangential or off-topic responses.\n\n### Depth\nThe level of detail and elaboration in responses. Multi-sentence answers with explanations, examples, and reasoning score higher than brief or surface-level replies. Depth also limits the overall score: an interview with very shallow free-text answers can't score highly, however many structured questions it completed. Depth decides how good a usable interview is (3, 4 or 5), not whether it counts: see the answered-questions rule above.\n\n### Coverage\nHow thoroughly the interview covered the key topics and questions in your study. Interviews where the participant engaged with most or all research areas score higher.\n\n### Completion\nWhether the participant completed the full interview or dropped off early. Finishing the entire conversation contributes positively to the score.\n\n### Structured Quality\nFor studies using [structured questions](/docs/structured-questions-guide), such as scales, multiple choice, rankings and yes/no questions, this dimension evaluates how thoughtfully the participant engaged with those interactive elements. Did they provide considered ratings and selections, or did they rush through? Did they explain their reasoning in follow-up probes?\n\n## Why Quality Scores Matter\n\nQuality scores serve three important purposes:\n\n### 1. Research Data Quality\n\nNot all interviews are equally valuable. A five-minute conversation where a participant gave one-word answers tells you far less than a twenty-minute deep dive. The quality score helps you quickly identify which interviews deserve your attention first and which ones might need to be supplemented with additional data collection.\n\nWhen you're reviewing results, start with the highest-scoring interviews to get the strongest signals, then work your way down.\n\n### 2. Fair Billing Through the Quality Gate\n\nHere's something that works strongly in your favor: **interviews scoring below 3 don't use any credits**. This is Koji's quality gate, and it exists to protect you.\n\nIf a participant rushes through your interview giving minimal answers, or if someone provides off-topic responses, you shouldn't have to pay for that. The quality gate ensures you only spend credits on interviews that actually deliver research value.\n\nLearn more about how this works in [How the Quality Gate Works](/docs/how-the-quality-gate-works).\n\n### 3. Report Filtering\n\nWhen you [generate a research report](/docs/generating-research-reports), Koji includes completed and partial interviews with a quality score of 3 or above. A participant who stopped partway but gave usable answers counts, and is billed, like anyone else. Dropped interviews and interviews you archive stay out. This ensures your aggregate analysis, theme detection, and recommendations are built from substantive data rather than being diluted by low-quality responses.\n\n## Improving Your Quality Scores\n\nWhile you can't control how individual participants respond, you can set the stage for better interviews:\n\n- **Write a clear study brief**: The better your research objectives and target questions are defined, the more relevant the AI interviewer's questions will be. Clear briefs lead to focused conversations.\n\n- **Target the right participants**: People with direct experience in your research topic naturally provide richer, more detailed responses. A product user will give you better feedback about your product than someone who's never used it. Reach your own users with personalized links when you want to track who answered, or a generic anonymous link when candor matters, or open Panel recruit, describe the audience, and Koji finds people who match across 130+ countries, with your screening questions running before the interview. Panel respondents go through the same quality gate as everyone else.\n\n- **Use structured questions strategically**: Adding [structured questions](/docs/structured-questions-guide) like scales and choice selections gives participants concrete ways to express their views. Pair them with open questions, because the score now depends on what people say in their own words too.\n\n- **Set expectations upfront**: Your study description (what participants see before starting) should explain what the interview is about and roughly how long it takes. Prepared participants tend to give more thoughtful answers.\n\n- **Design for engagement**: Studies with interesting, relevant topics naturally produce better interviews. If your research questions connect to things participants genuinely care about, they'll be more willing to share detailed responses.\n\n## Viewing Quality Scores\n\nYou can see quality scores in several places:\n\n- **Quality Evaluation card**: On your study's results, the **Responses** tab opens with a row of summary cards. The Quality Evaluation card covers every scored completed and partial response, Shallow ones included. Its caption tells you how many are in the report, such as \"7 of 10 in the report\", plus how many are still being analyzed. Underneath sits a thin stacked bar and a legend that split those responses into four levels, ordered from low to high (see below). Hover over any part of the bar to see how many responses fall into that level.\n- **Completed responses card**: Next to it, this card shows the number of completed responses. Any other responses are listed under it as extras, for example \"+8 partial, +2 dropped, +1 screened out\". Each of those is its own view in the Responses table. Interviews you archive leave every count.\n- **Interview length card**: The third card shows the median length of a completed interview, so one session left open for hours doesn't skew it.\n- **Evaluation column**: The results tables show each response's level (Shallow, Fair, Good or Excellent) in an Evaluation column. On the Responses tab you can sort by it, and Shallow responses are listed too, so you can read what was said before deciding what to change. When a study has any Shallow responses, an evaluation filter appears above the table with three options: **All evaluations**, **Fair and above** and **Shallow only**. Answer cells show what participants said in their own words, not a summary of it, and the CSV export includes each response's 1-5 score.\n- **Analysis Drawer**: When you click an interview, the drawer shows its quality label (Excellent, Good, Average, Below Average or Poor) alongside other insights.\n- **Insights view**: On the full transcript page, the Insights sidebar view displays the quality score with its dimensional breakdown (relevance, depth, coverage, completion, structured quality).\n\n### Reading the Quality Evaluation breakdown\n\nThe bar and the legend run from low to high, left to right:\n\n| Level | Scores | What it tells you |\n|-------|--------|-------------------|\n| **Shallow** | 1-2 | Too thin to analyze. Left out of the report, and free: Shallow responses don't use credits. |\n| **Fair** | 3 | Enough substance to count. Included in your report and billed. |\n| **Good** | 4 | Detailed, dependable interviews that carry most of your themes. |\n| **Excellent** | 5 | Rich interviews you can quote from and build findings on. |\n\nBecause the breakdown covers every scored response, it gives you an honest picture of the study as a whole. If the Shallow group is large, that is a signal to look at your recruitment source or the way your questions are worded, not just at the interviews that made it into the report. Filter the table to **Shallow only** to read those responses side by side. A study where a third of responses are Shallow shows a much wider Shallow band than one where nearly everyone scores Good or Excellent, even if both reports draw on the same number of interviews.\n\n## Key Things to Know\n\n- **Scores are final**: Quality scores are calculated once after the interview completes and don't change. This keeps your billing and analysis consistent.\n- **Scores are not participant ratings**: A low score doesn't mean the participant was \"bad.\" It means the conversation didn't produce enough usable research data for various possible reasons.\n- **You can't manually override scores**: The scoring is automated to ensure objectivity and consistency across all interviews.\n- **Analysis is automatic**: You do not need to click a button to trigger scoring. It happens immediately when an interview completes.\n\n## Related Articles\n\n- [How the Quality Gate Works](/docs/how-the-quality-gate-works): understanding why low-quality interviews don't use credits\n- [AI-Generated Insights](/docs/ai-generated-insights): what analysis Koji produces from your interviews\n- [Viewing Interview Transcripts](/docs/viewing-interview-transcripts): how to read and navigate interview conversations\n- [Structured Questions Guide](/docs/structured-questions-guide): how structured question types affect the quality score\n- [Generating Research Reports](/docs/generating-research-reports): how quality filtering shapes your reports\n- [Why Your First Three Interviews Always Go Worst](/docs/interview-guide-failure-rate-curve): reading a run of low scores as an instrument defect\n\n## Frequently Asked Questions\n\n**Q: Can a participant retake an interview to improve the quality score?**\nA: Each interview submission is scored independently. If you share the link again with the same participant, they could complete a new interview, which would receive its own quality score.\n\n**Q: Do low-quality interviews still generate insights?**\nA: Yes, Koji generates AI insights for every completed interview regardless of its quality score. However, the insights from lower-quality interviews will naturally be less detailed and less reliable, and they don't feed into the study report.\n\n**Q: Why doesn't the Quality Evaluation card show an average score?**\nA: An average hides where responses sit. The card shows the spread instead: how many scored, completed and partial responses are Shallow, Fair, Good and Excellent, and how many of them are in the report. The Shallow group shows how many responses were left out.\n\n**Q: Where can I see my Shallow responses?**\nA: On the Responses tab of your study's results. Shallow responses are listed in the table with the rest, and the evaluation filter's **Shallow only** option narrows the table to them. They aren't in your report and don't use credits.\n\n**Q: What if I think a quality score is wrong?**\nA: The scoring evaluates factors like response depth, relevance, coverage, completion, and structured quality. If an interview has a lower score than expected, review the transcript. You may find that while the participant said some useful things, overall depth or completeness was limited.\n\n**Q: Does the quality score affect report generation?**\nA: Yes. Research reports include completed and partial interviews scoring 3 or above (Fair, Good and Excellent). The caption on the Quality Evaluation card shows how many are in the report.\n\n## Further reading on the blog\n\n- [Koji vs Listen Labs: AI Interview Platforms Compared (2026)](/blog/koji-vs-listen-labs-2026) — Listen Labs raised $69M and powers enterprise research at brands across the Fortune 500. Koji is the accessible AI-native alternative starti\n- [Koji vs Looppanel: End-to-End AI Research vs Analysis-Only (2026)](/blog/koji-vs-looppanel-2026) — Looppanel is a great AI analysis tool — but it only handles half the workflow. You still need a separate platform to actually run interviews\n\n<!-- further-reading:blog -->\n","category":"Reports & Analysis","lastModified":"2026-11-14T05:07:27.99465+00:00","metaTitle":"Interview Quality Scores (1-5) | Koji Docs","metaDescription":"How Koji rates interview quality from 1 to 5, what Shallow, Fair, Good and Excellent mean on the Quality Evaluation card, and why only scores of 3+ use credits.","keywords":["interview quality score","qualitative research quality","quality gate","interview evaluation","research data quality","Koji scoring"],"aiSummary":"Koji automatically scores every interview from 1 to 5 across five dimensions: relevance, depth, coverage, completion, and structured quality. Only interviews scoring 3 or above count toward billing and are included in research reports. Analysis is auto-triggered on completion.","aiPrerequisites":["creating-your-first-study"],"aiLearningOutcomes":["Understand the 0-5 quality scoring scale","Identify what factors contribute to higher quality scores","Design studies that encourage higher-quality interviews","Know how quality scores affect billing through the quality gate"],"aiDifficulty":"beginner","aiEstimatedTime":"7 min read"}],"pagination":{"total":1,"returned":1,"offset":0}}