Understanding Quality Scores
Learn how Koji scores interview quality from 1 to 5, how to read the Quality Evaluation card, and why only Fair and above (3+) use credits or enter your report.
Every completed or partial interview in Koji receives a quality score from 1 to 5. The score tells you how useful the interview is for your research, and it decides two things: only interviews scoring 3 or above use credits, and only those interviews go into your research report. You can see how a whole study's scores spread across four levels, and how many interviews made the report, on the Quality Evaluation card in your study's results, and each response's level (Shallow, Fair, Good or Excellent) in the Evaluation column of the results tables.
How Quality Scoring Works
After each interview ends, whether the participant finished or stopped partway, Koji's AI automatically evaluates the conversation across multiple dimensions to produce a single quality score. The evaluation happens immediately upon interview completion and takes just a few seconds. There is no manual trigger required.
The score reflects the overall research value of the interview: not the participant's intelligence or effort, but how much actionable data the conversation produced. A high-quality interview is one that gives you clear, detailed, and relevant information you can use to make decisions.
Scores follow what the participant actually said
Since September 15, 2026, the score can't be higher than the substance of the participant's own answers allows. Before that change, someone could click through a set of multiple-choice questions, answer every follow-up with a few dots, and still receive a top score. Now thin or nonsense answers to open questions and follow-ups bring the score down, so those interviews stay out of your report and you aren't charged for them.
This applies to new interviews only. Scores you already have stay the same. Studies made up entirely of structured questions are scored on their structured answers, since there is no free text to judge.
Short, relevant answers still count
Since October 19, 2026, Koji also checks every interview question on its own and marks it answered, off topic or not answered. Follow-ups are judged together with the question they belong to.
- Answered means the reply relates to the question and gives a real experience, preference or opinion. Length doesn't matter: "Netflix", "twice a week", a single rating or a chosen option all count.
- Off topic means they replied but said nothing that answers it, such as "ok", "idk" or filler text.
- Not answered means the question was never asked or never replied to.
If most of the interview questions got a real, relevant answer, and the interview as a whole stayed on topic, the score is at least 3. That interview is billed and goes into your report. Depth then decides whether it scores 3, 4 or 5. This floor stops short-answer studies from losing real respondents just because people answered in a few words, while rushed or nonsense interviews still score low.
If the analysis of an interview fails and never produces a score, that interview stays out of your report and isn't charged.
The 1-5 Scale Explained
Quality scores range from 1 to 5. Here's what each score means in practice, with the label shown in an interview's analysis drawer:
| Score | Label | What It Means |
|---|---|---|
| 1 | Poor | The interview produced very little usable data. The participant may have given one-word answers, gone off-topic, or disengaged early. |
| 2 | Below Average | Some relevant information was shared, but responses lacked depth. There may be a few useful data points, but not enough for confident analysis. |
| 3 | Average | A solid interview with meaningful responses. The participant engaged with most questions and provided enough detail to identify themes and insights. This is the threshold for credits and report inclusion. |
| 4 | Good | A strong interview with detailed, thoughtful responses. The participant shared personal experiences, examples, and reasoning behind their opinions. |
| 5 | Excellent | An exceptional interview with rich, nuanced data. The participant went deep on multiple topics, provided vivid examples, and offered perspectives you didn't anticipate. |
The Quality Evaluation card and the results tables group these five scores into four levels: Shallow (1-2), Fair (3), Good (4) and Excellent (5). Fair and above is billed and goes into your report. Shallow is free and stays out of it. So an interview the drawer calls Average appears as Fair in the tables.
The cut-off is the same everywhere: a score has to be 3 or higher to count. A score in between, such as 2.5, is Shallow on the Quality Evaluation card, in the tables, in billing and in the report alike.
The Five Quality Dimensions
Koji evaluates each interview across five specific dimensions that combine into the overall score:
Relevance
How well the participant's responses addressed the research questions defined in your study brief. On-topic, focused answers score higher than tangential or off-topic responses.
Depth
The level of detail and elaboration in responses. Multi-sentence answers with explanations, examples, and reasoning score higher than brief or surface-level replies. Depth also limits the overall score: an interview with very shallow free-text answers can't score highly, however many structured questions it completed. Depth decides how good a usable interview is (3, 4 or 5), not whether it counts: see the answered-questions rule above.
Coverage
How thoroughly the interview covered the key topics and questions in your study. Interviews where the participant engaged with most or all research areas score higher.
Completion
Whether the participant completed the full interview or dropped off early. Finishing the entire conversation contributes positively to the score.
Structured Quality
For studies using structured questions, such as scales, multiple choice, rankings and yes/no questions, this dimension evaluates how thoughtfully the participant engaged with those interactive elements. Did they provide considered ratings and selections, or did they rush through? Did they explain their reasoning in follow-up probes?
Why Quality Scores Matter
Quality scores serve three important purposes:
1. Research Data Quality
Not all interviews are equally valuable. A five-minute conversation where a participant gave one-word answers tells you far less than a twenty-minute deep dive. The quality score helps you quickly identify which interviews deserve your attention first and which ones might need to be supplemented with additional data collection.
When you're reviewing results, start with the highest-scoring interviews to get the strongest signals, then work your way down.
2. Fair Billing Through the Quality Gate
Here's something that works strongly in your favor: interviews scoring below 3 don't use any credits. This is Koji's quality gate, and it exists to protect you.
If a participant rushes through your interview giving minimal answers, or if someone provides off-topic responses, you shouldn't have to pay for that. The quality gate ensures you only spend credits on interviews that actually deliver research value.
Learn more about how this works in How the Quality Gate Works.
3. Report Filtering
When you generate a research report, Koji includes completed and partial interviews with a quality score of 3 or above. A participant who stopped partway but gave usable answers counts, and is billed, like anyone else. Dropped interviews and interviews you archive stay out. This ensures your aggregate analysis, theme detection, and recommendations are built from substantive data rather than being diluted by low-quality responses.
Improving Your Quality Scores
While you can't control how individual participants respond, you can set the stage for better interviews:
-
Write a clear study brief: The better your research objectives and target questions are defined, the more relevant the AI interviewer's questions will be. Clear briefs lead to focused conversations.
-
Target the right participants: People with direct experience in your research topic naturally provide richer, more detailed responses. A product user will give you better feedback about your product than someone who's never used it. Reach your own users with personalized links when you want to track who answered, or a generic anonymous link when candor matters, or open Panel recruit, describe the audience, and Koji finds people who match across 130+ countries, with your screening questions running before the interview. Panel respondents go through the same quality gate as everyone else.
-
Use structured questions strategically: Adding structured questions like scales and choice selections gives participants concrete ways to express their views. Pair them with open questions, because the score now depends on what people say in their own words too.
-
Set expectations upfront: Your study description (what participants see before starting) should explain what the interview is about and roughly how long it takes. Prepared participants tend to give more thoughtful answers.
-
Design for engagement: Studies with interesting, relevant topics naturally produce better interviews. If your research questions connect to things participants genuinely care about, they'll be more willing to share detailed responses.
Viewing Quality Scores
You can see quality scores in several places:
- Quality Evaluation card: On your study's results, the Responses tab opens with a row of summary cards. The Quality Evaluation card covers every scored completed and partial response, Shallow ones included. Its caption tells you how many are in the report, such as "7 of 10 in the report", plus how many are still being analyzed. Underneath sits a thin stacked bar and a legend that split those responses into four levels, ordered from low to high (see below). Hover over any part of the bar to see how many responses fall into that level.
- Completed responses card: Next to it, this card shows the number of completed responses. Any other responses are listed under it as extras, for example "+8 partial, +2 dropped, +1 screened out". Each of those is its own view in the Responses table. Interviews you archive leave every count.
- Interview length card: The third card shows the median length of a completed interview, so one session left open for hours doesn't skew it.
- Evaluation column: The results tables show each response's level (Shallow, Fair, Good or Excellent) in an Evaluation column. On the Responses tab you can sort by it, and Shallow responses are listed too, so you can read what was said before deciding what to change. When a study has any Shallow responses, an evaluation filter appears above the table with three options: All evaluations, Fair and above and Shallow only. Answer cells show what participants said in their own words, not a summary of it, and the CSV export includes each response's 1-5 score.
- Analysis Drawer: When you click an interview, the drawer shows its quality label (Excellent, Good, Average, Below Average or Poor) alongside other insights.
- Insights view: On the full transcript page, the Insights sidebar view displays the quality score with its dimensional breakdown (relevance, depth, coverage, completion, structured quality).
Reading the Quality Evaluation breakdown
The bar and the legend run from low to high, left to right:
| Level | Scores | What it tells you |
|---|---|---|
| Shallow | 1-2 | Too thin to analyze. Left out of the report, and free: Shallow responses don't use credits. |
| Fair | 3 | Enough substance to count. Included in your report and billed. |
| Good | 4 | Detailed, dependable interviews that carry most of your themes. |
| Excellent | 5 | Rich interviews you can quote from and build findings on. |
Because the breakdown covers every scored response, it gives you an honest picture of the study as a whole. If the Shallow group is large, that is a signal to look at your recruitment source or the way your questions are worded, not just at the interviews that made it into the report. Filter the table to Shallow only to read those responses side by side. A study where a third of responses are Shallow shows a much wider Shallow band than one where nearly everyone scores Good or Excellent, even if both reports draw on the same number of interviews.
Key Things to Know
- Scores are final: Quality scores are calculated once after the interview completes and don't change. This keeps your billing and analysis consistent.
- Scores are not participant ratings: A low score doesn't mean the participant was "bad." It means the conversation didn't produce enough usable research data for various possible reasons.
- You can't manually override scores: The scoring is automated to ensure objectivity and consistency across all interviews.
- Analysis is automatic: You do not need to click a button to trigger scoring. It happens immediately when an interview completes.
Related Articles
- How the Quality Gate Works: understanding why low-quality interviews don't use credits
- AI-Generated Insights: what analysis Koji produces from your interviews
- Viewing Interview Transcripts: how to read and navigate interview conversations
- Structured Questions Guide: how structured question types affect the quality score
- Generating Research Reports: how quality filtering shapes your reports
- Why Your First Three Interviews Always Go Worst: reading a run of low scores as an instrument defect
Frequently Asked Questions
Q: Can a participant retake an interview to improve the quality score? A: Each interview submission is scored independently. If you share the link again with the same participant, they could complete a new interview, which would receive its own quality score.
Q: Do low-quality interviews still generate insights? A: Yes, Koji generates AI insights for every completed interview regardless of its quality score. However, the insights from lower-quality interviews will naturally be less detailed and less reliable, and they don't feed into the study report.
Q: Why doesn't the Quality Evaluation card show an average score? A: An average hides where responses sit. The card shows the spread instead: how many scored, completed and partial responses are Shallow, Fair, Good and Excellent, and how many of them are in the report. The Shallow group shows how many responses were left out.
Q: Where can I see my Shallow responses? A: On the Responses tab of your study's results. Shallow responses are listed in the table with the rest, and the evaluation filter's Shallow only option narrows the table to them. They aren't in your report and don't use credits.
Q: What if I think a quality score is wrong? A: The scoring evaluates factors like response depth, relevance, coverage, completion, and structured quality. If an interview has a lower score than expected, review the transcript. You may find that while the participant said some useful things, overall depth or completeness was limited.
Q: Does the quality score affect report generation? A: Yes. Research reports include completed and partial interviews scoring 3 or above (Fair, Good and Excellent). The caption on the Quality Evaluation card shows how many are in the report.
Further reading on the blog
- Koji vs Listen Labs: AI Interview Platforms Compared (2026) — Listen Labs raised $69M and powers enterprise research at brands across the Fortune 500. Koji is the accessible AI-native alternative starti
- Koji vs Looppanel: End-to-End AI Research vs Analysis-Only (2026) — Looppanel is a great AI analysis tool — but it only handles half the workflow. You still need a separate platform to actually run interviews
Related Articles
Can You Trust AI Interviewers? How Koji Prevents Hallucinations and Bias in Customer Research
A practical guide to how modern AI research platforms prevent hallucinations, model bias, and leading questions during auto-moderated customer interviews — with the verification techniques Koji uses to keep AI-generated insights faithful to the actual transcript.
AI Interviewer Tuning: How to Get Research-Grade Voice Interviews
A complete playbook for tuning Koji's AI interviewer — company context, probing depth, structured questions, and interview mode — to deliver interviews indistinguishable from a human researcher.
AI Transcription for Research Interviews: Speed Up Analysis by 10x
Learn how AI transcription transforms research interviews — from audio to analysis in minutes. Covers accuracy, speaker identification, theme extraction, quality scoring, and how Koji automates the entire pipeline.
Answer Confidence in Koji Reports: What High, Medium and Low Actually Mean
Every structured answer in a Koji report carries a high, medium or low confidence flag describing how certain the analysis is that it mapped the right transcript span to the right question. Here is what each level means and what to do about it.
The Base Rate Nobody Measured: Why Every Flag in Your Research Stack Has an Unknown Precision (2026)
Every AI tag, sentiment label and risk score is a diagnostic test whose precision depends on a prevalence nobody measured. Accuracy rises as precision collapses. How to audit the unflagged pile and publish a precision footer.
Company Context: How to Make Your AI Interviewer a Domain Expert
Learn how to configure Koji's company context setting so your AI interviewer asks sharper, more relevant follow-up questions across every study you run.
Generating Research Reports
Create comprehensive aggregate reports across all your interviews — including summaries, themes, recommendations, and statistics.
How the Quality Gate Works
Understand Koji's quality gate — conversations scoring below 3/5 are completely free and don't consume credits, protecting your research budget.
Interview Completion Flow
What happens when an interview ends — thank-you screen, participant feedback, and behind-the-scenes AI analysis.
Why Your First Three Interviews Always Go Worst: The Failure-Rate Curve of an Interview Guide
Interview guides fail on a curve, not at a constant rate. Learn the three regions - early-life, useful life, and wear-out - and the opposite fixes each one needs.
Structured, Exploratory, and Hybrid: Choosing the Right Interview Mode in Koji
A complete guide to Koji's three interview modes — structured, exploratory, and hybrid — and when to use each for your research goals.
Microsoft Teams + Koji: Post AI Interview Insights Into the Channel Where Decisions Get Made
How to pipe completed Koji AI interviews into Microsoft Teams using the Workflows webhook that replaced the retired Office 365 connector, with the exact payload fields available.
Productboard + Koji: Turn AI Interview Insights Into Roadmap Evidence Automatically
How to push grounded customer quotes from Koji AI interviews into Productboard as insights, so every roadmap item traces back to something a real customer said.
Survey Fraud & Respondent Quality: How to Detect Fake and Low-Effort Responses (2026)
Between 5% and 26% of survey responses are fraudulent, and AI-generated answers now pass standard quality checks. Learn the warning signs, the detection tactics that still work, and how Koji's conversational quality gate filters bad data before it reaches your report.
Viewing Interview Transcripts
How to read, navigate, and get value from your interview transcripts in Koji.
Accents, Dialects and AI Transcription: How to Check Your Voice Research Works for Everyone
Speech recognition error is not evenly distributed across your participants. Here is how to measure word error rate by speaker group, why concept error rate matters more than WER, and how to design voice studies that do not quietly exclude people.
Publishing & Sharing Reports
Make your research reports accessible to stakeholders, team members, and decision-makers.