How the Quality Gate Works

The Quality Gate: Never Pay for Bad Data

Koji’s quality gate is a built-in protection that ensures you only spend credits on meaningful conversations. Any interview that scores below 3 on Koji’s 1-5 quality scale is completely free — no credits consumed, no charges incurred.

This applies to every plan and every way of buying credits: plan credits, credit packs and auto-reload top-ups. It covers text and voice interviews, whether the participant finished or left partway through. Phone interviews work differently: they are charged per connected minute.

---

How Quality Scoring Works

Every text and voice interview on Koji is automatically scored on a 1–5 scale based on the depth and quality of the interaction. The app shows each score as one of four levels:

The cut-off is the same everywhere: a score has to be 3 or higher to count. A score in between, such as 2.5, is Shallow for billing and for the report alike. If the analysis of an interview fails and never produces a score, it is treated the same way: free, and left out of the report.

The scoring evaluates factors like:

• Response depth — Did the participant give substantive, detailed answers?
• Conversation completeness — Did the interview cover the key topics?
• Engagement level — Was the participant actively engaged throughout?
• Information value — Did the conversation generate actionable insights?

What the participant actually said decides the score

Since September 15, 2026, thin or nonsense answers bring the score down. Someone who clicks through the multiple-choice questions and answers every follow-up with a few dots no longer gets a high score, so that interview stays out of your report and you aren’t charged for it. This applies to interviews from that date on; earlier scores stay as they were.

Short answers are not the same as thin answers. A reply like “twice a week” or a single rating counts when it really answers the question. When most of the questions got a real, relevant answer and the interview stayed on topic, it scores at least 3. Understanding Quality Scores explains how each question is judged.

Interviews that end early

A participant who leaves partway through doesn’t automatically get a free pass. Since November 9, 2026, a partial interview is scored like any other: if it scores 3 or more, it is billed and goes into your report, because the answers it holds are usable. A partial interview that scores below 3 is free and stays out.

Dropped interviews and interviews you archive never count toward the report. You can read more about why people stop in Partial Interviews and Breakoff Analysis.

---

Why the Quality Gate Matters

Budget Protection

Without a quality gate, every conversation would cost credits — even ones where:

• Responses were too brief or off topic to be useful
• The participant answered with filler or nonsense
• The participant didn’t engage meaningfully with the questions

With Koji’s quality gate, these low-value interactions are filtered out automatically. You only pay for conversations that actually contribute to your research.

Real Impact on Your Budget

Consider an example study with 50 total conversations:

• 40 conversations score 3+ (Fair, Good or Excellent) — credits charged
• 10 conversations score below 3 (Shallow) — completely free

In that example, that’s a 20% saving on your credit budget, applied automatically.

---

Quality Gate Across All Plans

Quality Gate on Every Credit

The quality gate applies no matter where your credits came from: your plan, a credit pack, or an auto-reload top-up. If a conversation scores below 3 on the 1-5 scale, no credits are drawn from your balance. You are never billed for low-quality interactions, and Koji has no overage billing. When your balance runs out, new interviews pause until the workspace owner adds credits. See Running Out of Credits, Explained for packs, auto-reload and the monthly spending cap.

---

Quality Gate in Reports

The quality gate doesn’t just protect your budget — it also protects your research quality. Reports include completed and partial interviews that score 3 or above. Shallow conversations are excluded from report aggregation, theme analysis, and statistical summaries. This ensures your research findings are built on substantive data rather than being diluted by low-effort responses.

You can still read every Shallow response in the Responses tab, but it won’t affect your aggregated insights.

---

What Affects Conversation Quality

Several factors influence whether a conversation meets the quality threshold:

Participant-Side Factors

• Engagement — participants who take the interview seriously produce higher scores
• Substance — real, on-topic answers count, even short ones; filler and nonsense pull the score down
• Relevance — responses that address the actual questions score higher
• Completion — finishing the full interview improves the score

Study Design Factors (You Control These)

• Clear, open-ended questions — well-crafted questions encourage detailed responses
• Appropriate interview length — not too short (insufficient depth) or too long (fatigue)
• Good probe points — AI follow-up prompts that encourage elaboration
• Relevant guardrails — keeping conversations on topic improves quality
• Structured questions — mixing in scale, choice, and ranking questions helps maintain engagement and produces richer data

Tips to Maximize Quality Scores

1. Write open-ended questions — "Tell me about..." instead of "Do you like...?"
2. Set up probe points — guide the AI to dig deeper on key topics
3. Keep interviews focused — 5-8 core questions is the sweet spot
4. Use guardrails — prevent conversations from going off-topic
5. Target the right participants — better targeting = better engagement
6. Use voice interviews — voice conversations tend to produce richer, more detailed responses
7. Add structured questions — scale and choice questions keep participants engaged between open-ended exploration. Pair them with open questions, because the score depends on what people say in their own words too.

---

Viewing Quality Scores

You can see how your study did against the quality gate in its results:

1. Navigate to your study and open the Responses tab.
2. The Quality Evaluation card shows how scored responses split across Shallow, Fair, Good and Excellent, and its caption says how many are in the report (for example "7 of 10 in the report").
3. Each response shows its level in the Evaluation column, which you can sort by.
4. When a study has Shallow responses, an evaluation filter appears above the table: All evaluations, Fair and above or Shallow only.
5. Click a response to open its analysis, or export the table as CSV to get each response’s 1-5 score.

---

The Quality Gate and Research Integrity

The quality gate improves your research in several ways:

• Budget protection — you only spend credits on conversations that met the quality threshold
• Report accuracy — reports aggregate only interviews scoring 3 or above, so your insights are built from substantive data
• Targeting feedback — a large Shallow group may mean you need to adjust your participant criteria or recruitment source
• Study design insights — patterns in Shallow conversations reveal where to refine your questions

Related Articles

• Understanding Quality Scores: how each interview is scored and how to read the Quality Evaluation card
• Partial Interviews and Breakoff Analysis: why participants stop and what partial interviews tell you
• Running Out of Credits, Explained: packs, auto-reload and the spending cap
• Understanding Usage Limits: what each interview type costs
• Structured Questions Guide: question types that keep participants engaged