Health Score Calibration: A Customer Success Ops Playbook for Checking Your Score Against What Customers Say

A health score calibration is a one-off check of your customer health score against what customers say, run before you rebuild the score for next year's plan. With AI-moderated interviews you can talk to a stratified sample of 18 to 30 accounts in about two weeks and read the answers as one report. By the end of this guide you will know which accounts to pick, what to ask, how to run it in Koji and which score inputs to cut, keep or add.

Why health score calibration matters now

Retention is getting harder to hold, and the score most CS teams use to see trouble coming is rarely tested against the customer's own account of things.

Photo by ThisisEngineering on Unsplash

• Gross retention is slipping. The 2026 SaaS & AI Performance Benchmarks from Aleph and Benchmarkit cover 342 SaaS and AI-native companies. Median gross revenue retention for full-year 2025 was 84%, down from 88% a year earlier. The top quartile sits at 91% and the bottom quartile at 76% (Aleph).
• Expansion barely covers the leak. SaaS Capital's 2025 benchmarks, from more than 1,000 private B2B SaaS companies, show median net revenue retention of 101% and gross retention of 91% (SaaS Capital). At 101%, an account that leaves without a warning shows up in the number you report.
• Retention is the number CS is judged on. Gainsight and Benchmarkit's survey of more than 250 companies, fielded July to September 2024, found 55% name gross revenue retention as a top measure, and 52% were already building AI into their workflows (Gainsight).
• The pressure is on proof. TSIA's State of Customer Success 2026 describes teams being asked to justify technology, AI and staffing spend in a tougher economy, with fragmented data limiting what AI can deliver (TSIA).

Vendors say the same thing about their own product. ChurnZero has written that a score that sometimes produces false positives or false negatives is not a reliable metric, and that the old assumption of green means expand and red means churn no longer holds (ChurnZero). I could not find an independent 2026 survey that measures how often scores are right, and I would be wary of any "87% accurate" claim that does not publish its method.

There is also a base-rate trap. If 10% of your customers churn in a year, a score that says "nobody will churn" is right 90% of the time and tells you nothing. Raw accuracy is the wrong test. What you need to know is whether the accounts you call green really are safe.

The cost of skipping the check is a forecast built on a number you cannot defend, and a CS Ops team that spends the next planning cycle tuning weights by feel.

What good looks like

Calibration is a comparison, so you need two things on one page for each account: what the score said 120 days before the renewal date, and what the customer says now.

Backtest first, interview second. Pull the last 12 months of renewals. For every churned logo, find the score it had 120 days out. Count how many were green. That gives you a miss rate in an afternoon, with no customer contact. It also tells you how many interviews you need: if 1 in 4 of your churned logos were green at day 120, you have a blind-spot problem and the interviews should hunt for it.

Sample by score band, not by account size. Pick 6 to 10 accounts each from green, yellow and red, all renewing in the next 150 days, for 18 to 30 in total. If your book has fewer than 60 accounts, interview every one. Over-sampling red teaches you why red is red, which the score already says.

Interview two roles per account. The champion and the person who signs. Their answers disagree often enough that the gap is itself a signal.

Ask for events from the last 90 days. Who was in the room, what was decided, what changed in how they buy tools. A stated intention to renew is weak evidence. A story about a budget review is strong evidence.

Code every account before you read the score. A CSM or CS Ops lead reads the transcript summary and marks the account Safe, Watch or At risk, blind to the score. Then reveal the score. This is the one step teams skip, and it is the one that stops you rationalising.

Decision rules you can copy.

• If more than 1 in 4 of your green accounts come back At risk, the score is not safe to use for forecasting. Rebuild it.
• If more than 1 in 3 of your red accounts come back Safe, the score is crying wolf. Cut or reweight the noisy inputs, usually a single usage metric.
• If the reasons customers give cluster in one place (a budget review, a new leader, a competitor), add that as an input. If they scatter, the score is not the problem.
• If two roles at the same account disagree, flag the account for a human, whatever the score says.

Common mistakes

• Interviewing only red accounts, which tells you nothing about false greens.
• Letting the CSM who owns the account code it.
• Changing the weights after three interviews. Wait for 18.
• Measuring raw accuracy instead of the miss rate on green accounts.
• Treating the interview as a renewal conversation. Keep sales out of it.

How to set it up in Koji, step by step

You need a Koji account. The Free plan comes with signup credits for a first run; for 18 to 30 accounts you will want a paid plan with enough credits, and the Enterprise plan if you need a Data Processing Agreement or Bring Your Own Key (see the privacy and security guide). Previews are free and never count as responses.

1. Start from a template. Open the template gallery in the dashboard and pick QBR Prep Research under Customer Success, or start a blank study. Tell the assistant: "I want to check our health score against what customers say. Interview the champion and the budget owner at accounts renewing in the next 150 days about the last 90 days."
2. Shape the research brief. Set the goal, the two roles you want, and a guardrail so the interviewer does not discuss price or renewal terms. Say in the opening that this is a listening exercise and nobody will follow up with a quote.
3. Add structured questions. Keep the story open-ended and add a scale and a choice so the report can chart them across accounts. The full set is in the next section.
4. Choose voice or text, then preview. Participants choose how they answer. Run the preview yourself first, since it is free.
5. Invite by CSV. Export account name, role, score band and renewal date from your CRM and import it. Each person gets their own link, and the columns stay attached to every response, which is what lets you cut the report by score band.

6. Read the report. Koji analyses each interview and builds a report with themes, findings and the quotes behind each. Interviews that score below 3 on quality are not charged, so a half-finished call does not cost you.

Copy this interview guide

Paste the one-line brief into the "What do you want to research?" box, then add the questions below. Aim for a 10 to 12 minute conversation.

Research brief: Interview the champion and budget owner at accounts renewing in the next 150 days about the last 90 days of using our product, so we can test our health score against their own account of how things stand.

Opening: "Thanks for making time. I'm an AI interviewer working for [company]. A real person on the team reads every answer. This is a listening exercise: nothing will be sold to you afterwards."

1. "Tell me about the last time you used [product] for something that mattered. What happened?" Open-ended. Probe: "What did you do next?"
2. "What has changed on your team or in your priorities since you signed?" Open-ended. Probe: "Who made that call?"
3. "How likely is your team to still rely on [product] in 12 months?" Scale, 1 to 10. Probe: "What would move that number up or down?"
4. "Who else would have a say if you reviewed this tool?" Single choice: nobody / my manager / finance / IT or security / a new leader. Probe: "Have they asked about it?"
5. "Has anyone asked you to review or justify your software spend this year?" Yes/no. Probe: "What did they ask for?"
6. "When did [product] last save you real time or money? Be specific." Open-ended. Probe: "How would you explain that to your CFO?"
7. "Is there anything you were hoping we would do that we haven't?" Open-ended. Probe: "How long have you been waiting?"
8. "What would make you look at alternatives?" Open-ended. Probe: "Has that happened yet?"

Closing: "Is there anything you expected me to ask that I didn't?"

Aim for the recoding to take about five minutes per account: read the summary, mark Safe, Watch or At risk, and write one line of evidence.

What you get

A single report instead of 24 sets of call notes. You see which themes repeat, what share of interviewees raised each, and the quotes behind them. Because the score band was in your CSV, you can compare what green accounts say against what red accounts say.

Use it where decisions are made:

• Score review meeting: bring the 2×2 with counts in each box and the decision rule you applied.
• Planning deck: propose the input changes, each with the quote that justifies it.
• Forecast call: replace "feels green" with "5 of 8 green accounts raised a budget review".
• Account plans: every At risk account gets a named owner and a next step within two working days.

The same playbook elsewhere

The pattern, "test the model against the people it describes", carries well beyond CS.

• GTM teams check lead and deal scores against why buyers actually bought. See the win-loss interview questions guide and the no-decision loss interviews playbook.
• HR checks flight-risk models against stay and exit interviews, in the spirit of the customer exit interviews guide.
• PE operating partners check portfolio dashboards against what customers say, as in the 100-day customer baseline.
• Customer success already has the front-end version in the renewal risk interviews playbook. Calibration is the same conversation, aimed at the score instead of the account.

Why Koji for health score calibration

	Backtest only	NPS / CSAT survey	CSM-run calls	Koji AI interviews
Tells you why the score missed	No	Rarely	Yes	Yes, with follow-ups
Covers a stratified sample	Yes	Yes	Hard to staff	Yes
Independent of the account owner	Yes	Yes	No	Yes
Same questions every time	Yes	Yes	No	Yes
Time to read across accounts	Fast	Fast	Slow, manual notes	Report builds as interviews finish
Effort per account	None	Low	High	Low

In fairness: a backtest is where you should start, and it costs nothing. A survey gives a quick trend. A CSM call beats everything on trust. The weakness of calls is that the person who owns the account also marks its homework. A neutral interviewer asking the same eight questions of every account avoids that.

Why Koji for this job:

• Follow-up depth. The interviewer probes a vague answer, so "budget is tight" becomes whose budget and which decision.
• Structured and open questions in one flow. A 1 to 10 likelihood and the story behind it, charted across accounts.
• Voice, text and phone. Participants pick, and outbound calls can reach people who will not click a link.
• Traceable findings. Each finding links to the interviews and quotes behind it, so you can defend an input change in a review.
• A quality bar. Interviews are scored 1 to 5, and those under 3 are not charged.

Questions you might have about AI interviews

Will customers talk to an AI instead of their CSM? Many will if you are honest about it. Say in the invitation that it is an AI interviewer, that a person on your team reads the results, and why you are asking. Pilot with 10 accounts and read the completion rate before you scale.

Will it hurt the relationship? It can if it feels like surveillance or a sales tactic. Use a personal invite from the CSM, say that nothing is sold, and close the loop. Some customers find it easier to criticise a product to a neutral interviewer. Treat that as something to test, not a promise.

Where does the account data go? Data is encrypted in transit and at rest, and a Data Processing Agreement is available to business customers. Read the privacy and security guide and have your security team check it against your policy.

When Koji may not be the right fit

If you have fewer than 15 strategic accounts and your CSMs speak with the budget owner every month, run the eight questions yourself. If customers do not speak a language Koji supports, or a contract forbids third-party recording, the same guide works on a call. And if you have no score history to backtest, build that first: interviews tell you why a score is wrong, but they cannot tell you how often.

Metrics to track

• Miss rate: share of churned logos that were green at day 120
• Green accounts coded At risk, and red accounts coded Safe
• Completion rate by role
• Share of accounts with two roles interviewed
• Input changes adopted, with the quote behind each
• Miss rate again two quarters later

FAQ

How many accounts do I need? Aim for 18 to 30, split across green, yellow and red. Below 60 accounts in your book, interview all of them.

When should I run it? Six weeks before your planning cycle, so you have time to rebuild. The interviews take about two weeks, and recoding and review another two.

Can I do it without a CS platform? Yes. All you need is an export with account, role, score and renewal date.

Should the CSM code the account? No. Have a second person do it, blind to the score.

What if an interview reveals a serious problem? Treat it as a ticket for a human and reach out within two working days with the quote.

How is this different from renewal risk interviews? Renewal interviews are about each account. Calibration uses a sample of accounts to test the score itself.

Ready to start?

Open the template gallery, pick QBR Prep Research and paste the brief above, or read about how CS teams use Koji. All playbooks for this audience live in the Customer Success use-case library. More to read: Customer Research for Customer Success Teams, the customer health score guide, Importing participants via CSV, How the quality gate works and Generating research reports.