NPS Comment Analysis: Turn Open-Text Verbatims into Driver Themes

The Bottom Line

The NPS number tells you that customers feel a way; the comments tell you why. NPS comment analysis is the process of coding those open-text verbatims into a small set of driver themes, then linking each theme to the scores it appears in so you can rank what actually moves your number. Done by hand, this means reading hundreds of comments and tagging them in a spreadsheet. Done with a platform like Koji, the open-text follow-up is treated as a real open_ended question — the AI probes "what would move that to a 9?", then themes every answer automatically and ties each theme back to promoters, passives, and detractors.

Why the NPS Number Alone Is a Dead End

A single NPS score — say, 32 — is a thermometer, not a diagnosis. It moved from 28 last quarter; great, but why? Without the comments coded into drivers, you are guessing. The open-text "What is the main reason for your score?" question is where the real signal lives, and it is almost always the most under-analyzed field in the whole program because reading and tagging it is tedious.

The goal of comment analysis is to convert that free text into three usable things:

1. A ranked list of drivers — the themes that show up most often
2. Sentiment by driver — is "onboarding" praised by promoters or blamed by detractors?
3. Driver-to-score linkage — which themes cluster in 9–10 vs 0–6

Step 1 — Standardize the Follow-Up Question

Most weak NPS analysis starts with a weak question. After the 0–10 rating (a scale question), always ask an open follow-up. The best wording is segment-aware:

• Detractors (0–6): "What is the main reason for your score, and what would have to change?"
• Passives (7–8): "What would it take to make this a 9 or 10?"
• Promoters (9–10): "What do you value most — and who else would benefit?"

In Koji you do not have to send three separate forms. Set the NPS scale with an anchor follow-up and the AI asks the right version automatically based on the score the participant just gave. This is the single highest-leverage upgrade to any NPS program.

Step 2 — Code Verbatims into Themes (Open + Axial Coding)

Manual NPS coding follows the classic two-cycle method: first open coding (label each comment with a short descriptive code like "slow performance" or "responsive support"), then axial coding (cluster near-duplicate codes into a canonical driver). The hard part at scale is consistency — two analysts code the same comment differently, and the codebook drifts.

Koji performs this automatically. Each open_ended answer is coded into themes grounded in the participant's exact words, then near-duplicate themes are clustered into a canonical codebook across all responses during report aggregation. Every theme carries the count of how many people raised it and the verbatim quotes that justify it — so the analysis is both fast and auditable. (For the manual technique, see open, axial, and selective coding.)

Step 3 — Link Drivers to Score Bands

A theme list is useful; a theme list split by promoter / passive / detractor is decisive. The question is not "what do people mention?" but "what do my detractors mention that my promoters do not?" That delta is your roadmap. Because Koji keeps each NPS answer linked to its score via stable question IDs, you can see that, for example, "confusing setup" appears in 41% of detractor comments and 4% of promoter comments — a clear, prioritized fix.

Step 4 — Quantify Sentiment and Trend It

For each driver, track two numbers over time: frequency (how often it is mentioned) and sentiment (positive vs negative). A driver that is rising in frequency and negative in sentiment is an early warning the score has not caught up to yet. Koji's sentiment and theme tracking lets you trend drivers release over release, so you catch a deteriorating driver before it drags the headline number down.

Step 5 — Close the Loop

The payoff of comment analysis is action. Use the ranked driver list to brief product and CS, fix the top detractor driver, and re-survey to confirm the theme's frequency dropped. Because the AI already captured the specific friction (not just "support" but "support took three days to reply to billing tickets"), the fix is concrete instead of vague.

Why AI Interviews Beat Spreadsheet Tagging

Traditional NPS tools (the SurveyMonkeys and Qualtrics of the world) collect the comment and stop — the analysis is on you. The shift with an AI-native approach is twofold: the AI probes the comment in the moment (so the verbatim is richer), and it codes the comment automatically (so the analysis is instant and consistent). You get the why behind the number without a single hour of manual tagging.

A Worked Example: From 200 Comments to 5 Drivers

Imagine a quarter's NPS run with 200 scored responses, each with a comment. Manually, you would read all 200, invent codes as you go, re-read the first 50 because your codebook drifted, and produce a rough tally by Friday. With AI coding the flow is different:

1. The 200 comments are coded into ~30 raw descriptive codes ("slow sync", "great support rep", "confusing billing", and so on).
2. Near-duplicates collapse into ~5 canonical drivers: Performance, Support quality, Billing clarity, Onboarding, Pricing.
3. Each driver shows its frequency and a sentiment split, plus the verbatim quotes behind it.
4. Split by band, "Performance" appears in 38% of detractor comments and 6% of promoter comments — your number-one fix.

What took a weekend now takes minutes, and the codebook is applied uniformly across all 200 instead of drifting between the first and last comment you read.

Common NPS Comment Analysis Mistakes

• Analyzing only detractors. Promoter comments tell you what to protect and what to amplify in marketing — do not throw them away.
• Counting mentions without sentiment. "Pricing" mentioned 40 times could be 40 complaints or 40 compliments. Always pair frequency with sentiment.
• Letting the codebook drift. Manual coders invent new labels for the same idea over a long session. A consistent, auto-merged codebook fixes this.
• Stopping at the theme list. A ranked driver list is not the deliverable — the fix is. Tie each top driver to an owner and re-survey to confirm the theme shrank.
• Ignoring the silent middle. Passives rarely comment, so probe them deliberately ("what would make this a 9?") — their answers are the cheapest points to win.

From Number to Roadmap

The end state of good NPS comment analysis is a one-page picture: your score, its movement, and the three to five drivers behind it — each with a frequency, a sentiment, a band breakdown, and the verbatim quotes that make it undeniable in a stakeholder review. That artifact turns NPS from a vanity metric the leadership team glances at into a prioritized backlog the product team can act on. An AI-native platform like Koji produces it continuously, so the picture is always current rather than reconstructed once a quarter from a spreadsheet.

Related Resources

• Structured Questions Guide — scale and open_ended question types behind NPS
• NPS Survey Guide — running and scoring NPS end to end
• NPS Follow-Up Interviews — probing detractors and promoters
• Verbatim Analysis Guide — working with open-text at scale
• Open, Axial & Selective Coding — the manual coding method
• Thematic Analysis Guide — clustering codes into themes