AI Customer Research for AI Product Companies (2026)

Short answer: AI product companies have a research problem their analytics cannot solve: the product behaves differently for every user on every run, so usage data cannot tell you whether an output was good. The fix is to ask, systematically. A platform like Koji runs AI-moderated interviews by voice or text, asks adaptive follow-up questions about the specific thing a user was trying to do, and analyzes hundreds of conversations automatically. That is how you learn whether users trust your outputs too much or too little, what they did when the model failed, and which model upgrade quietly broke a workflow.

If you build with AI, you already know the uncomfortable version of this: your evals are green and your users are unhappy. This guide explains why, and what to run instead.

Why Researching an AI Product Is Uniquely Hard

Adoption has outrun the measurement practice around it. In Stanford HAI's AI Index, the share of survey respondents reporting AI use at their organization rose to 78 percent in 2024, from 55 percent in 2023. Teams are shipping AI features far faster than they are building ways to tell whether those features are any good in someone else's hands.

The product is nondeterministic, so complaints are not reproducible. A user says the summary was wrong. You run the same input and get a reasonable summary. Neither of you is lying; the run differed. Traditional bug-report and analytics workflows assume reproducibility, and they quietly discard the majority of real AI quality problems because nobody can reproduce them on demand.
Offline evaluation does not predict satisfaction. Benchmarks and eval suites measure what you thought to measure, on inputs you chose. They cannot see that a user abandoned the feature because the output was technically correct but written in the wrong register, arrived too late to be useful, or required more verification effort than doing the task manually. The eval gap is a research gap.
Trust is the actual product surface, and it fails in both directions. Users can over-rely, accepting a wrong answer because it was fluent, or under-rely, ignoring a correct answer because an earlier one was wrong. Both destroy value, both are invisible in a usage graph, and both are discoverable only by asking people what they did with the output and why.
The product changes underneath the user. Model upgrades, prompt changes, and retrieval updates shift behaviour between releases. A workflow someone built around a quirk can break without a single line of your own code changing, and your release notes will not mention it.

How AI-Moderated Interviews Fix It

You can ask about a specific run. Instead of asking whether users like the AI feature, Koji's interviewer anchors on an episode: tell me about the last time you used it, what you were trying to produce, and what you did with what came back. Nondeterminism stops being a measurement problem once the unit of analysis is the user's episode rather than the model's output.
Adaptive follow-ups catch the verification step. The important behaviour is what happens after the output appears. When a user says it was mostly right, Koji probes the rest: which part did you change, how did you know it was wrong, and how long did checking take? That is where the real cost of an AI feature lives.
Structured and qualitative in one pass. Koji supports six structured question types, so a single study can quantify trust and explain it. The types are open_ended, a free-form qualitative answer with AI follow-up probing; scale, a numeric rating such as a one-to-five confidence score; single_choice, pick one; multiple_choice, pick one or more; ranking, order items by preference; and yes_no for a binary. Details in the structured questions guide.
Interview modes that match the maturity of the question. Koji studies can run structured, which follows the key questions closely and suits regression checks after a model change; exploratory, open-ended discovery that follows interesting threads, which suits a feature nobody understands yet; or hybrid, starting structured and opening up when something interesting appears.
Real humans, not simulated ones. It is tempting to evaluate an AI product with synthetic personas. That substitutes your model's prior for your users' reality and will confirm whatever you already believe. Synthetic users: a methodology review sets out where simulation is and is not defensible. Koji is built to reach actual users at the scale that made simulation tempting in the first place.
Automatic analysis as responses land. Themes are coded across every transcript and the report assembles while fieldwork is still open, which matters when you are deciding whether to roll back a model change this week. See real-time research insights.
Quality scoring on spend. Only conversations scoring 3 or higher consume a credit; a text interview costs 1 credit and voice costs 3.

High-Value Research Playbooks for AI Products

1. Trust Calibration: Over-Reliance and Under-Reliance

Ask users what they did with the last output: shipped it unchanged, edited it, verified it against another source, or discarded it. A single_choice question gives you the distribution and open-ended follow-ups explain each branch. Over-reliance shows up as shipped unchanged on tasks the model is known to get wrong, which is a safety finding, not a satisfaction one.

2. The Degraded State

Every AI product fails: the model is slow, the context is missing, the retrieval returns nothing useful. What users experience in that moment determines whether they come back. Interview people about a failure they remember and what they did next. Graceful degradation for AI features covers the design side of the same question.

3. The Prompt-to-Intent Gap

Users do not describe tasks the way your prompt template expects. Collect what people were actually trying to achieve, in their words, before they touched the input box. This is the single highest-yield study for an AI product, because it reveals the jobs your interface is accidentally hiding.

4. Model-Upgrade Regression Research

Before and after a model change, run the same structured study and compare. Keep the questions identical so the comparison means something. Scale questions on output quality plus open-ended probes will surface the workflow that broke even when your aggregate evals improved.

5. Claim Perception Before You Advertise

What users hear when you say AI-powered is rarely what you meant, and the gap carries real regulatory exposure. Test the claim itself on the audience before it ships. AI claims substantiation explains how to prove an AI claim and what research can and cannot do there.

6. Pricing and Packaging for Usage Meters

Token, credit, and run-based pricing is hard for buyers to forecast, and unpredictability has a political cost inside their company. Interview buyers about how they budget for a meter they cannot predict, and use ranking questions across candidate meters to find the one that feels fairest rather than the one that is easiest to bill.

Why Not Just Use Analytics or Evals?

Analytics tell you that a user regenerated an output four times; they cannot tell you whether the fourth one was good or whether the user gave up and did it by hand. Evals tell you how the model performs on cases you selected; they cannot tell you that the output format is wrong for the user's downstream tool. Both are necessary and neither is sufficient, because the quantity you actually care about, whether this output helped this person do their job, exists only in the user's head. A survey does not get you there either, since it cannot ask the next question when someone says it was mostly fine. Koji's interviewer asks the next question, and its quality scoring means a thoughtful fifteen-minute answer and a one-word answer are not weighted the same. See AI interviews versus surveys.

Getting Started

Pick one AI feature and one decision about it, usually whether to keep it, change its interface, or roll back a model change.
Define the episode you want people to recall. Specificity beats generality: the last time you used it this week, not in general.
Write a short brief and let the AI consultant draft the guide, then add probes on the verification step.
Recruit from inside the product, ideally triggered just after a real use of the feature while memory is fresh.
Run structured if you are comparing against a baseline, exploratory if you are still learning what the feature is for.

A Worked Example: The Feature With Great Evals and Falling Usage

A team ships an AI summarisation feature inside a workflow tool. Eval scores are strong, latency is fine, and weekly usage falls for six straight weeks. Analytics show a high regeneration rate, which the team reads as users refining their prompts.

They run an async study across forty recent users, anchored on the last specific use, with a single_choice on what the user did with the output and open-ended follow-ups on the verification step. The report overturns the reading. Regeneration was not refinement; users regenerated because the summary omitted the one section they needed, so they ran it again hoping for different luck, then gave up and skimmed the document manually. Verification took longer than the task. The fix was not a better model but a control that let users say which sections mattered, plus a visible indication of what the summary had skipped. Usage recovered without a model change.

A Note on What Not to Research

Do not ask users to evaluate your model in the abstract, and do not ask them to predict whether they would use a hypothetical AI feature. Both questions produce confident, worthless answers, because people are poor forecasters of their own behaviour with tools they have not used. Anchor every question on something that actually happened. Equally, resist the temptation to replace this research with a simulated panel: when the thing you are measuring is how a real person reacted to a nondeterministic output, a model's guess about that reaction carries none of the information you need. Keep the humans in the loop, and keep the loop short.

Frequently asked questions

How do you research a product that gives different answers every time?

Change the unit of analysis from the output to the episode. Instead of asking whether the AI is accurate, ask about one specific recent use: what the person was trying to produce, what came back, and what they did next. Nondeterminism makes outputs incomparable but it does not make experiences incomparable, and twenty episodes will show you a clear pattern even though no two runs matched.

Why are our eval scores good while users are unhappy?

Because evals measure what you chose to measure on inputs you chose. They cannot see wrong register, bad output format for a downstream tool, latency arriving after the moment of need, or verification cost exceeding the time saved. Those are experience failures rather than model failures. Treat evals as a regression guard on known cases and use interviews to find the failure classes your eval suite does not contain yet.

Can we use synthetic or AI-simulated users instead?

Not for this question. The quantity you are measuring is how a real person reacted to a nondeterministic output, and a model asked to predict that reaction will reproduce your own assumptions rather than test them. Simulation can help you pilot a question wording or stress-test a guide, but it cannot substitute for the reaction itself. Reserve it for instrument development and keep real users for findings.

How do we tell over-reliance from genuine satisfaction?

Ask what the user did with the output, not how they felt about it. Shipping an output unchanged on a task where your model is known to be unreliable is over-reliance, even when the user reports high satisfaction. Pair a single_choice on action taken with an open-ended probe on how they checked it. A user who cannot describe any verification step and reports high confidence is the risk case.

When should we run research around a model upgrade?

Run the same structured study immediately before and shortly after the change, keeping the questions identical so the comparison is meaningful. Aggregate eval improvements routinely hide a specific broken workflow, and the users who built a process around previous behaviour are the ones who notice first. A short structured study is also fast enough to inform a rollback decision while rolling back is still cheap.

How many interviews do we need for an AI feature study?

For one feature and one decision, patterns typically stabilise between fifteen and twenty-five interviews. Coverage matters more than volume: include heavy users, people who tried the feature once and stopped, and users whose task sits at the edge of what the model handles well. A sample made only of enthusiasts will tell you the feature is excellent right up until it churns.

AI Customer Research for AI Product Companies: How to Research a Nondeterministic Product