{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-10-10T11:59:00.699Z"},"content":[{"type":"blog","id":"ea5ddef5-3778-403d-b05e-54fc0a704ca2","slug":"ai-feature-adoption-research-product-managers","title":"AI Feature Adoption Research: A Product Manager Playbook for Finding Out Why Users Do Not Trust or Use Your AI Feature","url":"https://www.koji.so/blog/ai-feature-adoption-research-product-managers","summary":"A product manager playbook for diagnosing low adoption of an AI feature. Covers the 2026 evidence (51% of software makers say under a quarter of customers use their AI features; trust is the top barrier), how to split users into four behaviour groups, example interview questions, and how to run the study with AI-moderated interviews in Koji.","content":"You shipped the AI feature. The launch post went out, the demo got applause, and now the dashboard shows a flat line. **AI feature adoption research** is the work of finding out why: which users tried it, which ones quietly went back to the old way, and what each group needed to see before they would rely on it. Run as AI-moderated interviews, you can talk to 30 or more of those users in a few days, split by what they did rather than what they say, and walk into your next roadmap review with reasons instead of guesses.\n\n## Why AI feature adoption research matters now\n\nMost teams have shipped AI into an existing product by now. Far fewer know whether anyone uses it.\n\n- In Banyan Software's 2026 benchmark of vertical software operators (more than 260 founder and CEO responses), 51% said fewer than one in four of their customers use the AI features they built. [CIO reported the survey in August 2026](https://www.cio.com/article/4207518/that-shiny-new-ai-feature-your-customers-wont-use-it.html). Max Risen, Banyan's president of M&A, told IT Brew the likely cause is companies \"building for the sake of building\".\n- Userflow's 2026 survey of 107 senior product leaders found that 44% name trusting the output as the place users struggle most. That ranks above understanding the feature (26%), fitting it into a workflow (25%) and discovering it exists (21%). The same survey says only 33% of teams measure outcomes such as task completion or time saved, and 30% have no reliable way to tell whether the AI helps at all. Details are on [Userflow's write-up](https://www.userflow.com/blog/why-ai-feature-adoption-is-stalling). Note that this is what product leaders believe about their users, not what users said.\n- The problem is not new. In 2024, [Mind the Product and Pendo](https://www.mindtheproduct.com/product-survey-findings-only-15-of-users-are-embracing-ai-features/) reported that only 15% of the 300+ product leaders they surveyed said their users were embracing AI features.\n\nRead those three together. Trust is the leading barrier, and the metrics most teams watch (clicks, sessions, weekly active users) cannot see trust. A PM who is measured on adoption, retention and support load is flying with the one instrument that does not read the main variable.\n\nThe cost of not finding out is specific. You keep funding a feature most of your base ignores, or you cut one that a small segment depends on, and you will not know which until someone leaves.\n\n## What good looks like\n\nTeresa Torres, who wrote _Continuous Discovery Habits_, defines the practice as the team building the product talking to customers at least weekly, in small research activities tied to a desired outcome. Her story-based approach is the right base here: ask for a specific time the user did something, not their opinion of the feature. Everything below builds on that.\n\n**1. Sample by behaviour, not by plan tier.** Split users into four groups:\n\n- _Never tried it._ They saw it, or never saw it.\n- _Tried once and stopped._ This is usually your richest group.\n- _Use it, but check everything it does._ Heavy verification is a trust signal.\n- _Rely on it._ They tell you what a good outcome looks like.\n\nPull each group from your analytics. If you use a product analytics tool, a cohort for each is a ten-minute job.\n\n**2. Interview enough to see each group's reasons repeat.** Hennink, Kaiser and Marconi (2017) found that new _codes_ stopped appearing at around 9 interviews, but understanding the _meaning_ behind them took 16 to 24. Nielsen Norman Group makes a similar point: interview studies vary more than usability tests, so five is often too few. With four groups, aim for 6 to 10 per group. That is about 30 to 40 conversations, which is where an AI moderator earns its place.\n\n**3. Ask about the last real task, not the feature.** \"Walk me through the last time you had to do X\" beats \"What do you think of the summary feature?\" Then ask what they did next: did they use the output as is, edit it, or redo it by hand?\n\n**4. Probe trust at the moment of use.** Ask what they looked at to decide whether the output was right. Ask what would have to be visible for them to stop checking. Trust problems hide in answers like \"I just glance at it\", which a good probe turns into specifics.\n\n**5. Get a number, then ask about the number.** A 1 to 10 confidence score is useful only when followed by \"why that number and not two points higher?\"\n\n**6. Map every finding to an outcome metric.** Pick one before you start: task completion, time on task, tickets avoided. If a finding cannot connect to it, park it.\n\n### Common mistakes\n\n- Interviewing only the people who use the feature. Non-adopters hold the biggest barriers.\n- Asking \"Did you like it?\" You get politeness.\n- Treating low usage as a discoverability problem when it is a trust problem, or the reverse. The interview should tell you which.\n- Running one study and calling it done. Trust shifts after model changes, new data sources and bad outputs, so repeat after any meaningful change.\n- Letting a loud example win. One dramatic quote does not outweigh a theme from 12 interviews.\n\n## How to set it up in Koji\n\nYou can run this on any plan for text interviews. Voice interviews need the Interviews or Enterprise plan.\n\n1. **Start a study from a template.** Create a new study and choose the Post-Launch Feedback Interviews template. It is written for exactly this situation: it asks about the need before the feature, how the person found it, what they did the first time, and why they came back or did not. Rename it for the feature you are researching.\n2. **Shape the research brief.** Fill in the decision this study informs (\"keep investing, redesign the trust cues, or cut\"), your current hypothesis (for example \"adoption is low because users cannot tell when the output is wrong\"), and who counts as a participant. The brief assistant can change any part of the brief on request, including screening questions and follow-up depth, and tells you what it changed.\n3. **Add screening questions for your four groups.** A single-choice question such as \"Which best describes your use of the AI assistant in the last 30 days?\" can qualify the groups you want and screen out the rest. Screening questions are checked before the interview starts. They apply to studies created from September 5, 2026, so create a fresh study if you are updating an older one.\n4. **Add structured questions where you need a number or a choice.** Koji supports open-ended, scale, single choice, multiple choice, ranking and yes/no questions. Use a scale for confidence and a ranking for \"what would make you trust it more\". Use open-ended questions for everything else and let the interviewer follow up.\n5. **Preview it.** The Preview tab lets you take your own interview by text or voice. Previews are free, do not count as responses and never appear in the report. Fix anything that sounds like a leading question before you invite real users.\n6. **Choose text or voice, then invite people.** A text interview costs 1 credit and a voice interview costs 3. Share the interview link, or use personalized links to send each user their own. If you want to find users inside your analytics, the [Mixpanel](/docs/mixpanel-research-integration) and PostHog integration guides describe how to trigger interviews from product events. If you need people who are not your users (for example, to understand why prospects distrust AI in your category), panel recruitment is available on paid plans.\n7. **Read the report.** Koji produces a report with themes and quotes, and charts for your structured questions. Only interviews that score 3 or higher on the quality score use credits or enter the report.\n\n### Example questions for this study\n\n- \"Think about the last time you needed to [task the AI feature helps with]. Walk me through what you did, from the start.\"\n- \"Did you use the AI suggestion at any point? What did you do with what it gave you?\" (probe: used as is, edited, discarded)\n- \"When it gave you a result, how did you decide whether it was right?\" (probe: what did you look at?)\n- \"On a scale of 1 to 10, how much do you trust it to do this task without you checking?\" (scale, then probe the number)\n- \"What would have to be true for you to stop double-checking its work?\"\n- \"Rank these in order of what would make you rely on it more: showing its sources, a way to undo, seeing how it reached the answer, a track record, an option to review before it acts.\" (ranking, adapt the options to your product)\n\n### Plans and cost\n\nFree includes a one-time grant of 10 credits and text interviews. Insights is €29 a month with 29 credits, enough for about 29 text interviews. Interviews is €79 a month with 79 credits, which covers 79 text interviews or 26 voice interviews. Check the pricing page for the current figures before you budget, because plans change.\n\n## What you get and how a PM uses it the next day\n\nWithin a few days you have a report that separates the four groups and shows where they diverge. The next-day uses are concrete:\n\n- **In the roadmap review:** \"Of the 'tried once and stopped' group, most could not tell when the summary was wrong. That is a design problem, not a marketing one.\" That sentence changes the conversation.\n- **In the backlog:** two or three trust cues with evidence behind them, such as showing sources or adding a review step.\n- **In the metrics plan:** the outcome metric you will use to judge the redesign, chosen from what users said mattered.\n- **In the go or no-go on cutting the feature:** a clear view of who depends on it. For the other side of that decision, see the guide to [feature sunset research](/docs/feature-sunset-research).\n\nYou can also chat with the study data to ask follow-up questions across all the interviews, such as \"which non-adopters mentioned data privacy?\"\n\n## The same playbook elsewhere\n\nThe structure carries over to other roles. Customer success teams run the same four-group split after a new onboarding flow. HR and people ops teams use it to understand why employees avoid an internal AI tool; Koji's guide to [employee AI adoption research](/docs/employee-ai-adoption-research) covers that. Founders use a lighter version in the first month after launching a feature to a handful of early customers. In all of them, the move is the same: segment by behaviour, ask about the last real task, and probe trust.\n\n## Why Koji for AI feature adoption research\n\n| What a PM weighs | Analytics alone | Survey or in-app prompt | Manual interviews | Koji |\n| --- | --- | --- | --- | --- |\n| Explains why | No | Thin | Yes | Yes, with follow-up probing |\n| Time to results | Immediate | Days | Weeks to schedule | Days |\n| Reach across 4 behaviour groups | Yes, but no reasons | Biased to responders | Limited by PM hours | 30+ interviews in parallel |\n| Consistency across interviews | Not applicable | High | Varies by moderator | Same brief for every interview |\n| Candour about a feature your team built | Not applicable | Social desirability | Social desirability | Participants talk to an AI, not the builder |\n| Analysis effort | Low | Low | High | Themes, quotes and charts generated |\n| Cost | Tool cost | Tool cost | PM and researcher time | From €29 a month |\n\nThe reasons, each tied to something you can check in the product:\n\n- **It probes.** Each open-ended question can have follow-ups, so \"I just glance at it\" turns into what they glance at.\n- **It mixes numbers and conversation.** Scale, choice and ranking questions sit inside the interview and show up as charts in the report.\n- **It screens.** You can send one link and let screening questions sort people into your four groups.\n- **It protects your budget.** Low-effort interviews are filtered out by the quality score and do not use credits.\n- **You can rehearse.** Free previews mean you hear the interview before your users do.\n\n## Questions you might have about AI interviews\n\n**Will users be honest with an AI about a feature my team built?** Some people criticise more freely when nobody from the team is listening live, though that is not guaranteed. Frame the opening as \"we want to hear what does not work\", and use the preview to hear how it sounds before you invite anyone.\n\n**Can an AI moderator probe as well as an experienced researcher?** It follows the depth you set per question and asks follow-ups based on what the person said. Read the first five transcripts and adjust the brief, as you would after a pilot with a new human interviewer. Koji's guide on [AI interview quality](/docs/ai-interview-hallucinations-bias-mitigation) explains the safeguards.\n\n**Where does the data go?** The [privacy and security guide](/docs/ai-interview-data-privacy-security) covers how interview data is handled, and the [GDPR guide](/docs/gdpr-compliant-ai-user-research) covers consent. If your company has strict requirements, read them before you invite customers.\n\n## When Koji may not be the right fit\n\nIf you need to watch people struggle in the interface, with clicks and eye movement, you want a usability test or session replay, and the interview is the follow-up. If your user base is only a few dozen people, a handful of calls you run yourself may serve you better. And if the real question is whether the AI output is accurate, that is an evaluation problem for your engineering team; interviews tell you whether users believe it is.\n\n## Metrics to track\n\n- Share of eligible users who tried the feature, and who used it more than once\n- Task completion or time on task for users of the feature versus a comparable group\n- Trust score (1 to 10) by behaviour group, before and after changes\n- Support tickets that mention the feature\n- Proportion of interviews that mention each trust barrier, so you can see which ones your changes move\n\n## Where to start\n\nCreate a study from the Post-Launch Feedback template, add the four-group screening question, and preview it. If you want a wider view of the role, see [Koji for product managers](/for/product-managers). For related reading, the [Product Manager's Guide to Customer Discovery with AI](/blog/product-manager-guide-customer-discovery-ai), the [2026 playbook for researching AI products](/blog/user-research-for-ai-products-2026) and the [Continuous Discovery Handbook](/blog/continuous-discovery-handbook-weekly-customer-interviews) pair well with this one. For the general method, see [feature adoption research](/docs/feature-adoption-research).\n","category":"Use Cases","lastModified":"2026-10-09T22:22:23.93344+00:00","metaTitle":"AI Feature Adoption Research for PMs","metaDescription":"Why do users not use your AI feature? A PM playbook: split users by behaviour, probe trust, and run 30+ AI-moderated interviews in days.","keywords":["ai feature adoption research","ai feature adoption","why users do not use ai features","product manager user research","post-launch feedback interviews","ai feature trust"],"aiSummary":"A product manager playbook for diagnosing low adoption of an AI feature. Covers the 2026 evidence (51% of software makers say under a quarter of customers use their AI features; trust is the top barrier), how to split users into four behaviour groups, example interview questions, and how to run the study with AI-moderated interviews in Koji.","aiKeywords":["ai feature adoption","product management","post-launch research","user trust in ai","ai-moderated interviews","product discovery"],"aiContentType":"guide","faqItems":[{"answer":"Aim for 6 to 10 per behaviour group (never tried, tried once, verify everything, rely on it), so about 30 to 40 interviews. Research on saturation suggests new themes stop appearing around 9 interviews but their meaning takes 16 to 24.","question":"How many users should I interview about an AI feature?"},{"answer":"In Userflow's 2026 survey of 107 product leaders, 44% said trusting the output is where users struggle most, ahead of understanding the feature, fitting it into a workflow or finding it. Interviews are how you confirm which applies to your users.","question":"Why do users stop using AI features?"},{"answer":"Ask about the last time they did the task the feature supports, whether they noticed the feature, and what they did instead. Avoid asking whether they liked it.","question":"What should I ask users who never tried the AI feature?"},{"answer":"Text interviews work on every plan, including Free with a one-time credit grant. Voice interviews need the Interviews or Enterprise plan. A text interview costs 1 credit and a voice interview 3.","question":"Which Koji plan do I need?"},{"answer":"A survey records the answer the user chooses. An interview asks what they did the last time, then follows up on the specifics, which is how you find trust problems that rating scales hide.","question":"How is this different from a post-launch survey?"}],"relatedTopics":["ai feature adoption research","post-launch feedback","product managers"]}],"pagination":{"total":1,"returned":1,"offset":0}}