If you are a founder with a pilot running at an enterprise customer, the person who signed it is not the person who decides whether it becomes a contract. Interview the people who actually use your product, twice during the pilot and once before the review, and you walk into that review with evidence from the users instead of a hopeful message from the sponsor. This playbook shows you how to run those interviews with AI-moderated conversations in Koji, what to ask, and how to turn the answers into a one-page pilot readout.
Why pilots die in the gap between sponsor and users
Say you are a two-person startup and a VP at a 3,000-person company signed a 10-week pilot with you. Twelve of her people have access. Every Friday she tells you it is "going well". In week nine, she asks for a call with her boss, procurement and a security reviewer. You have never spoken to any of them, and you have spoken to only three of the twelve users.
That gap is where pilots fail. The buying side is bigger than the person you know:
- Forrester's 2026 State of Business Buying report, released in January 2026, found the typical B2B purchase involves 13 people inside the buyer's company and nine outside influencers, and that procurement is a decision-maker in 53% of buying cycles (Forrester, January 2026).
- Deloitte's State of AI in the Enterprise 2026, a survey of 3,235 business and IT leaders in 24 countries fielded in August and September 2025, found that only a quarter of organisations say 40% or more of their AI experiments have reached production. Deloitte calls this the proof-of-concept trap (Deloitte 2026, reported by Unite.AI).
- Gartner predicted in June 2025 that over 40% of agentic AI projects will be canceled by the end of 2027, citing rising costs, unclear business value and inadequate risk controls (Gartner press release, 25 June 2025).
- MIT's NANDA initiative reported in August 2025 that about 5% of generative AI pilots reached rapid revenue acceleration, while tools bought from specialised vendors succeeded about 67% of the time (Fortune, August 2025). The headline measures business impact, not technical success.
Three of those four numbers are about AI pilots, so treat them as the sharpest evidence rather than a law of nature. The mechanism is the same for any enterprise pilot: the budget holder, the reviewers and the daily users each hold a veto, and a sponsor's optimism tells you about one of them.
The cost of not interviewing the users is that you learn about a stalled adoption from silence. Logins fall in week six, the sponsor goes quiet, and the "we will revisit next quarter" email arrives with no reason attached.
What good looks like
The best founders treat a pilot as a research project with a decision date. Here is the method, and it is prescriptive on purpose.
1. Write the decision down in week 0. Put three things in one paragraph that the sponsor agrees to in writing: what "success" means in numbers, who decides, and on which date. Your interviews test those three things, not general satisfaction.
2. Interview in three waves. Day 5 to 7 (first-use friction), around the midpoint (habit and workarounds), and 14 days before the review (verdict and conditions). A single exit interview at the end is too late to fix anything.
3. Sample by role, not by volume. Use this rule, which is Koji's own working guideline built from the committee sizes above, not an industry standard:
| Group | Who | How many |
|---|---|---|
| Sponsor | The person who signed | 1 per wave |
| Daily users | People with access and a task to do | Everyone if 12 or fewer; otherwise 12 chosen across teams and usage levels |
| Non-users | Access granted, no logins | At least 3, they teach you the most |
| Reviewers | IT, security, procurement, the boss's boss | 1 each, once, before the review |
4. Ask about the week, not the product. "Tell me about the last time you opened it" beats "Do you like it?". If you want a refresher on question hygiene, the Mom Test guide covers it.
5. Separate adoption from value. A user can love a tool and never use it again because their manager does not ask for the output. Ask one question about each.
6. Decide in advance what a "no" looks like. If fewer than half the daily users name a task they would now do with your product, that is a no on adoption, regardless of what the sponsor says.
Common mistakes
- Interviewing only the sponsor and the two friendliest users.
- Asking "would you buy this?" of someone with no budget.
- Skipping the non-users, who hold the reason it stalled.
- Waiting until week nine. Findings from week two can still change the pilot.
- Promising features in the interview. You are collecting, not selling.
- Sending the sponsor raw quotes with names, which can embarrass users who spoke honestly. Agree in advance whether answers are attributed.
How to set it up in Koji, step by step
Koji runs each interview as a conversation with an AI moderator that asks follow-ups, so twelve users can answer on their own time without you booking twelve calls inside a customer's calendar. Text interviews work on every plan. Voice interviews need the Interviews plan. A text interview costs 1 credit and a voice interview 3, so 12 users across two waves is 24 credits in text or 72 in voice, against 29 credits a month on Insights and 79 on Interviews. Previews are not counted or charged. Start from the quick start guide if you are new.
- Start the study. In the dashboard, open the template gallery, filter by Founders and Startups and pick Solution Validation Interviews. It is the closest template: it confirms the current workflow before testing the concept. Or paste the brief below into the "What do you want to research?" box.
- Shape the research brief. Name the decision (renew, expand or stop), the review date, and the three success numbers you wrote in week 0. Upload the pilot scope document as a context file so follow-up questions stay on topic.
- Add structured questions where you need a number. Keep 2 or 3 scale or single-choice questions so you can compare waves. Everything else stays open ended.
- Choose voice or text. Text is easier for users at their desk and with security-conscious employers. Voice produces richer answers for busy sponsors. You can offer both.
- Set guardrails and preview. Tell the interviewer what to avoid, for example quoting your pricing or naming other customers. Run the preview yourself first.
- Invite participants. Use personalized interview links so every user gets their own link and you can see who has answered. On the Interviews plan you can import the user list from a CSV. Ask the sponsor to forward the invitation: it carries more weight from them than from you.
- Read the report. The report groups answers into themes and links each finding to the quotes behind it.
Example questions for the day-7 wave
- Walk me through the last time you used it. What were you trying to get done?
- What did you do before we arrived, for that same task?
- On a scale of 1 to 10, how likely are you to still use it a month from now?
- What gets in the way of using it more? (single choice: access, time, trust in the output, my manager does not ask for it, other)
- Who else would need to be happy for this to stay?
- If it vanished tomorrow, what would you do instead?
Copy this interview guide
You can paste this into any tool. The brief for Koji is one line: "We are 5 weeks into a 10-week pilot at [company]. We need to know which users have made it part of their work, what blocks the rest, and what each stakeholder needs to see to approve a contract on [date]."
Opening. "Thanks for helping. I am not testing you, and there are no wrong answers. Honest criticism helps most."
| # | Question | Type | Probe |
|---|---|---|---|
| 1 | Tell me about the last time you used it. | Open | What happened right before and after? |
| 2 | What were you doing for this task before? | Open | What was the slowest part? |
| 3 | How likely are you to use it again next week? | Scale 1 to 10 | What would move that up two points? |
| 4 | What stops you using it more? | Single choice | Tell me about the last time that happened. |
| 5 | Which task would you hand over to it completely? | Open | What would you need to see to trust it? |
| 6 | Who in your team uses it, and who does not? | Open | Why do you think that is? |
| 7 | What would your manager say about it? | Open | What would they want to see in numbers? |
| 8 | If we ended the pilot today, what would you lose? | Open | How would you cope without it? |
| 9 | What is the one thing we must fix before the review? | Open | What happens if we do not? |
Closing. "Is there anything I did not ask that I should have?"
Wave 2 repeats 3, 4 and 8 so you can compare. The reviewer wave swaps in: "What would need to be true for you to approve this?" and "What have you seen go wrong with tools like this before?"
What you get
Within hours of each wave closing you have a report with themes, the share of users behind each one, sentiment and the quotes that support them. Open a finding and you see the interviews behind it, which is what you want when someone in the review asks "who said that?" (see tracing findings to evidence).
How you use it the next day:
- To the sponsor, 14 days out: a one-page readout. Three themes from users, the adoption numbers, and the two blockers you will fix before the review. A sponsor with that page can defend the contract to their boss.
- To your own team: the blockers become the next sprint. A user who says "I do not trust the output" is a roadmap item, not a sales problem.
- To the reviewers: a short answer to the concerns they gave you, with their own words quoted back.
- To yourself: a decision on whether the pilot is a yes, a not-yet with named conditions, or a no worth knowing now.
The same playbook elsewhere
Customer success teams run it as an onboarding check-in, asking the whole account instead of the champion: see the renewal risk playbook. Sales teams run the buyer-side version after a deal stalls: no-decision loss interviews. Private equity operating partners use it to read a new portfolio company's customers: the 100-day customer baseline. The parts you can borrow are the committee sampling and the pre-agreed definition of success. For the investor-facing version of your evidence, see investor-ready customer evidence.
Why Koji for pilot interviews
| Criterion | You calling users | Usage survey | Koji interviews |
|---|---|---|---|
| Depth | High, one call at a time | Shallow | Follow-up questions on every answer |
| Time to results | Days of scheduling | Fast | Users answer when free |
| Reach | 3 to 5 users | Whole group, low response | Whole group, personal links |
| Consistency | Varies by call | Fixed | Same guide every time |
| Candour | Users guard words with the vendor | Moderate | Conversation with an AI, not the vendor |
| Analysis effort | Notes to write up | Charts only | Themes with linked quotes |
| Wave-to-wave comparison | Manual | Possible | Repeat scale questions and compare |
Reasons that hold up in practice:
- You are not in the room. Users answering an AI moderator can criticise a vendor's product without worrying about your feelings or their sponsor's.
- Personal links tell you who has not answered. The non-users are the most useful group and the hardest to reach.
- Structured and open questions sit in one interview. You get a comparable number and the reason behind it.
- Findings trace to quotes. That matters when a reviewer pushes back.
- Voice or text per user. A security-conscious team can answer in text.
Questions you might have about AI interviews
"Will users at a large company talk to an AI?" They see a landing page that says how long it takes and how the answers are used, and they choose voice or text. In pilots the invitation should come from the sponsor, which does more for response than the tool does. Preview it yourself, and keep it to ten minutes.
"Will it make the sponsor feel bypassed?" Not if you ask first. Offer to share a one-page readout in return for the invitation. Sponsors tend to welcome evidence they did not have to chase.
"Can I trust the analysis?" Every finding in the report links back to the interviews that support it, so you can check a claim yourself before you put it in front of anyone. The interview also scores each response for quality, so thin answers do not drive a theme.
"Is it safe to put customer staff data in it?" Decide what you collect. You can keep names out of the readout and use only themes and unattributed quotes. If the customer needs contractual controls such as bring-your-own-key, the Enterprise plan offers them (BYOK docs).
When Koji may not be the right fit
If your pilot has two users, call them. A conversation you run yourself will teach you more than a tool, and the tool adds nothing. The same is true if the customer bans third-party tools for staff, or if the decision is political and nothing the users say will move it. In those cases, work the stakeholder map by hand and use interviews later.
Metrics to track
- Share of invited users who answered, by wave (aim for over half of daily users)
- Share of daily users who name a task they now do with your product
- Day-7 to day-35 change in the "will you still use it" score
- Number of non-users reached and the top blocker they name
- Blockers fixed before the review
- Pilot outcome: converted, extended with conditions, or ended
Run it this week
Pick the pilot with the nearest review date. Write the three success numbers, put the guide above into a study, preview it, and send the first wave to your users through the sponsor. To start from the template, sign up and open Solution Validation Interviews in the gallery, or see how other founders use Koji at /for/founders and browse more playbooks at /use-cases/founders. Related reading: the customer discovery interview guide.
