Quarterly Planning Evidence Sprint: A Product Manager Playbook for Walking Into Planning With Customer Proof

Say your planning kickoff is three weeks away and the draft roadmap has five bets on it. Two came from customers, one from the CEO, one from a competitor's launch and one from you. This playbook shows you how to run a three-week evidence sprint of 24 to 30 short AI-moderated interviews, so that on planning day every bet carries a grade for how well customers back it, and the argument is about what to do instead of whose opinion wins.

Why the planning evidence gap matters now

Planning is where a quarter of engineering time gets committed in one meeting. It is also where the least customer evidence shows up. Four recent surveys explain why.

Photo by Kelly Sikkema on Unsplash

• Frameworks lose to escalations. ProductPlan and Product-Led Alliance's State of Product Management 2026, the 11th annual edition, draws on 250 product leaders. Its headline finding is that more than 60 percent of prioritization frameworks get overridden by leadership escalations (ProductPlan and Product-Led Alliance, 2026). Your RICE scores are not what loses the argument. Your inputs are: nobody can check a "confidence: high" against anything.
• There is no time to find out. Atlassian's State of Product 2026 surveyed more than 1,000 product professionals in the US and Europe. 49 percent said they lack enough time for strategic planning, roadmap development or deep analysis (Atlassian, 2026, as summarised by Mind the Product). The same coverage notes that 85 percent feel empowered to lead or influence strategy, yet 84 percent worry their products won't succeed in the market.
• The customer habit is thin. IdeaPlan's State of PM 2026 summary reports that only 28 percent of PMs hold two or more customer conversations a week, and 11 percent rarely or never talk to customers (IdeaPlan, 2026). A plan built on conversations from last quarter is a plan built on last quarter.
• New bets arrive without customers attached. UXCam's 2026 survey of 157 product professionals in 34 countries found that 70 percent of leaders grew their AI budgets while only 6 percent set AI objectives (UXCam, 2026). UXCam sells product analytics, so read it as a signal about pressure, not a measure of customer demand. That is the point: AI bets enter roadmaps because someone senior asked, and nobody has checked whether customers did.

You are measured on whether what ships moves adoption, retention and revenue. Planning is the one meeting where you can change what ships at almost no cost. Walk in without evidence and the loudest voice sets the quarter. The cost shows up two quarters later, as a feature in the bottom of the usage report and a retro that says "we should have talked to customers first".

What good looks like

Treat planning prep as a three-week sprint with a fixed shape. Teresa Torres defines continuous discovery as at a minimum, weekly touchpoints with customers, by the team building the product, where that team is conducting small research activities, in pursuit of a desired product outcome. If you already do that, the sprint tops up the bets the weekly calls did not cover. If you don't, the sprint is how you start.

1. Grade every bet before you talk to anyone. Put the draft bets in a list and give each one an evidence grade. We use a five-step ladder, compiled by Koji from common product discovery practice: each step is stronger than the one below it.

Most bets start at "assumed" or "anecdotal". That is normal. The sprint exists to move them up, or to show they can't move.

2. Cap it at three bets per study. Each participant should hear no more than three problem areas. More than that and they answer about whichever one they find easiest to talk about. Five bets means two studies with a shared opening.

3. Interview three groups per bet, not one. Requesters are the people who already asked for it. They are over-represented in your tickets and under-represented in your revenue. Add:

• Requesters who asked for it (about a third of completes). They give the story behind the ask.
• The silent middle: active customers in the same segment who never asked (about half). They tell you whether the problem is common.
• Lapsed or non-adopters: churned, downgraded or never activated (about a sixth). They tell you what the bet would not have saved.

4. Decision rule for sample size. Get at least 8 relevant completes per bet before you grade it above "anecdotal". Total 24 to 30 per study. Below 8 on a bet, it stays "anecdotal" however loud it is. Past 30 you mostly hear the same themes again, so stop.

5. Run it on a clock. Counted back from planning day (T):

• T-21 (Monday): list the bets, grade them, write the one decision each could change ("build as scoped, shrink, swap, or wait").
• T-20: build the three groups from your analytics and CRM and send personalised links.
• T-19 to T-8: field. Send one reminder on T-14.
• T-7: read the report. Re-grade each bet.
• T-5: write a one-page evidence brief per bet (grade, counts, two quotes, the decision).
• T-3: pre-wire the executive most likely to override. Show them the brief, not the deck.
• T: planning. Bets with grade 3 or higher get the top slots. Bets below 3 go in as labelled hunches, or not at all.
• T+30: rerun a ten-person pulse on the bets that were picked, so the next planning cycle starts with fresh evidence.

6. Ask about last week, not next year. "Would you use this?" produces yes. "Tell me about the last time you hit this" produces facts. Anchor on the most recent real event and ask what they did.

Common mistakes

• Interviewing only the people who asked. They confirm the ask.
• Describing the bet in the question. You will hear your own pitch back.
• Counting a vivid quote as a pattern. Look for the same barrier in at least three people.
• Grading your own bets the night before planning. Grade before the sprint and again after, and keep both grades visible.
• Presenting findings as slides nobody can trace. Show the count and the supporting quotes.

Copy this interview guide

Paste this into any tool. Replace [job] with the thing your customers are trying to get done, and [problem A], [problem B], [problem C] with the problems behind your three bets, written as a customer would say them, never as a feature.

Research brief (one line for Koji's "What do you want to research?" box): "Which of our three planned bets for next quarter solve a problem that active customers in [segment] have recently worked around, paid for or switched over, and which rest on an assumption?"

Opening line: "Thanks for helping. We are planning next quarter and want to hear what is really getting in your way. There are no wrong answers, and you won't hurt anyone's feelings."

1. (Open) "Tell me about the last time [job] didn't go the way you wanted. What happened?" Probe: "What did you do next?"
2. (Open) "What matters most for your team in the next three months?" Probe: "What could get in the way?"
3. (Ranking) "Rank these by how much they cost you right now: [problem A], [problem B], [problem C], none of these." Probe: "Tell me about the one you put first."
4. (Single choice) "How do you handle [problem A] today? I don't / a spreadsheet or workaround / another tool / someone does it for us / it's not a problem." Probe: "Walk me through the last time."
5. (1 to 10 scale) "How much does [problem A] get in the way of your goals this quarter?" Probe: "What makes it that number and not two lower?"
6. (Open) "What have you tried to fix it? What happened?" Probe: "Why did that stop?"
7. (Yes/no) "Has this ever affected a decision to renew, switch or buy something?" Probe: "Tell me about that."
8. (Open) "Who else in your company feels this, and how do they cope?" Probe: "Who decides whether it gets fixed?"
9. Close: "Is there anything I should have asked and didn't?"

Repeat questions 4 to 7 for each of the other problems, keeping it to three. The matching template in Koji's gallery is "Continuous Discovery Interviews" under Product Managers: it already has the open, scale and follow-up structure, and you swap in your bets.

How to set it up in Koji, step by step

1. Start a study. In the dashboard open the template gallery, filter by Product Managers and pick Continuous Discovery Interviews, or paste the one-line brief above into the "What do you want to research?" box.
2. Shape the brief. Name the decision ("which bets get the top slots next quarter") and your hypotheses. Upload the draft roadmap summary and any bet descriptions as context files so the interviewer knows your product's vocabulary. Then set guardrails: list the topics the interviewer must not discuss, such as unreleased features, pricing and names of other customers. Customers should tell you about their problems, not hear your roadmap.
3. Write the questions. Use the nine above. Set question 3 as ranking, question 4 as single choice, question 5 as a 1 to 10 scale, question 7 as yes/no and the rest as open with one or two follow-ups.
4. Choose voice or text. Text interviews work on every plan and cost 1 credit each. Voice interviews need the Researcher plan or above and cost 3 credits each. Thirty text interviews are 30 credits. Run the preview yourself first: it is saved with its own analysis and does not count as a response.
5. Send personalised links by group. Create a link for each person from your three groups so you know which group said what. Personalised links work on every plan. Importing a CSV from your CRM needs the Researcher plan or above. Set a response limit at your target so you stop at 30.
6. Read the report and publish the brief. The report groups answers into themes with the number of participants behind each, and each finding opens to the quotes that support it. Share the published report link with the executive you pre-wired. Refreshing the report as late responses arrive costs 5 credits.

Read more: Research-driven roadmap prioritization, the roadmap evidence audit, personalised interview links and importing participants via CSV.

What you get

On T-7 you hold a report, not a pile of transcripts.

• A grade per bet. Each bet moves up, or stays where it was, with the counts to show it.
• Quotes you can paste. Each finding links to the people behind it, so the debate is about the decision.
• A ranking distribution. If you ask question 3 the same way for every participant, you see how many put each problem first, split by group.
• A published report link for design, engineering and the executives who will weigh in.

Use it the next day: put the one-page brief for each bet at the top of the planning doc, with the grade and the decision you wrote down on T-21. When someone overrides the order, the override is now a visible choice against a visible grade, and you can write it down.

The same playbook elsewhere

The idea carries wherever a decision is about to be made on opinion.

• Agencies: interviewing every team at a prospect before writing a proposal. See new-client stakeholder discovery.
• Business analysts: checking AI-drafted requirements with the people who will use them. See stakeholder validation interviews.
• Product, after the plan: once a bet ships, the launch-window playbook tells you whether it worked. If the bet is an AI feature, see AI feature adoption research.
• Founders: the same sprint before a fundraise, aimed at investor reference calls.

Why Koji for a planning evidence sprint

Analytics and ticket tags are essential and already exist. Scheduled calls give the deepest conversation if you can book them. Surveys reach many people fast. The trouble is the combination you need in a three-week window: depth, reach into people who did not ask, and the same questions for everyone.

1. Follow-up on every answer. Each open question can have one or two follow-ups, so "it's a hassle" becomes the step where they got stuck.
2. No calendar. A personalised link lets each person answer in voice or text when it suits them, which matters when recruitment is the thing blocking your weekly conversations.
3. Mixed question types in one conversation. Ranking and scale questions give you the distribution, open questions give you the story, and you get both from the same person.
4. The same guide for every participant. You can compare requesters with the silent middle because both heard the same questions with the same probing.
5. Findings with counts and quotes. The report shows how many people stand behind each theme and opens to the supporting quotes, so you can check a finding before you present it.

Questions you might have about AI interviews

Will customers tell an AI the truth about our roadmap? They don't see your roadmap. You list the problems, never the solutions, and set guardrails so the interviewer never discloses unreleased work. Participants see the time estimate and privacy note on a welcome page before they start, and can choose voice or text.

Can it probe like a researcher would? It follows your questions and the follow-up setting you chose, and reads the context files you uploaded. Preview the interview yourself and read the first five transcripts before the rest come in. Each interview also gets a quality rating, so you can set aside thin ones.

Where does the data go? Interview data is stored in the EU, and Koji does not use customer interview data to train AI models. On Enterprise plans you can bring your own key.

When Koji may not be the right fit

If you have fewer than 30 active customers in the segment, call them. You will learn more from eight conversations than from any tool. If a bet turns on how people use a screen, you need usability testing with recordings. And if planning is next week, run a ten-person pulse on your top two bets rather than the full sprint.

Metrics to track

• Share of roadmap bets at grade 3 or higher on planning day, versus last quarter.
• Number of overrides of the evidence order, and whether they were logged.
• Completes per bet, and response rate per group.
• Days from sprint start to the first documented change in a bet's scope.
• Share of shipped bets still used at day 30 (see the launch-window playbook).

FAQ

How far ahead of planning should I start? Three weeks. Invite on T-20, field through T-8, read the report on T-7 and write briefs by T-5.

How many customers should I interview? 24 to 30 per study, with at least 8 relevant completes for each bet you want to grade above "anecdotal".

Who should I include besides people who asked for the feature? Active customers in the same segment who never asked, and lapsed or non-adopting accounts. Requesters alone confirm the ask.

What if leadership still overrides the evidence? Then the override becomes visible. Log the bet, its grade and the decision. Over two quarters that record is the most persuasive evidence you will have.

Which plan do I need? Text interviews, personalised links and report publishing work on every plan, and the Free plan includes 10 interviews to pilot. Voice interviews and CRM import need the Researcher plan or above.

Start from the template

Open Koji, choose the Continuous Discovery Interviews template, paste the brief and send your first ten personalised links this week. See the Product Managers page for more, or browse every playbook for your role in the use-case library.