{"site":{"name":"Koji","description":"AI-native customer research platform that helps teams conduct, analyze, and synthesize customer interviews at scale.","url":"https://www.koji.so","contentTypes":["blog","documentation"],"lastUpdated":"2026-10-28T16:27:32.332Z"},"content":[{"type":"blog","id":"cfa28a2f-d6fd-4968-b115-39e388f57cb4","slug":"quarterly-planning-evidence-sprint-product-manager-playbook","title":"Quarterly Planning Evidence Sprint: A Product Manager Playbook for Walking Into Planning With Customer Proof","url":"https://www.koji.so/blog/quarterly-planning-evidence-sprint-product-manager-playbook","summary":"A product manager playbook for the three weeks before quarterly planning: grade every roadmap bet on a five-step evidence ladder, interview requesters, the silent middle and lapsed accounts, use a ready nine-question guide, and run it in Koji with personalised links and a report that shows counts and supporting quotes.","content":"Say your planning kickoff is three weeks away and the draft roadmap has five bets on it. Two came from customers, one from the CEO, one from a competitor's launch and one from you. This playbook shows you how to run a three-week evidence sprint of 24 to 30 short AI-moderated interviews, so that on planning day every bet carries a grade for how well customers back it, and the argument is about what to do instead of whose opinion wins.\n\n## Why the planning evidence gap matters now\n\nPlanning is where a quarter of engineering time gets committed in one meeting. It is also where the least customer evidence shows up. Four recent surveys explain why.\n\n![A product team gathered at a whiteboard with sticky notes, planning which roadmap bets to back next quarter](https://images.unsplash.com/photo-1507925921958-8a62f3d1a50d?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3w5NDUzMDN8MHwxfHNlYXJjaHw3fHx0ZWFtJTIwcGxhbm5pbmclMjB3aGl0ZWJvYXJkJTIwc3RpY2t5JTIwbm90ZXN8ZW58MXwwfHx8MTc5MzE3MzgxMHww&ixlib=rb-4.1.0&q=80&w=1080 \"Planning is the one meeting where a quarter gets decided.\")\n\n_Photo by [Kelly Sikkema](https://unsplash.com/@kellysikkema?utm_source=koji&utm_medium=referral) on [Unsplash](https://unsplash.com/?utm_source=koji&utm_medium=referral)_\n\n- **Frameworks lose to escalations.** ProductPlan and Product-Led Alliance's State of Product Management 2026, the 11th annual edition, draws on 250 product leaders. Its headline finding is that [more than 60 percent of prioritization frameworks get overridden by leadership escalations](https://www.productplan.com/ebooks/the-state-of-product-management-report-2026) (ProductPlan and Product-Led Alliance, 2026). Your RICE scores are not what loses the argument. Your inputs are: nobody can check a \"confidence: high\" against anything.\n- **There is no time to find out.** Atlassian's State of Product 2026 surveyed more than 1,000 product professionals in the US and Europe. [49 percent said they lack enough time for strategic planning, roadmap development or deep analysis](https://www.mindtheproduct.com/inside-atlassians-state-of-product-2026-key-takeaways/) (Atlassian, 2026, as summarised by Mind the Product). The same coverage notes that [85 percent feel empowered to lead or influence strategy, yet 84 percent worry their products won't succeed in the market](https://www.mindtheproduct.com/inside-atlassians-state-of-product-2026-key-takeaways/).\n- **The customer habit is thin.** IdeaPlan's State of PM 2026 summary reports that [only 28 percent of PMs hold two or more customer conversations a week](https://www.ideaplan.io/blog/state-of-product-discovery-2026), and 11 percent rarely or never talk to customers (IdeaPlan, 2026). A plan built on conversations from last quarter is a plan built on last quarter.\n- **New bets arrive without customers attached.** UXCam's 2026 survey of 157 product professionals in 34 countries found that [70 percent of leaders grew their AI budgets while only 6 percent set AI objectives](https://uxcam.com/resources/ebooks/ai-in-product-management-2026/) (UXCam, 2026). UXCam sells product analytics, so read it as a signal about pressure, not a measure of customer demand. That is the point: AI bets enter roadmaps because someone senior asked, and nobody has checked whether customers did.\n\n![Bar chart of how often product managers talk to customers: 28 percent two or more times a week, 19 percent weekly, 24 percent two or three times a month, 18 percent monthly or less, 11 percent rarely or never](/visuals/planning-conversation-frequency-pm-survey/planning-conversation-frequency-pm-survey.png \"Only 28 percent of PMs talk to customers twice a week or more. · Source: IdeaPlan, State of PM 2026, 2026\")\n\nYou are measured on whether what ships moves adoption, retention and revenue. Planning is the one meeting where you can change what ships at almost no cost. Walk in without evidence and the loudest voice sets the quarter. The cost shows up two quarters later, as a feature in the bottom of the usage report and a retro that says \"we should have talked to customers first\".\n\n## What good looks like\n\nTreat planning prep as a three-week sprint with a fixed shape. Teresa Torres defines continuous discovery as [at a minimum, weekly touchpoints with customers, by the team building the product, where that team is conducting small research activities, in pursuit of a desired product outcome](https://www.mindtheproduct.com/getting-to-a-team-based-approach-to-continuous-discovery-by-teresa-torres). If you already do that, the sprint tops up the bets the weekly calls did not cover. If you don't, the sprint is how you start.\n\n**1. Grade every bet before you talk to anyone.** Put the draft bets in a list and give each one an evidence grade. We use a five-step ladder, compiled by Koji from common product discovery practice: each step is stronger than the one below it.\n\n![Framework with five evidence grades for roadmap bets, from assumed to behavioural](/visuals/roadmap-bet-evidence-ladder/roadmap-bet-evidence-ladder.png \"Grade each bet honestly before the sprint, and again after it.\")\n\nMost bets start at \"assumed\" or \"anecdotal\". That is normal. The sprint exists to move them up, or to show they can't move.\n\n**2. Cap it at three bets per study.** Each participant should hear no more than three problem areas. More than that and they answer about whichever one they find easiest to talk about. Five bets means two studies with a shared opening.\n\n**3. Interview three groups per bet, not one.** Requesters are the people who already asked for it. They are over-represented in your tickets and under-represented in your revenue. Add:\n\n- **Requesters** who asked for it (about a third of completes). They give the story behind the ask.\n- **The silent middle**: active customers in the same segment who never asked (about half). They tell you whether the problem is common.\n- **Lapsed or non-adopters**: churned, downgraded or never activated (about a sixth). They tell you what the bet would not have saved.\n\n**4. Decision rule for sample size.** Get at least 8 relevant completes per bet before you grade it above \"anecdotal\". Total 24 to 30 per study. Below 8 on a bet, it stays \"anecdotal\" however loud it is. Past 30 you mostly hear the same themes again, so stop.\n\n**5. Run it on a clock.** Counted back from planning day (T):\n\n- **T-21 (Monday):** list the bets, grade them, write the one decision each could change (\"build as scoped, shrink, swap, or wait\").\n- **T-20:** build the three groups from your analytics and CRM and send personalised links.\n- **T-19 to T-8:** field. Send one reminder on T-14.\n- **T-7:** read the report. Re-grade each bet.\n- **T-5:** write a one-page evidence brief per bet (grade, counts, two quotes, the decision).\n- **T-3:** pre-wire the executive most likely to override. Show them the brief, not the deck.\n- **T:** planning. Bets with grade 3 or higher get the top slots. Bets below 3 go in as labelled hunches, or not at all.\n- **T+30:** rerun a ten-person pulse on the bets that were picked, so the next planning cycle starts with fresh evidence.\n\n**6. Ask about last week, not next year.** \"Would you use this?\" produces yes. \"Tell me about the last time you hit this\" produces facts. Anchor on the most recent real event and ask what they did.\n\n### Common mistakes\n\n- Interviewing only the people who asked. They confirm the ask.\n- Describing the bet in the question. You will hear your own pitch back.\n- Counting a vivid quote as a pattern. Look for the same barrier in at least three people.\n- Grading your own bets the night before planning. Grade before the sprint and again after, and keep both grades visible.\n- Presenting findings as slides nobody can trace. Show the count and the supporting quotes.\n\n## Copy this interview guide\n\nPaste this into any tool. Replace [job] with the thing your customers are trying to get done, and [problem A], [problem B], [problem C] with the problems behind your three bets, written as a customer would say them, never as a feature.\n\n**Research brief (one line for Koji's \"What do you want to research?\" box):** \"Which of our three planned bets for next quarter solve a problem that active customers in [segment] have recently worked around, paid for or switched over, and which rest on an assumption?\"\n\n**Opening line:** \"Thanks for helping. We are planning next quarter and want to hear what is really getting in your way. There are no wrong answers, and you won't hurt anyone's feelings.\"\n\n1. **(Open)** \"Tell me about the last time [job] didn't go the way you wanted. What happened?\" Probe: \"What did you do next?\"\n2. **(Open)** \"What matters most for your team in the next three months?\" Probe: \"What could get in the way?\"\n3. **(Ranking)** \"Rank these by how much they cost you right now: [problem A], [problem B], [problem C], none of these.\" Probe: \"Tell me about the one you put first.\"\n4. **(Single choice)** \"How do you handle [problem A] today? I don't / a spreadsheet or workaround / another tool / someone does it for us / it's not a problem.\" Probe: \"Walk me through the last time.\"\n5. **(1 to 10 scale)** \"How much does [problem A] get in the way of your goals this quarter?\" Probe: \"What makes it that number and not two lower?\"\n6. **(Open)** \"What have you tried to fix it? What happened?\" Probe: \"Why did that stop?\"\n7. **(Yes/no)** \"Has this ever affected a decision to renew, switch or buy something?\" Probe: \"Tell me about that.\"\n8. **(Open)** \"Who else in your company feels this, and how do they cope?\" Probe: \"Who decides whether it gets fixed?\"\n9. **Close:** \"Is there anything I should have asked and didn't?\"\n\nRepeat questions 4 to 7 for each of the other problems, keeping it to three. The matching template in Koji's gallery is \"Continuous Discovery Interviews\" under Product Managers: it already has the open, scale and follow-up structure, and you swap in your bets.\n\n![Question bank with eight interview questions for a quarterly planning evidence sprint, grouped by past behaviour, priorities and consequence](/visuals/planning-evidence-sprint-question-bank/planning-evidence-sprint-question-bank.png \"Eight questions you can reuse for any three bets.\")\n\n## How to set it up in Koji, step by step\n\n1. **Start a study.** In the dashboard open the template gallery, filter by Product Managers and pick Continuous Discovery Interviews, or paste the one-line brief above into the \"What do you want to research?\" box.\n2. **Shape the brief.** Name the decision (\"which bets get the top slots next quarter\") and your hypotheses. Upload the draft roadmap summary and any bet descriptions as context files so the interviewer knows your product's vocabulary. Then set guardrails: list the topics the interviewer must not discuss, such as unreleased features, pricing and names of other customers. Customers should tell you about their problems, not hear your roadmap.\n3. **Write the questions.** Use the nine above. Set question 3 as ranking, question 4 as single choice, question 5 as a 1 to 10 scale, question 7 as yes/no and the rest as open with one or two follow-ups.\n4. **Choose voice or text.** Text interviews work on every plan and cost 1 credit each. Voice interviews need the Researcher plan or above and cost 3 credits each. Thirty text interviews are 30 credits. Run the preview yourself first: it is saved with its own analysis and does not count as a response.\n5. **Send personalised links by group.** Create a link for each person from your three groups so you know which group said what. Personalised links work on every plan. Importing a CSV from your CRM needs the Researcher plan or above. Set a response limit at your target so you stop at 30.\n6. **Read the report and publish the brief.** The report groups answers into themes with the number of participants behind each, and each finding opens to the quotes that support it. Share the published report link with the executive you pre-wired. Refreshing the report as late responses arrive costs 5 credits.\n\n![Process of six steps to run a quarterly planning evidence sprint in Koji, from template to published report](/visuals/planning-evidence-sprint-setup-steps/planning-evidence-sprint-setup-steps.png \"Six steps, from template to a report you can send.\")\n\n![Koji guardrails settings listing topics the AI interviewer should avoid, like prices and other customers](/email/shots/shot-guardrails.jpg \"Guardrails keep unreleased roadmap items out of the conversation.\")\n\n![Koji report summary with a side panel of supporting participant quotes behind one finding](/email/shots/shot-evidence-panel.jpg \"Every finding opens to the quotes behind it, so the room can check it.\")\n\nRead more: [Research-driven roadmap prioritization](/docs/research-driven-roadmap-prioritization), [the roadmap evidence audit](/docs/roadmap-evidence-audit), [personalised interview links](/docs/personalized-interview-links) and [importing participants via CSV](/docs/importing-participants-csv).\n\n## What you get\n\nOn T-7 you hold a report, not a pile of transcripts.\n\n- **A grade per bet.** Each bet moves up, or stays where it was, with the counts to show it.\n- **Quotes you can paste.** Each finding links to the people behind it, so the debate is about the decision.\n- **A ranking distribution.** If you ask question 3 the same way for every participant, you see how many put each problem first, split by group.\n- **A published report link** for design, engineering and the executives who will weigh in.\n\nUse it the next day: put the one-page brief for each bet at the top of the planning doc, with the grade and the decision you wrote down on T-21. When someone overrides the order, the override is now a visible choice against a visible grade, and you can write it down.\n\n## The same playbook elsewhere\n\nThe idea carries wherever a decision is about to be made on opinion.\n\n- **Agencies:** interviewing every team at a prospect before writing a proposal. See [new-client stakeholder discovery](/blog/new-client-stakeholder-discovery-agency-playbook).\n- **Business analysts:** checking AI-drafted requirements with the people who will use them. See [stakeholder validation interviews](/blog/ai-drafted-requirements-validation-business-analyst-playbook).\n- **Product, after the plan:** once a bet ships, the [launch-window playbook](/blog/launch-window-interviews-product-manager-playbook) tells you whether it worked. If the bet is an AI feature, see [AI feature adoption research](/blog/ai-feature-adoption-research-product-managers).\n- **Founders:** the same sprint before a fundraise, aimed at investor reference calls.\n\n## Why Koji for a planning evidence sprint\n\n![Table comparing Koji with analytics and tickets only, scheduled calls and surveys for roadmap planning evidence](/visuals/planning-evidence-sprint-koji-vs-alternatives/planning-evidence-sprint-koji-vs-alternatives.png \"What each approach can and cannot tell you before planning day.\")\n\nAnalytics and ticket tags are essential and already exist. Scheduled calls give the deepest conversation if you can book them. Surveys reach many people fast. The trouble is the combination you need in a three-week window: depth, reach into people who did not ask, and the same questions for everyone.\n\n1. **Follow-up on every answer.** Each open question can have one or two follow-ups, so \"it's a hassle\" becomes the step where they got stuck.\n2. **No calendar.** A personalised link lets each person answer in voice or text when it suits them, which matters when recruitment is the thing blocking your weekly conversations.\n3. **Mixed question types in one conversation.** Ranking and scale questions give you the distribution, open questions give you the story, and you get both from the same person.\n4. **The same guide for every participant.** You can compare requesters with the silent middle because both heard the same questions with the same probing.\n5. **Findings with counts and quotes.** The report shows how many people stand behind each theme and opens to the supporting quotes, so you can check a finding before you present it.\n\n## Questions you might have about AI interviews\n\n**Will customers tell an AI the truth about our roadmap?** They don't see your roadmap. You list the problems, never the solutions, and set guardrails so the interviewer never discloses unreleased work. Participants see the time estimate and privacy note on a welcome page before they start, and can choose voice or text.\n\n**Can it probe like a researcher would?** It follows your questions and the follow-up setting you chose, and reads the context files you uploaded. Preview the interview yourself and read the first five transcripts before the rest come in. Each interview also gets a quality rating, so you can set aside thin ones.\n\n**Where does the data go?** Interview data is stored in the EU, and Koji does not use customer interview data to train AI models. On Enterprise plans you can [bring your own key](/docs/bring-your-own-key).\n\n## When Koji may not be the right fit\n\nIf you have fewer than 30 active customers in the segment, call them. You will learn more from eight conversations than from any tool. If a bet turns on how people use a screen, you need usability testing with recordings. And if planning is next week, run a ten-person pulse on your top two bets rather than the full sprint.\n\n## Metrics to track\n\n- Share of roadmap bets at grade 3 or higher on planning day, versus last quarter.\n- Number of overrides of the evidence order, and whether they were logged.\n- Completes per bet, and response rate per group.\n- Days from sprint start to the first documented change in a bet's scope.\n- Share of shipped bets still used at day 30 (see the [launch-window playbook](/blog/launch-window-interviews-product-manager-playbook)).\n\n## FAQ\n\n**How far ahead of planning should I start?** Three weeks. Invite on T-20, field through T-8, read the report on T-7 and write briefs by T-5.\n\n**How many customers should I interview?** 24 to 30 per study, with at least 8 relevant completes for each bet you want to grade above \"anecdotal\".\n\n**Who should I include besides people who asked for the feature?** Active customers in the same segment who never asked, and lapsed or non-adopting accounts. Requesters alone confirm the ask.\n\n**What if leadership still overrides the evidence?** Then the override becomes visible. Log the bet, its grade and the decision. Over two quarters that record is the most persuasive evidence you will have.\n\n**Which plan do I need?** Text interviews, personalised links and report publishing work on every plan, and the Free plan includes 10 interviews to pilot. Voice interviews and CRM import need the Researcher plan or above.\n\n## Start from the template\n\nOpen Koji, choose the Continuous Discovery Interviews template, paste the brief and send your first ten personalised links this week. See the [Product Managers page](/for/product-managers) for more, or browse every playbook for your role in the [use-case library](/use-cases/product-managers).\n","category":"Use Cases","lastModified":"2026-10-28T07:50:10.722182+00:00","metaTitle":"Quarterly Planning Evidence Sprint for Product Managers","metaDescription":"A three-week sprint to grade every roadmap bet with customer evidence before planning day: who to interview, what to ask, and how to run it in Koji.","keywords":["quarterly planning","roadmap evidence","product discovery","customer interviews","roadmap prioritization","product manager playbook"],"aiSummary":"A product manager playbook for the three weeks before quarterly planning: grade every roadmap bet on a five-step evidence ladder, interview requesters, the silent middle and lapsed accounts, use a ready nine-question guide, and run it in Koji with personalised links and a report that shows counts and supporting quotes.","aiKeywords":["quarterly planning","roadmap prioritization","customer evidence","AI-moderated interviews","product discovery","product management"],"aiContentType":"guide","faqItems":[{"answer":"Three weeks. Invite on T-20, field through T-8, read the report on T-7 and write one-page evidence briefs by T-5.","question":"How far ahead of planning should I start?"},{"answer":"24 to 30 per study, with at least 8 relevant completes for each bet you want to grade above anecdotal.","question":"How many customers should I interview before planning?"},{"answer":"Active customers in the same segment who never asked, and lapsed or non-adopting accounts. Requesters alone only confirm the ask.","question":"Who should I include besides people who asked for the feature?"},{"answer":"The override becomes visible. Log the bet, its evidence grade and the decision, so the record builds over two quarters.","question":"What if leadership still overrides the evidence?"},{"answer":"Text interviews, personalised links and report publishing work on every plan, and the Free plan includes 10 interviews to pilot. Voice interviews and CRM import need the Researcher plan or above.","question":"Which Koji plan do I need?"}],"relatedTopics":["quarterly planning","roadmap evidence","product manager research"]}],"pagination":{"total":1,"returned":1,"offset":0}}