How to build a reference class for product ideas

A five-step protocol for working out your own team's hit rate on product ideas, from choosing the decision point to scoring each idea as improved, flat, hurt or unknown. The rule that matters is to define the class by what was known when the bet was made, so cancelled ideas count too. It takes about an afternoon and turns the published one-in-three base rate into a number for your own product.

Where this appears

• The Base Rate for a Product Bet: What Experiment Portfolios Say About How Often a Feature Works

What's in this process

1. Choose the decision point: Usually the moment an idea entered the roadmap, since that is the decision the base rate has to inform.
2. List every candidate that reached it: The last four to six quarters, cancelled ideas included. Twenty to forty items is enough.
3. Record each target metric: As stated at the time. Mark the rest no target: the size of that group is a finding in itself.
4. Score each outcome: Improved, flat or hurt, from the measurement you have. Unknown where nothing was measured, never a guess.
5. Compute the three fractions: Then split them by category, such as new surface area, flow optimisation, pricing and onboarding.