Using Scoreboards to Create Data-Driven Roadmaps

Problem

Without data, product decisions are often made by HiPPO (highest paid person's opinion). Most cases this is the CEO. Sometimes the head of sales has a big customer that is asking for features xyz. These product decisions may occasionally prove to be right. But rarely. Creating a data collection pipeline measuring a set of core business/product metrics allows for a conversation to occur between the stakeholders. From there decisions around product roadmap can based on data and long-term strategy.

Actions taken

• Outline of the collection method:

1. Collect your data in a 1 week trailing window. I do Thursday-2-Thursday since I hold the meeting on Friday.
2. My collection consists of a python script that talks to our database and mixpanel. I also by hand pull data from Intercom and a few other sources.
3. This data is manually inserted into a Google-Sheet that is a single tab for the quarter with the 12 weeks listed. The number of metrics shouldn't exceed a length that the score-card requires scrolling. That is too many metrics! BE CONCISE ! Measure What Matters and keep it in a single non-scrolling panel.

• I track metrics early on and do that week to week, but displayed quarterly.
• In the backlog, I then have conversations when looking at the roadmap and ask if this is going to move any of the metrics we see on the scorecard. We want to have a hypothesis that new features will drive a set of metrics.

Lessons learned

• Scorecards are very powerful in keeping people onboard with objectives with the idea around ownership & accountability.
• You do not need a big fancy dashboard. Compiling data from a text file into a google sheet is organic and minimal work. The more low-fi you go, the easier it is to add things as time goes on.
• If your metrics are not going to move, then you are either not measuring the right thing, or it is not really that important to the roadmap. Be mindful that these metrics but don't set them in-stone. Or track things that don't matter.
• Things in your roadmap and how you prioritize will not be data-driven. The most important person in the room will drive the roadmap and to get that set up you will need to prioritize that with an engineer. Therefore, product and engineering need to have a good relationship.
• Putting text files into Github is helpful to keep a paper trail, but also fosters accountability and ownership.
• We successfully used a scorecard to measure and address the issue of customers canceling meetings.
• I'm now the CEO. I've adapted the Scorecard to manage non product/engineering. So there is a larger conversation here about scorecards for each team. Prod/Engineering should have their own scorecard. Sales should have their own card. And at my weekly-closeout for the entire team I have a general scorecard that we all go over. If I'm mentoring prod/engineers that one day want to be CEO, they need to learn how to manage other types of people.