Shaping Metric Improvements

Problem

It is common to struggle with the process of setting metrics for your team. You want to ensure the metrics that you set are useful, helpful, and correctly capture and reflect the success or failures of your team. Below are ideas on ways to implement metrics and how to hold your team accountable for them.

Actions taken

• Teams that are working on the same metric should be centralized in some way. It creates an incentive system and makes for better communication. Use your influence to affect the actual structure so that teams are working on a metric together.
• It is important to pick metrics ahead of time, publicly. Write out in an email defining the metrics that you want to test and their significance. This is to ensure transparency, and also that you don't make up a hypothesis to fit the data after the fact.
• A/B test components. One half of the test is a control and the other is your variant. The variant will be testing the lift so hold yourself to the metric lifts. Test your hypothesis and observe the wins and the losses. A failed test in not a failure of the team, it just means that you iterated faster or grew larger.
• Hold yourself accountable to metric lifts with respect to control, as opposed to absolute numbers. Aim your focus on the improvement of metrics rather than on the numbers.

Lessons learned

• You can't hold yourself to rigorous metrics if there's not a way to rigorously test.
• It's impossible to know the exact thing that needs to be changed. Set short-term metrics that you care about and long-term metrics that you want to have be impactful.
• Working with a great product manager who cares about metrics and knows the ins and outs can take a lot of stress off your shoulders. He/ She is also notable person to brainstorm ideas with about different and interesting tests to run.