Making the Most of User Data in Early-Stage Products
4 June, 2021
One could track many things in new or early-stage products, but narrowing it down to the right set of user data could separate success from failure. As expected, it will not always be an easy decision. For example, an early-stage product could have too thin and too few data, and overreliance on that kind of data can lead to erroneous conclusions. However, a seasoned product leader will know when to rely on user data and when to back it up with qualitative insights.
Regardless of the specifics of any given situation, two main challenges when tracking data in early-stage products are:
- Knowing what to measure and how to measure it;
- Being careful not to be blindsided by data. Data can tell any story depending on how it is interpreted. However, there are a number of techniques one can use in conjunction with data analysis to more comprehensively understand a user problem.
Before setting up any metrics, a product leader should have a profound understanding of how a product would be used. They should list down all the instances of expected user behavior and then set the metrics accordingly. At the same time, they should complement data with qualitative metrics based on the recordings or interviewing users. Whenever possible, I would rely on unmediated interaction with users to verify the hypothesis. Then, I would closely examine how those different insights would relate to the data acquired from the metrics we set.
For example, we recently launched a product and started to collect data on the industry of companies signing up. The idea was to better understand users who were signing up and thus better define our next set of features and user acquisition strategy. Some eight weeks and 2000 signups later, we took a look at the data analysis and noticed two verticals standing out -- education technology and medical healthcare providers. But here was the catch: if we would solely rely on that data, we would erroneously conclude that our product was being used mainly by education companies and medical providers and that we should double-down across these verticals. And that was not the case. We had other insights on what was happening in the market in general; the second wave of Covid-19 was plaguing India, and these two verticals were the only ones doing well. These were the only industries able to afford to invest in software, unlike everybody else, so our data was showing which verticals were doing well, not who was using our product.
Furthermore, one needs to understand the full context of data. For example, we build marketing software that helps people run campaigns on Facebook. They would create templates that are then submitted to Facebook for approval. Our data showcased that people were applying for 30 or 40 templates within the first week, which felt like they wanted to create a lot of templates. But when we dug deeper to understand better user behavior, we learned that 90 percent of the templates our users created within the first week were rejected. Because of the low acceptance rate, they kept creating template after template. So the real problem was not to optimize for how many templates one could apply and view on the dashboard but to help our users get through the approval process more easily.
When we did thorough research using various techniques, we learned more about our users’ behavior, their motivation, and the outcome of their actions. If we acted without learning about the entire context, the bare data would lead us astray. Therefore, we started to invest in helping them be successful in the first few attempts, and we managed to drastically increase their acceptance rate.
- Think of data as one of the tools that could help our overall understanding of users. Unlike interviews and other qualitative techniques, data can include a large group of users while at the same time navigating biases common for individual respondents. But data can become a double-edged sword if not understood within its own constraints and set up correctly.
- Data is all about reading and interpretation. It has to be coupled with a holistic understanding of a product and its users. You should never track only one metric, no matter how important it is, but combine it with other metrics that can enable you to understand the problem in its entirety. Oftentimes, people get obsessed with tracking one data point, leading them into a ditch from where their perspective will be narrowed down to that ditch perspective.
- Let the data speak for itself. Suspend your interpretations and approach data with a fresh mind. Get a good understanding of numbers before putting those data in conjunction with other insights and make them unfold into a story. However, many people let their presupposed narratives precede the reading of data and have them influence the interpretation. That particularly applies to situations when you measure something for the first time. Your first-time analysis should be as holistic as possible.
Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader
Jason Mawdsley, Director System Software at NVIDIA, explains how to determine the maturity of AI features and why validation and verification are the most important part of the process.
Director, System Software at nVidia
Heiko Reintsch, Head of Product at GetYourGuide, begs the question: should his company be focusing on improving what it produces currently, or should it be looking to expand its operations?
Head of Product at GetYourGuide
Apurva Mudgal, Director of Product and Partnerships at Haptik, explains what and how to measure in early-stage products, emphasizing how selective and/or biased interpretation of user data can lead to inaccurate conclusions.
Head of Consumer Product at Simpl
Sailendra Kumar, VP Product Management & Design at IndeedFlex, describes his effort to improve a product that had some inherent limitations.
Vice President - Product Management & Design at IndeedFlex ( Indeed.com)
Rachel Wasko, Growth Product Manager at Lyft, swears by the pre-mortem process when developing a new product or service for her stakeholders.
Product Manager at Lyft
You're a great engineer.
Become a great engineering leader.
Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.