Trailing New Products: What to Go for -- Quantity or Quality?
30 September, 2020
A few years ago I was working on a technically complicated mobile voice application -- first-time use case, a lot of dependencies, early-stage technology, etc. We invested significantly in this product and were late with going to market. In addition, expectations on what we were going to deliver were exceedingly high.
We wanted to get as many internal staff onto the beta trial of our product as possible. They would be using our product, providing us feedback, evangelizing, and would get a chance to use it before we would launch it to our customers.
We promoted the product internally and drummed up a lot of interest. Around 1500 people signed up to use the application across different teams. We developed some basic support collateral and FAQs to address the most common questions our trialists would have. We were clear that it was an early-stage product so there would be some bugs.
When we rolled it out, we rolled it out quite quickly and encouraged everyone to download the application and provide us with their feedback. However, we were suddenly overwhelmed with questions by people who didn’t know how to use it, how to set it up, or confused by some of the early limitations of the beta product. We were inundated with support questions and had to hire one person full-time to address all of their concerns.
Over time we realised that although we had 1500 trialists, only about 3% were providing us with meaningful and useful feedback that would help us further improve the product. These people tended to be people that needed the product most and were full of useful regular feedback about how to make it better. The remaining 97% of trialists took up a lot of effort to manage and yet did little to provide useful feedback. We soon realized that we chose the wrong approach and that we shouldn’t have gone to that many people while there were still a lot of bugs in the product.
After the launch, we wanted to set up a new beta trail community to test new features we were developing. This time we streamlined our focus. We wanted early adopters who were comfortable if things were not working well, but who were also vocal and good at providing constructive criticism. We sent out a survey asking people why they wanted to be trialists and what things we should be focusing on with the application. Based on their responses, we selected 30 users who became our product trialists. They gave useful and regular feedback and were much easier to manage as they were largely self-sufficient. Because they were the right target they understood the limitation of the product and were good at providing constructive feedback.
Be very clear what you want to achieve for your product by running a trail and consequently, be clear about the type of trialist you want to recruit. If your product is fairly complex and early on -- as was our case -- you should be targeting early adopters who are comfortable with technology and giving constructive feedback.
Deepak Paramanand, Product Lead at Hitachi, shares how he built three different AI projects that all had one thing in common -- the ability to create or input data.
Product Lead at Hitachi
Deepak Paramanand, Product Lead at Hitachi, highlights the key differences between enterprise product management and the one that builds products for consumers and thus helps aspiring product managers choose the right career for them.
Product Lead at Hitachi
Deepak Paramanand, Product Lead at Hitachi, explains how his early efforts in stand-up comedy and experience of interaction between a performer and audience helped him better understand his colleagues and customers.
Product Lead at Hitachi
Prabha Matta, Senior Product Manager at SquareTrade, talks about her personal experience of looking for a PM job during the Covid-19 pandemics and how the changed circumstances affected her job search and interviewing process.
Senior Product Manager at Square Trade
Caroline Parnell, previously managed product teams at O2 and Vodafone, recalls how she made a mistake by going for quantity rather than quality of product trialists, and how that prevented her from receiving the best feedback on her product.
Most recently Head of New Product Innovation at Previously O2 and Vodafone
You're a great engineer.
Become a great engineering leader.
Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.