We've just launched plato for individuals

🔥

login


Google Sign inLinkedIn Sign in

Don't have an account? 

Trailing New Products: What to Go for -- Quantity or Quality?

Users Feedback
Product

30 September, 2020

Caroline Parnell, previously managed product teams at O2 and Vodafone, recalls how she made a mistake by going for quantity rather than quality of product trialists, and how that prevented her from receiving the best feedback on her product.

Problem

A few years ago I was working on a technically complicated mobile voice application -- first-time use case, a lot of dependencies, early-stage technology, etc. We invested significantly in this product and were late with going to market. In addition, expectations on what we were going to deliver were exceedingly high.
 

We wanted to get as many internal staff onto the beta trial of our product as possible. They would be using our product, providing us feedback, evangelizing, and would get a chance to use it before we would launch it to our customers.
 

Actions taken

We promoted the product internally and drummed up a lot of interest. Around 1500 people signed up to use the application across different teams. We developed some basic support collateral and FAQs to address the most common questions our trialists would have. We were clear that it was an early-stage product so there would be some bugs.
 

When we rolled it out, we rolled it out quite quickly and encouraged everyone to download the application and provide us with their feedback. However, we were suddenly overwhelmed with questions by people who didn’t know how to use it, how to set it up, or confused by some of the early limitations of the beta product. We were inundated with support questions and had to hire one person full-time to address all of their concerns.
 

Over time we realised that although we had 1500 trialists, only about 3% were providing us with meaningful and useful feedback that would help us further improve the product. These people tended to be people that needed the product most and were full of useful regular feedback about how to make it better. The remaining 97% of trialists took up a lot of effort to manage and yet did little to provide useful feedback. We soon realized that we chose the wrong approach and that we shouldn’t have gone to that many people while there were still a lot of bugs in the product.
 

After the launch, we wanted to set up a new beta trail community to test new features we were developing. This time we streamlined our focus. We wanted early adopters who were comfortable if things were not working well, but who were also vocal and good at providing constructive criticism. We sent out a survey asking people why they wanted to be trialists and what things we should be focusing on with the application. Based on their responses, we selected 30 users who became our product trialists. They gave useful and regular feedback and were much easier to manage as they were largely self-sufficient. Because they were the right target they understood the limitation of the product and were good at providing constructive feedback.
 

Lessons learned

Be very clear what you want to achieve for your product by running a trail and consequently, be clear about the type of trialist you want to recruit. If your product is fairly complex and early on -- as was our case -- you should be targeting early adopters who are comfortable with technology and giving constructive feedback.


Related stories

AI Products Are Data Products
19 October

Deepak Paramanand, Product Lead at Hitachi, shares how he built three different AI projects that all had one thing in common -- the ability to create or input data.

Product
Impact
Deepak Paramanand

Deepak Paramanand

Product Lead at Hitachi

How Different Is Enterprise Product Management
19 October

Deepak Paramanand, Product Lead at Hitachi, highlights the key differences between enterprise product management and the one that builds products for consumers and thus helps aspiring product managers choose the right career for them.

Product
Deepak Paramanand

Deepak Paramanand

Product Lead at Hitachi

How Stand-Up Comedy Helped Me Better Understand People
19 October

Deepak Paramanand, Product Lead at Hitachi, explains how his early efforts in stand-up comedy and experience of interaction between a performer and audience helped him better understand his colleagues and customers.

Internal Communication
Personal growth
Users Feedback
Deepak Paramanand

Deepak Paramanand

Product Lead at Hitachi

Looking for a PM Job During the Covid-19 Pandemics
12 October

Prabha Matta, Senior Product Manager at SquareTrade, talks about her personal experience of looking for a PM job during the Covid-19 pandemics and how the changed circumstances affected her job search and interviewing process.

Hiring
Product
Managing Expectations
Prabha Matta

Prabha Matta

Senior Product Manager at Square Trade

Trailing New Products: What to Go for -- Quantity or Quality?
30 September

Caroline Parnell, previously managed product teams at O2 and Vodafone, recalls how she made a mistake by going for quantity rather than quality of product trialists, and how that prevented her from receiving the best feedback on her product.

Users Feedback
Product
Caroline Parnell

Caroline Parnell

Most recently Head of New Product Innovation at Previously O2 and Vodafone

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.