Trailing New Products: What to Go for -- Quantity or Quality?

Caroline Parnell

Director of Product Innovation at PhotoBox at PhotoBox



A few years ago I was working on a technically complicated mobile voice application -- first-time use case, a lot of dependencies, early-stage technology, etc. We invested significantly in this product and were late with going to market. In addition, expectations on what we were going to deliver were exceedingly high.

We wanted to get as many internal staff onto the beta trial of our product as possible. They would be using our product, providing us feedback, evangelizing, and would get a chance to use it before we would launch it to our customers.

Actions taken

We promoted the product internally and drummed up a lot of interest. Around 1500 people signed up to use the application across different teams. We developed some basic support collateral and FAQs to address the most common questions our trialists would have. We were clear that it was an early-stage product so there would be some bugs.

When we rolled it out, we rolled it out quite quickly and encouraged everyone to download the application and provide us with their feedback. However, we were suddenly overwhelmed with questions by people who didn’t know how to use it, how to set it up, or confused by some of the early limitations of the beta product. We were inundated with support questions and had to hire one person full-time to address all of their concerns.

Over time we realised that although we had 1500 trialists, only about 3% were providing us with meaningful and useful feedback that would help us further improve the product. These people tended to be people that needed the product most and were full of useful regular feedback about how to make it better. The remaining 97% of trialists took up a lot of effort to manage and yet did little to provide useful feedback. We soon realized that we chose the wrong approach and that we shouldn’t have gone to that many people while there were still a lot of bugs in the product.

After the launch, we wanted to set up a new beta trail community to test new features we were developing. This time we streamlined our focus. We wanted early adopters who were comfortable if things were not working well, but who were also vocal and good at providing constructive criticism. We sent out a survey asking people why they wanted to be trialists and what things we should be focusing on with the application. Based on their responses, we selected 30 users who became our product trialists. They gave useful and regular feedback and were much easier to manage as they were largely self-sufficient. Because they were the right target they understood the limitation of the product and were good at providing constructive feedback.

Lessons learned

Be very clear what you want to achieve for your product by running a trail and consequently, be clear about the type of trialist you want to recruit. If your product is fairly complex and early on -- as was our case -- you should be targeting early adopters who are comfortable with technology and giving constructive feedback.

Be notified about next articles from Caroline Parnell

Caroline Parnell

Director of Product Innovation at PhotoBox at PhotoBox

Feedback Techniques

Connect and Learn with the Best Eng Leaders

We will send you a weekly newsletter with new mentors, circles, peer groups, content, webinars,bounties and free events.


HomeCircles1-on-1 MentorshipBounties

© 2024 Plato. All rights reserved

LoginSign up