Back to resources

Trailing New Products: What to Go for -- Quantity or Quality?

Product
Users

30 September, 2020

Caroline Parnell
Caroline Parnell

Director of Product Innovation at PhotoBox at PhotoBox

Caroline Parnell, previously managed product teams at O2 and Vodafone, recalls how she made a mistake by going for quantity rather than quality of product trialists, and how that prevented her from receiving the best feedback on her product.

Problem

A few years ago I was working on a technically complicated mobile voice application -- first-time use case, a lot of dependencies, early-stage technology, etc. We invested significantly in this product and were late with going to market. In addition, expectations on what we were going to deliver were exceedingly high.

We wanted to get as many internal staff onto the beta trial of our product as possible. They would be using our product, providing us feedback, evangelizing, and would get a chance to use it before we would launch it to our customers.

Actions taken

We promoted the product internally and drummed up a lot of interest. Around 1500 people signed up to use the application across different teams. We developed some basic support collateral and FAQs to address the most common questions our trialists would have. We were clear that it was an early-stage product so there would be some bugs.

When we rolled it out, we rolled it out quite quickly and encouraged everyone to download the application and provide us with their feedback. However, we were suddenly overwhelmed with questions by people who didn’t know how to use it, how to set it up, or confused by some of the early limitations of the beta product. We were inundated with support questions and had to hire one person full-time to address all of their concerns.

Over time we realised that although we had 1500 trialists, only about 3% were providing us with meaningful and useful feedback that would help us further improve the product. These people tended to be people that needed the product most and were full of useful regular feedback about how to make it better. The remaining 97% of trialists took up a lot of effort to manage and yet did little to provide useful feedback. We soon realized that we chose the wrong approach and that we shouldn’t have gone to that many people while there were still a lot of bugs in the product.

After the launch, we wanted to set up a new beta trail community to test new features we were developing. This time we streamlined our focus. We wanted early adopters who were comfortable if things were not working well, but who were also vocal and good at providing constructive criticism. We sent out a survey asking people why they wanted to be trialists and what things we should be focusing on with the application. Based on their responses, we selected 30 users who became our product trialists. They gave useful and regular feedback and were much easier to manage as they were largely self-sufficient. Because they were the right target they understood the limitation of the product and were good at providing constructive feedback.

Lessons learned

Be very clear what you want to achieve for your product by running a trail and consequently, be clear about the type of trialist you want to recruit. If your product is fairly complex and early on -- as was our case -- you should be targeting early adopters who are comfortable with technology and giving constructive feedback.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

Scaling a Team in Two Parts: The Product and Manager

2 August

Viswa Mani Kiran Peddinti, Sr Engineering Manager at Instacart, walks through his experience scaling a team, product and his skills as a leader.

Managing Expectations
Product
Scaling Team
Leadership
Meetings
Viswa Mani Kiran Peddinti

Viswa Mani Kiran Peddinti

Sr Engineering Manager at Instacart

How Product Management Chose Me

23 June

My accidental journey into product management

Product
Personal Growth
New PM
Career Path
Michael Castro

Michael Castro

Sr. Manager, Product Management at Capital One

How Product Marketing Can (and Should) Help Product Development

20 June

Pavel Safarik, Head of Product at ROI Hunter, discusses the frequently overlooked role of product marketing in getting high user adoption rates for your product.

Goal Setting
Product Team
Product
Different Skillsets
Cross-Functional Collaboration
Pavel Safarik

Pavel Safarik

Head of Product at ROI Hunter

How to Successfully Rebuild Your Product

6 June

Adir Nashawi, Senior Product Manager at Hibob, shares his insight and experience from rebuilding a product to handle many feature requests and offerings.

Customers
Product
Dev Processes
Users
Prioritization
Adir Nashawi

Adir Nashawi

Senior Product Manager at Hibob

How to Empower Teams to Build Out a Product Portfolio During Company Growth

6 June

Ivo Minjauw, Global Product Director at OTA Insight, discusses the importance of structuring your teams when undergoing company growth.

Alignment
Goal Setting
Product
Ownership
Performance
Ivo Minjauw

Ivo Minjauw

Global Product Director at OTA Insight