Back to resources

Trailing New Products: What to Go for -- Quantity or Quality?

Product
Users

30 September, 2020

Caroline Parnell
Caroline Parnell

Director of Product Innovation at PhotoBox at PhotoBox

Caroline Parnell, previously managed product teams at O2 and Vodafone, recalls how she made a mistake by going for quantity rather than quality of product trialists, and how that prevented her from receiving the best feedback on her product.

Problem

A few years ago I was working on a technically complicated mobile voice application -- first-time use case, a lot of dependencies, early-stage technology, etc. We invested significantly in this product and were late with going to market. In addition, expectations on what we were going to deliver were exceedingly high.

We wanted to get as many internal staff onto the beta trial of our product as possible. They would be using our product, providing us feedback, evangelizing, and would get a chance to use it before we would launch it to our customers.

Actions taken

We promoted the product internally and drummed up a lot of interest. Around 1500 people signed up to use the application across different teams. We developed some basic support collateral and FAQs to address the most common questions our trialists would have. We were clear that it was an early-stage product so there would be some bugs.

When we rolled it out, we rolled it out quite quickly and encouraged everyone to download the application and provide us with their feedback. However, we were suddenly overwhelmed with questions by people who didn’t know how to use it, how to set it up, or confused by some of the early limitations of the beta product. We were inundated with support questions and had to hire one person full-time to address all of their concerns.

Over time we realised that although we had 1500 trialists, only about 3% were providing us with meaningful and useful feedback that would help us further improve the product. These people tended to be people that needed the product most and were full of useful regular feedback about how to make it better. The remaining 97% of trialists took up a lot of effort to manage and yet did little to provide useful feedback. We soon realized that we chose the wrong approach and that we shouldn’t have gone to that many people while there were still a lot of bugs in the product.

After the launch, we wanted to set up a new beta trail community to test new features we were developing. This time we streamlined our focus. We wanted early adopters who were comfortable if things were not working well, but who were also vocal and good at providing constructive criticism. We sent out a survey asking people why they wanted to be trialists and what things we should be focusing on with the application. Based on their responses, we selected 30 users who became our product trialists. They gave useful and regular feedback and were much easier to manage as they were largely self-sufficient. Because they were the right target they understood the limitation of the product and were good at providing constructive feedback.

Lessons learned

Be very clear what you want to achieve for your product by running a trail and consequently, be clear about the type of trialist you want to recruit. If your product is fairly complex and early on -- as was our case -- you should be targeting early adopters who are comfortable with technology and giving constructive feedback.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

How to Build and Manage the First Product Roadmap for a Startup

19 January

Angel Jamie, Chief Product Officer at Yayzy, shares how he created the first product roadmap at a startup, and a simple process to keep it alive.

Alignment
Product
Meetings
Roadmap
Prioritization
Angel Jaime

Angel Jaime

CPO at yayzy

How to Spark Sales-Driven Change

19 January

Nani Nitinavakorn, Sr Product Owner at Revolut, recalls her experience initiating a structural change to optimize her entire company.

Customers
Innovation / Experiment
Leadership
Meetings
Impact
Users
Nani Nitinavakorn

Nani Nitinavakorn

Sr Product Owner at Revolut

Completing a Successful Reorganization within a Well-Established Company

19 January

Jason De Oliveira, CTO for more than 10 years, shares his experience completing a reorganization, implementing agile, and collaborating with multiple teams.

Alignment
Product
Leadership
Reorganization
Agile / Scrum
Jason De Oliveira

Jason De Oliveira

CTO at Kolquare

Aligning a Company Vision with Personal Values

19 January

Anjul Bahuguna, Product Manager at Facebook, gives his expertise on the process of transitioning a company vision and the challenges that come with it.

Alignment
Customers
Mission / Vision / Charter
Product
Motivation
Users
Toxic Atmospheres
Anjul Bahuguna

Anjul Bahuguna

Product Manager at Facebook

Effective Cross-Functional Collaboration Between Technical and Product Leaders

4 January

Manu Gurudatha, Senior Director, Software Engineering at PagerDuty, shares how he established a trust-based relationship between product and engineering, which led to many successful product deliveries.

Product
Leadership
Collaboration
Ownership
Coaching / Training / Mentorship
Cross-Functional Collaboration
Manu Gurudatha

Manu Gurudatha

Senior Director, Software Engineering at PagerDuty

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.