Improving Product Discovery

Problem

Our teams are somewhat independent and take part in the delivery as well as product discovery. They are essentially responsible for the final value of the team. Product management had solicited faster prototyping and experimentation, thus urging developers to make changes in production more quickly than they were used to. 

Actions taken

Knowing that the code would not be perfect by doing a prototype or experiment, we began by changing our peer review, test and release system to make it more flexible, faster and automated without sacrificing quality. Whereas we usually released every one to two weeks depending on the module, we gradually transitioned to daily releases. Smaller chunks of code were easier and faster to review, test and release. 

We implemented feature toggles to hide features that were not yet ready through configuration on the production. We used the external tool at the beginning and then started to implement the internal tooling as it became more relevant for us. Our tool also allowed us to do canary releases and A/B testing, which targeted various groups of customers based on country, plan or account.  

A subsequent effect of the SaaS application’s feature toggles was the overload of our helpline. We were running about 40 toggles at the same time and they were confused which toggles were on/off to the specific customer they were speaking with. We had to implement toggle overview into our CRM system in order to solve this problem.

In order to have qualitative data about prototypes, we started a user research program in cooperation with our customer success team. In other words: product teams can do demos with real customers on a daily basis and gain valuable feedback instantly.

The final issue we needed to solve was to have online data. We are now using a single tool for online data analytics and we also put all the data into a data warehouse (single source of truth) in order to see results from experiments and trends over time. 

Lessons learned

• Improving product discovery was a compound of several efforts. Each of its counterparts holds significant importance to the final outcome. 
• You need to be able to do some qualitative and quantitative analytics with the customers. 
• When working with prototypes and experiments that have imperfect codes, you want to work with smaller chunks of code and check them more frequently. Releasing the main modules every day is advisable. 
• Dealing with 40 different feature toggles in the production is a lot and was not a challenge I had expected from the beginning. It created several miscommunications that we had to solve on the fly.