Back to resources

Increasing Test Coverage

Dev Processes

16 March, 2021

Guy Baron

Guy Baron

Head of R&D at Wix.com

Guy Baron, Head of R&D at Wix, explains how he increased test coverage by developing and implementing a data-driven test strategy.

Problem

Many years ago I was leading a team of 250 engineers and was tasked to figure out how much we should invest in our test coverage. The product was suffering from quality deficiencies, and I had no doubts that we had to increase our test coverage. While it was clear that the existing test coverage was not sufficient to make an impact on the quality, I was still unsure if we should engage all of 250 engineers to increase our test coverage.

Actions taken

First off, I conducted a thorough study that resulted in quantifiable data. The study mapped out our coverage across different components of the system segmented by different types of tests (unit test, integration test, etc.).Then I calculated a defect density score that showcased how many defects each component suffered which I normalized by the size of the software component. That provided a rough quality score to start with.

I plotted all the data on a graph with two axes -- a percentage of coverage on the x-axis and a defect score on the y-axis. Each point on the graph referred to a specific component. I was able to identify what was the coverage for any type of test (unit test, integration test, etc.) that was needed to create a substantial decrease in the defect rate.

We found that the integration test yielded the most results, and 70 percent scored in the integration test provided me a clear understanding of how much more we should invest in order to increase the quality. Also, since the experiment was segmented by components, we could make diligent decisions on the amount of effort we should allocate to achieve our goal and where the threshold at which allocating more resources wouldn’t yield as many results.

Following my strategy, we were able to identify where we should invest more. We refined the list of components and ranked them based on their criticality within the system. We tackled first the most critical components that were lacking the coverage of the integration test. We invested in bringing their coverage up the threshold which resulted in improved quality since the defect rate for those components was reduced significantly.

Lessons learned

  • As engineers, we often don’t rely on data as much as we should. If we relied more on data to optimize our execution, our results would be better. While we expect product managers and business people to be data-driven, we rarely have the same expectation for engineers.
  • I find many software engineers to be biased toward their intuition, and it was not always easy to convince other engineers to adopt the data-driven approach.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

How to Successfully Rebuild Your Product

6 June

Adir Nashawi, Senior Product Manager at Hibob, shares his insight and experience from rebuilding a product to handle many feature requests and offerings.

Customers
Product
Dev Processes
Users
Prioritization
Adir Nashawi

Adir Nashawi

Senior Product Manager at Hibob

Navigating Disagreements When It Comes to Priorities

9 May

Pavel Safarik, Head of Product at ROI Hunter, shares his insights on how to deal with disagreements about prioritization when building a product.

Innovation / Experiment
Product Team
Product
Dev Processes
Conflict Solving
Internal Communication
Collaboration
Convincing
Strategy
Prioritization
Pavel Safarik

Pavel Safarik

Head of Product at ROI Hunter

Why You Should Take Technology Risks in Product Development

25 April

Matias Pizarro, CTO and VP of Residents at ComunidadFeliz, recalls a time in his early career when he took a technology risk that had wide-ranging benefits to his product's user experience.

Innovation / Experiment
Product
Scaling Team
Dev Processes
Matias Pizarro

Matias Pizarro

CTO and VP of Residents at ComunidadFeliz

Company Growth: The Good, the Bad and the Ugly

18 March

Renaldi, Director of Engineering at Boku Inc., shares his guide for improving problem-plague processes into strategic initiatives.

Dev Processes
Feedback
Strategy
Team Processes
Renaldi Renaldi

Renaldi Renaldi

Director of Engineering at Boku Inc

The Evolution of a Startup Co-Founder

15 March

Shawn Sullivan, Co-founder & CTO at Phase Genomics, shares how his career has spanned from working at a tech giant to co-founding a startup in every stage of his growth.

Changing A Company
Dev Processes
Company Culture
Impact
Convincing
Changing Company
Career Path
Shawn Sullivan

Shawn Sullivan

Cofounder & CTO at Phase Genomics