Increasing Test Coverage
16 March, 2021
Head of R&D at Wix.com
Many years ago I was leading a team of 250 engineers and was tasked to figure out how much we should invest in our test coverage. The product was suffering from quality deficiencies, and I had no doubts that we had to increase our test coverage. While it was clear that the existing test coverage was not sufficient to make an impact on the quality, I was still unsure if we should engage all of 250 engineers to increase our test coverage.
First off, I conducted a thorough study that resulted in quantifiable data. The study mapped out our coverage across different components of the system segmented by different types of tests (unit test, integration test, etc.).Then I calculated a defect density score that showcased how many defects each component suffered which I normalized by the size of the software component. That provided a rough quality score to start with.
I plotted all the data on a graph with two axes -- a percentage of coverage on the x-axis and a defect score on the y-axis. Each point on the graph referred to a specific component. I was able to identify what was the coverage for any type of test (unit test, integration test, etc.) that was needed to create a substantial decrease in the defect rate.
We found that the integration test yielded the most results, and 70 percent scored in the integration test provided me a clear understanding of how much more we should invest in order to increase the quality. Also, since the experiment was segmented by components, we could make diligent decisions on the amount of effort we should allocate to achieve our goal and where the threshold at which allocating more resources wouldn’t yield as many results.
Following my strategy, we were able to identify where we should invest more. We refined the list of components and ranked them based on their criticality within the system. We tackled first the most critical components that were lacking the coverage of the integration test. We invested in bringing their coverage up the threshold which resulted in improved quality since the defect rate for those components was reduced significantly.
- As engineers, we often don’t rely on data as much as we should. If we relied more on data to optimize our execution, our results would be better. While we expect product managers and business people to be data-driven, we rarely have the same expectation for engineers.
- I find many software engineers to be biased toward their intuition, and it was not always easy to convince other engineers to adopt the data-driven approach.
Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader
Adir Nashawi, Senior Product Manager at Hibob, shares his insight and experience from rebuilding a product to handle many feature requests and offerings.
Senior Product Manager at Hibob
Pavel Safarik, Head of Product at ROI Hunter, shares his insights on how to deal with disagreements about prioritization when building a product.
Head of Product at ROI Hunter
Matias Pizarro, CTO and VP of Residents at ComunidadFeliz, recalls a time in his early career when he took a technology risk that had wide-ranging benefits to his product's user experience.
CTO and VP of Residents at ComunidadFeliz
Renaldi, Director of Engineering at Boku Inc., shares his guide for improving problem-plague processes into strategic initiatives.
Director of Engineering at Boku Inc
Shawn Sullivan, Co-founder & CTO at Phase Genomics, shares how his career has spanned from working at a tech giant to co-founding a startup in every stage of his growth.
Cofounder & CTO at Phase Genomics