Increasing Test Coverage
16 March, 2021
Head of R&D at Wix.com
Many years ago I was leading a team of 250 engineers and was tasked to figure out how much we should invest in our test coverage. The product was suffering from quality deficiencies, and I had no doubts that we had to increase our test coverage. While it was clear that the existing test coverage was not sufficient to make an impact on the quality, I was still unsure if we should engage all of 250 engineers to increase our test coverage.
First off, I conducted a thorough study that resulted in quantifiable data. The study mapped out our coverage across different components of the system segmented by different types of tests (unit test, integration test, etc.).Then I calculated a defect density score that showcased how many defects each component suffered which I normalized by the size of the software component. That provided a rough quality score to start with.
I plotted all the data on a graph with two axes -- a percentage of coverage on the x-axis and a defect score on the y-axis. Each point on the graph referred to a specific component. I was able to identify what was the coverage for any type of test (unit test, integration test, etc.) that was needed to create a substantial decrease in the defect rate.
We found that the integration test yielded the most results, and 70 percent scored in the integration test provided me a clear understanding of how much more we should invest in order to increase the quality. Also, since the experiment was segmented by components, we could make diligent decisions on the amount of effort we should allocate to achieve our goal and where the threshold at which allocating more resources wouldn’t yield as many results.
Following my strategy, we were able to identify where we should invest more. We refined the list of components and ranked them based on their criticality within the system. We tackled first the most critical components that were lacking the coverage of the integration test. We invested in bringing their coverage up the threshold which resulted in improved quality since the defect rate for those components was reduced significantly.
- As engineers, we often don’t rely on data as much as we should. If we relied more on data to optimize our execution, our results would be better. While we expect product managers and business people to be data-driven, we rarely have the same expectation for engineers.
- I find many software engineers to be biased toward their intuition, and it was not always easy to convince other engineers to adopt the data-driven approach.
Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader
Consideration for starting a multi year software infrastructure ( V2 ) project that involves hundreds of globally distributed engineers.
VP Software Engineering at human
Why DevSecOps matter and what's really in it for you, the team and the organisation?
Head of Engineering at Xero
Teams have tremendous impact on the products on they build. T.E.A.M definition - Together Everybody Achieves More is true. A collaborative and empowered team builds great product versus the good ones.
Senior Software Engineering Manager at Anaplan
A proposal for how to create an org structure that will deliver software systems that you want, not ones you get stuck with.
Principal / Founder at id8 inc
Adir Nashawi, Senior Product Manager at Hibob, shares his insight and experience from rebuilding a product to handle many feature requests and offerings.
Senior Product Manager at Hibob