Alleviating the Infamous QA Bottleneck
Senior Director of QA and Data Products at Niche.com
Our organization has doubled in size in the last year, including the QA team I run. With a significant influx of people and changes in processes, we inevitably ran into hiccups. One of the challenges we experienced was that many tasks were piling up on the QA within our product teams, slowing down overall velocity. I had to figure out how to alleviate the fabled “QA bottleneck” by identifying the concrete challenges the QA team faced, and mapping out solutions for each.
To start, I gathered two QA managers who reported to me and whose teams were focused on manual testing. The three of us sat down for several hours and discussed the challenges their teams faced in-depth. “QA bottleneck” is a blanket term that can hide multiple smaller problems within it; we had to dive deeper and identify the underlying root causes.
First, we listed out all the different points in our current development life cycle that tended to cause delays or simply slow velocity, and then we ranked these periods of waiting in order of time lost. This revealed a range of opportunities for improvement. Some were matters of timing -- elements of the development life cycle that were serialized when they could be parallel, for instance, such as code review and test planning. Ultimately, though, we found that our QA team test plan review process -- where testers receive feedback from fellow testers on the completeness and clarity of their test plan -- was the area that caused the longest wait times and thus carried the highest cost.
Because most of our product teams only had one QA on them, almost every request for test plan review required resources from another team. Thus, the process had a double cost -- QA on one team had to wait to receive a review, while QA on another team had to pause their own work to perform the review. Test plan review added real value, catching gaps in coverage and presenting an organic opportunity to build context. Our challenge became clear: retain that value while reducing its cost.
Ultimately, the QA managers and I decided that we had to enable every individual product team to perform its own test plan reviews. We started by developing written guidelines, bringing more transparency to how QA conducted test plan review. I initiated the process, but since the managers below me would own the execution of any solution, I encouraged them to take more of a lead role. I wanted to step back and be more of a facilitator, asking questions and encouraging discussion that would help us define excellence.
The guidelines we came up with allowed any member of a product team -- whether they were a software developer, data analyst, or otherwise -- to give input on QA test plans. That reduced all wait times for test plan reviews significantly. We saw immediate improvements in terms of teams meeting sprint goals, estimation being more accurate in terms of both time and effort. We also saw ancillary benefits. QA’s process became more transparent, and developers were even able to identify potential bugs in their code just by reading QA’s plan, causing bug fixes to get pushed before testing even began.
We piloted this approach on one product team to start with. Once we had initial results, we didn’t have to push other teams to adopt it; they actively wanted to.
By rethinking how a test plan review could be performed, we turned a weakness in our process into a strength that allowed us to drive a discussion about quality beyond our functional team. That was much more in line with the broader mission of my department, which is to encourage all members of the organization to think more about quality. We believe that quality is not a QA team responsibility, but a company responsibility; it was great to be able to put that principle into practice while solving a real problem.
- Embrace being out of your comfort zone and keep looking for opportunities to improve. This can be challenging and stressful, but the best opportunities often dwell within uncertainty.
- By encouraging test plan review to be thought of as a product team activity, rather than a functional team activity, we encouraged a better dialogue about quality within product teams and helped to break down a silo that was hindering velocity.
- Being a rapidly growing company with distributed teams and relationships still forming, it can be challenging to push for large-scale changes. Starting small on one product team and achieving results that we could showcase to the rest of the organization made adoption easier. Having quick wins to share got us instant buy-in.
- As a leader at a director level, this was also a great opportunity to practice managing through my managers beneath me. In the past, when we were smaller in size, this could have been an initiative that I would personally spearhead. I didn’t think it would be the right approach now because I was not going to be involved in the execution. By working with and trusting my managers, we were successful beyond expectations.
Be notified about next articles from Michael Scotto
Senior Director of QA and Data Products at Niche.com
Connect and Learn with the Best Eng Leaders
We will send you a weekly newsletter with new mentors, circles, peer groups, content, webinars,bounties and free events.