How I Improved Test Automation

Problem

Each company has its own unique set of problems when it comes to testing and development and how the two work together. One of the challenges I faced in my previous role was to move faster, and for that to happen, we had to release faster. Since I was an EM on the quality team, I was tasked to improve test automation.

Actions taken

For starters, I reached out to a program manager handling releases to learn firsthand how a release cycle worked and what were the challenges they were facing. As it turned out, we were releasing only monthly, which was far less frequent than what we were hoping to do. One of the main problems that slowed us down was our dependence on an offshore third-party testing company. In general, it took them a couple of days to do testing, and then a time zone difference would add a half or even a full day on top of it. Once they would finish testing, they would report bugs, and a person on the quality team would triage those bugs and then get back to them with feedback. The whole process was rather stretched out and inefficient.

"We were releasing only monthly, which was far less frequent than what we were hoping to do."

I could tell this is where most of the time was lost, so I tried to dissect each of those steps and identify how each of those was time-consuming. That helped me better understand how my team could help because we were focused on building tools and test frameworks. If my team could build out the test framework and then add tests, we wouldn’t have to rely on the third-party testing team, which would speed up the process. In addition, we would also save some $300,000 annually, which is how much we were paying the offshore testing team. I put all numbers on paper and shared this with the program managers responsible for releases.

Then I had to arrange for a meeting with engineering stakeholders, mainly directors, to discuss a more technical side of the problem. Adding automation tests alone wouldn’t speed things up. We would need to run these tests against PRs and make sure that developers would look into test failures. That had to happen in parallel with the work my team was doing on building frameworks. My task was to work with directors and help them with moving forward with the technical side.

We came up with a strategy: I would work with core service owners to have test coverage for each of the teams within different verticals. More specifically, we would work with a designated person within each team who would serve as a point of contact. That person would look into test failures and be in close touch with my team informing us of issues emerging when running tests against PRs or pull requests. We did a pilot run with some of the core services team, and that went really well.

Finally, I was constantly reflecting on whether we were working in alignment. A lot of times, when people try to solve similar problems, they would run into a rabbit hole and focus on a single issue that bears little relevance to the bigger picture. Therefore, I reflected throughout the process on how we wanted to proceed forward and to the desired destination of moving from the monthly to the weekly release cycle. I also ensured that all stakeholders were on the same page throughout the process, and we were constantly interacting with them, updating them on all our progress.

It took us around 6 months to move from a monthly to a weekly release cycle. We worked with several different core services, and we wanted to start the pilot efforts small and then expand. We started with five core services that were critical for the company, and then we broadened our initiative to include the remaining services. That approach took more time but was far safer.

Lessons learned

• When I proposed the initial changes, one of my peers told me that I would need more formal authority to implement them. I disagreed: my approach was to influence without authority. Even as an EM of a small test and automation tools team, I wanted to make a difference. What we try to achieve is more important than what our title is.
• I replaced formal authority with data points and used them to convince people. People tend to listen more attentively if you have sufficient data points to corroborate your arguments. I find that to be as compelling as having a C-suite title.
• Every step I took along this journey was a result of deep reflection on why we were doing X vs Y. It is a cognitive load to constantly think about your actions. It is critical to help you understand what is working and what you should do differently.
• Choosing a smaller subset of teams to start with was critical. We could have started big, but then we would have failed big as well. By starting small, we choose to add small wins, one by one, while being prepared for small fails.