Making the Unpopular Decision to Focus on Stability Before New Products

Operational Errors and Pressure from Leadership

As an engineering leader, I’ve found that one of the hardest things to do is slow down development to focus on stability. Usually, this is not a popular decision and may impact relationships with stakeholders and people who do not understand the value if not communicated well. My team saw an uptick in operational errors, bad deployment, and bugs after production on projects. At the same time, my team faced a lot of pressure from our product and commercial partners to develop new features to deliver value to our users.

My team was behind on key initiatives and projects, partly due to a build-up of tech debt and fixing previous issues with our software. My team kept on track, delivering value while continuing to mitigate our older challenges. At a certain time, though, I realized that my team was trending in an unsuccessful direction, and unless there was a drastic change, our situation would only escalate.

Using Data to Educate a Company

Sparking Conversation:

When I noticed my team’s trend, I opened a line of communication dedicated to our challenges. I acknowledged and brought awareness to the problem, and even though I didn't have an immediate solution, I was working on one that will get us back to a healthy place. Not only was this communication for my team, but also for the product and commercial partners. I tried to bring empathy to these conversations, as I knew that our organization needed to continue to deliver value, but I ensured my partners knew the technical difficulties my team was having was preventing that and made it to my team that I was aware of the problems and was committed to getting us to a better place.

Measurement:

I began measuring the number of incidents found within our products and the discrepancy of time between firefighting or delivering new products. I wanted to be able to bring other leaders data that clearly visualized the challenges of our team and the amount of ad hoc tasks that were taking away from our delivery of new features. I also tried to understand the outcomes that we were struggling with (high availability, reduction in deployment errors, increased cycle time of taking a feature from development to production) and try to understand what good would look like in those areas. I made this data visible to our teams and brainstormed with them the various things that were slowing down our delivery as well as the root cause for these issues.

Use Data To Tell The Story:

I took the data around these outcomes and communicated to and educated both internal and external teams about the data and how we were struggling in those areas. I knew that it was a difficult decision for our organization to slow down our product development and wider buy-in was needed but I knew that the situation was only going to get worse if not addressed. I took the data, the suggestions, and communicated the story to leadership:

• What the current problem (using data) and where we will be if nothing gets done.
• My rough plan for what we can do to ensure we get to a healthier state where this won't just come again.
• What the tradeoffs were in the short / medium term. How long until we can get back to a healthier state.
• What the new norms of balancing tech devt and product development would have to be to avoid getting into this situation in the future.
  

By coming prepared with data and telling a compelling story of what the end-value will be to the user (product stability + faster future product development which would drive more customer value) I was able to get leadership and the stakeholders to buy in and shifted our teams focus to pay down a lot of tech debt and get our system into a healthier place.

Understand the Outcomes You Want to Create a Clear Plan

• Understand the Outcomes you want to drive. Don’t just list out a few initiatives you want to deliver or bugs you want to fix. What is the goal you want to achieve, what will the state of the world look like where you are not swimming in debt. Have a vision and understand the outcomes you want to drive. Aligning the team towards an outcome is more effective than just having them focus on seemingly random tasks.

• Create a clear plan for what you need to do to achieve those outcomes. The worst thing an engineering leader can do when making a difficult decision is to begin conversations without a plan or a plan that is not ambitious enough that you will end up in this same situation in 3 months' time. Try to address the root cause of the issues or blockers/slowness in the delivery pipeline. Don’t just add more bandaids that will need to be fixed a few months down the line again.

• Make sure to build up enough buffer into the plan to give the team enough time to experiment or have some slack if there are delays. Timeboxing improvement efforts can be an effective tactic as well as outcome-based initiatives (we are working on various improvements in this area until we achieve outcome X).

• Ensure that you negotiate and establish a new normal after this effort is done to have a healthy split of product development and focus on paying down tech debt. You do not want to have this problem again in 6 months. Try to continuously pay down your tech debt rather than waiting and doing it in big chunks.