Managing Tech Debt: A Success Story
7 September, 2021
In the early days, we iterated fast, which accumulated significant technical debt. Some of it was intentional, some certainly not. As I transitioned to manage multiple teams, I realized how tech debt slowed us down and made sluggish some team members fighting it. At that point, I could witness a piece of code getting ramified into thousands of lines of code, which was perplexing to engineers and prevented them from changing any functionality. Most engineers were afraid to come nowhere near these pieces. Also, tech debt affected our ability to scale. The system was breaking all the time. People would be paged in the middle of the night because the traffic in Europe or Japan would be peaking up, and we couldn’t handle the load.
Most importantly, tech debt was hammering the team, impacting their ability to ship features. We didn’t have a proper mechanism to deal with tech debt, so people working on features would spend a couple of weeks completing their tasks instead of days. Finally, because of our inaction in managing tech debt, we noticed that people started to fix it randomly -- and without almost any visibility -- rather than focusing on work they were assigned to.
In essence, I knew we needed to have an overall tech debt strategy that would make things more transparent. We had to surface the problem of tech debt; it couldn’t stay hidden anymore. Then, it has to be properly prioritized as we could fix everything at once. Furthermore, we had to create an environment for the team to address tech debt more proactively and continuously. Enough time should be allocated to manage tech debt, and measuring success should be a priority from the start. In the end, we should be able to communicate with stakeholders how much debt has been solved or how much velocity has increased, etc. We broke this process down into smaller steps and took a step at a time.
First and foremost, I initiated a series of conversations with stakeholders, convincing them that tech debt is real. We had to face it or bear the consequences of denial. I had to ensure that all stakeholders were willing to acknowledge the problem and understand the need to solve it. I believed we should do features and tech debt planning together from the start since I wanted to ensure full visibility and alignment by involving all stakeholders.
Then I deep-dived into a voluminous literature to understand how different people addressed the problem. I came across Steve Garnett’s blog and his Strategies for Technical Debt Prioritization that developed the tech debt prioritization strategy based on various criteria. The first criterion dissects how frequently tech debt happens. For example, if tech debt is hidden in a large function and people need to use it often, it is obviously a problem. But if the function runs by itself and no one needs to touch it for months, then you should direct your focus elsewhere. The amount of effort a team needs to spend fixing it is another criterion: is it a quick fix or a large investment? The shorter the amount, the sooner it should be fixed. Sometimes a large effort can be broken down into smaller pieces. The third criterion evaluates both the customer and engineering efficiency impact once the debt is fixed. In terms of the customer impact, you should ask yourself how fixing tech debt would improve reliability or usability for the customer and how significantly. If not for the customer, does it have an impact on engineering velocity?
Next, I had to enable the whole team to service various pieces of technical debt. It should be more a bottom-up, then top-down effort, where the team could surface it on Jira board or GitHub. Once surfaced as a ticket, I would work with a team to measure frequency, impact, and efficiency. The best way to approach it is by assigning all pieces a score, and those with a high score would become priorities. All of that, I had to communicate with the team and make sure they understood and aligned on these high-priority items.
The amount of time allocated to the team to fix tech debt is always a subject of debate. We allocated 25 percent of the time in each sprint, making sure that we had enough time to tackle it. One of the things I learned was not to be too attached to a specific percentage. Sometimes urgency would appear from the product side, sometimes from the technical side; therefore, I tried to be flexible on a week-by-week basis. I was committed to keeping it 25 percent on the annual average, then sticking to the number on a weekly basis.
Finally, I had to create a mechanism that would enable us to measure success. We started to track how much tech debt we could fix over time. That became a number that didn’t tell us much. We used a number of tickets to measure success. During some sprints, we addressed five or ten tech debts, during the other one or two. It would fluctuate a lot without telling us why and what was the impact of fixing that tech debt. Later we started to track our engineering velocity too. If we could improve tech debt, we would see improvement of engineering velocity reflecting in a number of story points.
- Addressing tech debt is a collaborative effort that should coalesce top-down and bottom-up approaches. A top-down approach will -- by means of a well-defined strategy -- ensure the alignment of stakeholders. But day-to-day execution should be bottom-up. Make sure you secure buy-in from the team. The team should have a say in how tech - debt should be addressed and prioritized.
- When we started servicing tech debt issues, we created a tech debt backlog. There were hundreds of items, which scared people a great deal. The sheer amount of items had a devastating psychological effect on engineers. We learned later that half of the issues were unimportant or we would never be able to address them, so we just got rid of them. I was driven by the belief that people should focus on what mattered the most. I was concerned that we would never know if we got rid of something important by discarding some of the issues. But, my logic was that if it is something important, it will come back again.
Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader
Scott Jones, Senior Product Manager of Cloud Data Services at the Hewlett Packard Enterprise, shares his approach to market research.
Senior Product Manager (Master Level) at Hewlett Packard
Ben Picolo, Engineering Manager at PolicyGenius Inc., shares how together with his team, they gamified the whole technical debt solving process.
Engineering Manager at PolicyGenius Inc.
Deepak Kumar Singh, Principal Product Manager at Omise, shares his experience during a customer's behavior on payments and how to convince the CEO of specific actions.
Deepak Kumar Singh
Senior Product Manager at Yelp
Beier Cai, Co-founder, and CTO at Commit, outlines his meticulously developed approach to managing tech debt, explaining how the process was refined along the way.
Co-founder and CTO at Commit
Anna Nicanorova, VP of Engineering at Annalect, broadly speaks about the degree to which a system’s components can be separated and recombined with the benefit of flexibility and variety in use.
VP of Engineering at Annalect
You're a great engineer.
Become a great engineering leader.
Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.