Back to resources

Managing Tech Debt: A Success Story

Strategy
Tech Debt

4 October, 2021

Beier Cai
Beier Cai

Co-founder and CTO at Commit

Beier Cai, Co-founder, and CTO at Commit, outlines his meticulously developed approach to managing tech debt, explaining how the process was refined along the way.

Problem

In the early days, we iterated fast, which accumulated significant technical debt. Some of it was intentional, some certainly not. As I transitioned to manage multiple teams, I realized how tech debt slowed us down and impeded team members who were fighting it. At that point, I could witness a piece of code getting ramified into thousands of lines of code, which was perplexing to engineers and prevented them from changing any functionality. Most engineers were afraid to come nowhere near these pieces. Also, tech debt affected our ability to scale. The system was breaking all the time. People would be paged in the middle of the night because the traffic in Europe or Japan would be peaking up, and we couldn’t handle the load.

Most importantly, tech debt was hammering the team, impacting their ability to ship features. We didn’t have a proper mechanism to deal with tech debt, so people working on features would spend a couple of weeks completing their tasks instead of days. Finally, because of our inaction in managing tech debt, we noticed that people started to fix it randomly -- and without almost any visibility -- rather than focusing on work they were assigned to.

Actions taken

In essence, I knew we needed to have an overall tech debt strategy that would make things more transparent. We had to surface the problem of tech debt; it couldn’t stay hidden anymore. Then, it has to be properly prioritized as we could fix everything at once. Furthermore, we had to create an environment for the team to address tech debt more proactively and continuously. Enough time should be allocated to manage tech debt, and measuring success should be a priority from the start. In the end, we should be able to communicate with stakeholders how much debt has been solved or how much velocity has increased, etc. We broke this process down into smaller steps and took a step at a time.

First and foremost, I initiated a series of conversations with stakeholders, convincing them that tech debt is real. We had to face it or bear the consequences of denial. I had to ensure that all stakeholders were willing to acknowledge the problem and understand the need to solve it. I believed we should do features and tech debt planning together from the start since I wanted to ensure full visibility and alignment by involving all stakeholders.

Then I deep-dived into a voluminous literature to understand how different people addressed the problem. I came across Steve Garnett’s blog and his Strategies for Technical Debt Prioritization that developed the tech debt prioritization strategy based on various criteria. The first criterion dissects how frequently tech debt happens. For example, if tech debt is hidden in a large function and people need to use it often, it is obviously a problem. But if the function runs by itself and no one needs to touch it for months, then you should direct your focus elsewhere. The amount of effort a team needs to spend fixing it is another criterion: is it a quick fix or a large investment? The shorter the amount, the sooner it should be fixed. Sometimes a large effort can be broken down into smaller pieces. The third criterion evaluates both the customer and engineering efficiency impact once the debt is fixed. In terms of the customer impact, you should ask yourself how fixing tech debt would improve reliability or usability for the customer and how significantly. If not for the customer, does it have an impact on engineering velocity?

Next, I had to enable the whole team to service various pieces of technical debt. It should be more a bottom-up, then top-down effort, where the team could surface it on Jira board or GitHub. Once surfaced as a ticket, I would work with a team to measure frequency, impact, and efficiency. The best way to approach it is by assigning all pieces a score, and those with a high score would become priorities. All of that, I had to communicate with the team and make sure they understood and aligned on these high-priority items.

The amount of time allocated to the team to fix tech debt is always a subject of debate. We allocated 25 percent of the time in each sprint, making sure that we had enough time to tackle it. One of the things I learned was not to be too attached to a specific percentage. Sometimes urgency would appear from the product side, sometimes from the technical side; therefore, I tried to be flexible on a week-by-week basis. I was committed to keeping it 25 percent on the annual average, then sticking to the number on a weekly basis.

Finally, I had to create a mechanism that would enable us to measure success. We started to track how much tech debt we could fix over time. That became a number that didn’t tell us much. We used a number of tickets to measure success. During some sprints, we addressed five or ten tech debts, during the other one or two. It would fluctuate a lot without telling us why and what was the impact of fixing that tech debt. Later we started to track our engineering velocity too. If we could improve tech debt, we would see improvement of engineering velocity reflecting in a number of story points.

Lessons learned

  • Addressing tech debt is a collaborative effort that should coalesce top-down and bottom-up approaches. A top-down approach will -- by means of a well-defined strategy -- ensure the alignment of stakeholders. But day-to-day execution should be bottom-up. Make sure you secure buy-in from the team. The team should have a say in how tech - debt should be addressed and prioritized.
  • When we started servicing tech debt issues, we created a tech debt backlog. There were hundreds of items, which scared people a great deal. The sheer amount of items had a devastating psychological effect on engineers. We learned later that half of the issues were unimportant or we would never be able to address them, so we just got rid of them. I was driven by the belief that people should focus on what mattered the most. I was concerned that we would never know if we got rid of something important by discarding some of the issues. But, my logic was that if it is something important, it will come back again.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

How to Organize, Manage, and Grow Your Team

12 July

Vineet Puranik, Senior Engineering Manager at DocuSign, discusses the impact of roadmaps, organization, and proper management for your teams to procure growth.

Managing Expectations
Delegate
Collaboration
Roadmap
Strategy
Vineet Puranik

Vineet Puranik

Senior Engineering Manager at DocuSign

Bootstrapping a Startup While Working Full-Time

23 June

Lucjan Suski, CEO & Co-founder of Surfer, relates how he started a company as a side project and shares his insights on bootstrapping tech startups.

Innovation / Experiment
Motivation
Strategy
Lucjan Suski

Lucjan Suski

Co-founder, formerly CTO and CEO at Surfer

Managing Through a Team Reorganization

15 June

Mugdha Myers, former Engineering Manager at Google, discusses the challenges of leading a team through the ambiguity and anxiety caused by a large-scale team restructuring.

Alignment
Changing A Company
Strategy
Changing Company
Mugdha Myers

Mugdha Myers

Engineering Manager at N/A

Streamlining Product Processes After a Reorganization

16 May

Snehal Shaha, Lead Technical Program Manager at Momentive (fka SurveyMonkey), details her short-term technical strategy to unify processes among teams following an acquisition.

Acquisition / Integration
Product Team
Product
Building A Team
Leadership
Internal Communication
Collaboration
Reorganization
Strategy
Team Processes
Cross-Functional Collaboration
Snehal Shaha

Snehal Shaha

Technical Program Management at Apple Inc.

Navigating Disagreements When It Comes to Priorities

9 May

Pavel Safarik, Head of Product at ROI Hunter, shares his insights on how to deal with disagreements about prioritization when building a product.

Innovation / Experiment
Product Team
Product
Dev Processes
Conflict Solving
Internal Communication
Collaboration
Convincing
Strategy
Prioritization
Pavel Safarik

Pavel Safarik

Head of Product at ROI Hunter