Reducing Tech Debt: Roll Up Your Sleeves
13 August, 2021
Our product had a lot of tech debt. It contained an enormous amount of bugs, many of the foundational elements had been poorly designed, and it lacked abstractions and reusable code. It was one of those tightly coupled monoliths that had sprawled over a period of five to six years because no one was driving the architecture forward or looking at it from the technical debt perspective.
As a result, whenever we had to make a change, that change would propagate all over the place, and simple things that should take an hour would take weeks and break completely unrelated code. With little test coverage, it was risky to make any change - we even had business logic tied to random CSS style properties, which we jokingly called “load bearing CSS”. One in five changes required a rollback which was an enormously high change failure rate and indicated a high likelihood that something would break down catastrophically.
In addition, we had junior engineers continuing to add to tech debt. They worked hard but were not productive. All of that impacted the vision for the product our CEO had, which we couldn’t deliver on given our current state. My mission was to enable and ultimately deliver on that vision.
My first focus was to ensure that we stopped adding to the pile. In an established system, significant effort can be conserved by reusing much of the code. In our technology, engineers were building things over and over again; they were never using reusable libraries or tooling. The sheer amount of variants that propagated as a result was the leading cause of bugs. The first thing I had to do was to create common systems and libraries.
I designed and created numerous modules with frameworks, libraries, and tools. Some I built myself, others I designed and had lead engineers built it. Every single feature we were building carried an opportunity to be reused. We started to always look for ways we could separate the Why from the How and What -- separating the mechanism that we were building and using from the use case that we were building and using it for. Once those items were created we trained the team on how to use them. We did presentations, lunch & learns, and then the team started leveraging those items.
We also made a key decision to shift technologies. Previously we used ReactJS, and we decided to switch to using Vue.js, which was easier to train developers on. With Vue.js, it would take a couple of weeks to teach developers to write maintainable, clean, and production-ready code. Next, we drafted a proof of concept where we tried out every possible application pattern we could think of. It looked at all the patterns holistically: how would we do page loading, web requests, bootstrapping, component communication, store data, etc. Once we decided to shift, it was easy because we had all the questions already answered, and we just had to build the features we wanted to build.
The changes we made were a massive accelerator for us. We stopped adding to the pile and, over time, transitioned to feature development that used common patterns. We did a major refactoring over the course of several years to get rid of all our old legacy code, except for a remaining piece of core functionality. That resulted in a new practice where developing features was not about learning how to code things over and over or write new stuff but how to integrate, tie together and compose using the pieces we had available.
By stopping tech debt from expanding, we could direct our actual engineering capabilities on larger problems like data and system architecture. Because we were using reusable libraries and tools, the features were easier to test. The bug rate went down notably, and the quality of the codebase went up, which accelerated development even further. The metrics we were following indicated significant progress: our cycle time went down from about 14 to three days, and most of that was of increased technical quality. We delivered on a tremendous number of features, unlocked a brand new customer segment through our work, and were able to find an entirely new line of business that ultimately created a previously non-existent line of B2B2C business with huge revenue potential. By improving our technical quality, we were able to develop a platform on which we could build the CEO’s vision.
- If you are in an under-resourced startup, it’s important to make engineering as boring as possible. There shouldn’t be any surprises; everything should be as expected. The more you can achieve consistency and reliability, the more you can shift your focus on problems that matter, such as how to increase customer retention and acquisition, how to improve user acquisition, how to monetize your customer base, etc.
- You have to roll up your sleeves. Many EMs or directors think they shouldn’t code -- and that is in general true. But that has to be balanced with the competencies that are available on your team. If you have a junior team with meager experience that you are trying to show a new way, they need to learn what good looks like. You need to demonstrate and model that for them. Sometimes that means doing work yourself before creating guidelines and guardrails for them to pick it up.
- Communicating the changes you made is more important than the actual changes. For example, if your team doesn’t know that a certain library exists or doesn’t know how to use it, they will continue to use the old thing that was working for them. The only way to make your team use it is to make them aware of it and make the actual use as easy and frictionless as possible.
- Expect pushbacks. In general, engineers like to argue, but that is not the reason not to pursue change. All decisions have their pros and cons, along with failure modes that could be risky. Solid decision-making is about consistently making decisions that provide benefits that bring you closer to the desired outcome. It takes a lot of repetition in communication and supervision to ensure that those decisions are implemented. Sometimes I felt uncomfortable with that amount of repetition and supervision, but it was key to achieving the desired outcomes.
Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader
Brad Jayakody outlines the roadmap to maintaining a healthy balance between technical debt and team growth. However, just as balancing acts go it is important to have a strong foundation.
Director of Engineering at Motorway
Tejas Kokje, Senior Software Engineer at Netflix, Inc., highlights how long-term thinking, planning, and organizing can reduce technical debt in a product organization.
Senior Software Engineer at Netflix, Inc
For Today’s Q&A, we have Ron Pragides. Ron is currently the VP of Engineering at Trustly. Previously he was an engineering lead at Carta, AppDirect, BigCommerce, Twitter, and Salesforce, as well as an advisor to many startups. Welcome!
SVP Engineering at Trustly Group AB
Mary Fisher, Software Engineering Manager at DrChrono, shares how diligently she worked with teams within her organization to retain customers.
Software Engineering Manager at Curative
Ben Picolo, Engineering Manager at PolicyGenius Inc., shares how together with his team, they gamified the whole technical debt solving process.
null at PolicyGenius Inc.