Reducing significant inherited technical debt

Problem

"When inheriting the team and codebase of a start-up with a product that has going through many transformations to find product-market fit, I discovered quickly that there was a ton of technical debt, including silent failures, huge performance problems, outages, and regressions. The CEO was very used to getting what he needed at a moment's notice, and this merely added to the problem."

Actions taken

"The most important immediate action I took was to give visibility to the CEO of the culture that had been created and the tangible, negative impact of him being used to getting what he wanted at a moment's notice. By educating him and the rest of the executive team in a very tangible way, I was able to get their commitment to change their behavior by changing the process of how things were requested and serving as the 'shield' for the team whenever the CEO wanted something new." "With that fire out, I focused on diving into the codebase deeply, working with the team to identify new or existing technologies that could be used to improve the current situation. From this, we created a 'Technical Debt Roadmap' which identified all of the things we could do to make the situation better, prioritized based on impact and time-to-implement. We then committed about 20% of all working time to items on this roadmap and over many months focused on getting this done." "Aside from this, we committed to post-mortem every issue that came up, particularly regressions and outages. We also made it a requirement that no post-mortem could be done without requiring a process change as part of the post-mortem itself -- it was never enough to say 'Person X should do Y differently next time', particularly because humans are imperfect. Instead, we would require adding a process, whether it's an automated test, a new documented process to follow, or something else. While it may seem like this could lead to a lot of bureaucracy, in fact it only led to preventing the same mistakes from happening again, and it kept us honest with exactly what was required to maintain all of our existing features and functionality."

Lessons learned

"When inheriting a new team or codebase, it can be easy to focus on the 'symptoms' of technical debt, but if the focus is on fixing these symptoms without addressing the core problem of how culturally the technical debt is accrued, you will only ensure that this debt will come back overtime. Unfortunately, difficult conversations are often necessary, as it may require behavior change of very senior members of the organization to change some of these bad habits. In addition, it's important to find a way to still innovate quickly and accrue some technical debt along the way if it's what is in the best financial interest of the company."