Plato Elevate Winter Summit has been announced (Dec 7th-8th)

🔥

Back to resources

Tech Debt Bankruptcy

Tech Debt

17 February, 2021

Mike Weaver
Mike Weaver

VP of Engineering at Invoca

Mike Weaver, VP of Engineering at Invoca, explains how he successfully dealt with tech debt bankruptcy by introducing a three-level priority system that prioritized production incidents over planned work.

Problem

After a multi-month long series of customer-impacting events caused by known issues we hadn’t prioritized, we realized that our fixed, 20 percent tech debt budget was not working. We could anticipate most of the problems; these were the things we knew were brittle or a bit broken, but we kept putting them off.

Actions taken

Gathering Data

We all knew that teams were being regularly sidetracked by surprises that came from production incidents or customers, but not how much. To quantify this, we starting tagging all items that appeared and began work within the same week. It turned out that 40 percent of our time was going to this “unplanned” work. This was such a huge number that it was easy to get the Product and Executive team on board with making significant changes to the way we worked.

 

Setting Clear Priorities

We created a three-level priority system for work that put working on product features LAST:

  • Production issues (Hotfixes, RCA items, security vulnerabilities, visibility issues);
  • Blockers (deploy pipeline, phantom tests, etc.);
  • Planned work (both tech work AND product work).

This really isn’t anything shocking; all teams prioritize production incidents over planned work. What was different was what we classified as “production.” Critically, we included many things that were not directly customer-impacting, like fixing issues identified in Root Cause Analysis (RCA) from previous incidents and issues with our monitoring, alerting, and logging systems. Also different was our second priority, Blockers: anything that prevents or slows down a team’s work. By putting this above the planned work, we ensured that all teams were unblocked and able to work efficiently.

Establishing Ownership

We were all aware of the service ownership (aka “Spotify model”) of team organization but did not think we were big enough to warrant such a structure. Work was being prioritized top-down, sent out to teams based on the judgement of engineering leaders like myself. However, after establishing the development priorities, it became obvious that a lack of clear ownership was causing two significant problems:

  • A “tragedy of the commons”: If everyone owns it, then no one does.
  • Those doing the top-down prioritization lacked awareness of the issues with the services.

The combination of problems caused many critical issues to never get worked on. To resolve these issues, we did two things:

  • Enumerated every service and significant feature in the platform and assign a team to own it.
  • Completely rebuilt our prioritization process to be bottom-up with the teams responsible for establishing their own roadmaps.

The first, enumerating the services, was relatively easy. Switching to bottom-up prioritization required some significant effort with the product team and executive leadership. Despite copious research showing this methodology worked, ultimately, they remained skeptical and would only commit to a six-month test. That six-month test was a resounding success. Unplanned work dropped in half (and continued dropping after that), production incidents virtually disappeared, and developer happiness (as measured via a survey) doubled.

Lessons learned

  • Service Ownership Works at Small Scales. After seeing this organizational structure in place with only four teams, I am convinced that it would work well with any number of teams larger than one.
  • There is more to “production” than your customer-facing components. Our commitment to prioritize all things that were impacting product, customer visible or not, made a massive impact to the long term stability of our platform. This especially applied to RCA items. If you can only prioritize one thing above planned work, prioritize those.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

Strategic Ways to Stop Losing Customers

13 October

Mary Fisher, Software Engineering Manager at DrChrono, shares how diligently she worked with teams within her organization to retain customers.

Alignment
Convincing
Dev Processes
Innovation / Experiment
Prioritization
Product
Tech Debt
Mary Fisher

Mary Fisher

Software Engineering Manager at DrChrono

Prioritizing Technical Debt

24 September

Ben Picolo, Engineering Manager at PolicyGenius Inc., shares how together with his team, they gamified the whole technical debt solving process.

Dev Processes
Tech Debt
Team Processes
Ben Picolo

Ben Picolo

Engineering Manager at PolicyGenius Inc.

Managing Tech Debt: A Success Story

4 October

Beier Cai, Co-founder, and CTO at Commit, outlines his meticulously developed approach to managing tech debt, explaining how the process was refined along the way.

Strategy
Tech Debt
Beier Cai

Beier Cai

Co-founder and CTO at Commit

Introducing Modularity and Creating Componentization

26 August

Anna Nicanorova, VP of Engineering at Annalect, broadly speaks about the degree to which a system’s components can be separated and recombined with the benefit of flexibility and variety in use.

Dev Processes
Tech Debt
Anna Nicanorova

Anna Nicanorova

VP of Engineering at Annalect

From Building Features to Leading Product Development

16 August

Jan Macek, VP of Engineering at Vendavo, shares how his team moved away from operating as feature factory.

Tech Debt
Prioritization
Jan Macek

Jan Macek

VP of Engineering at Vendavo

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.