Crush Tech Debt with Taxation, not Micromanagement

Collaboration
Productivity
Team processes
OKR

17 January, 2019

Eugene Marinelli describes one failed (roadmap-based) and one successful (taxation-based) approach to eliminating flaky tests that increased the productivity and accountability of the team at Blend.

Problem

Merging changes to our most active codebase requires passing a suite of end-to-end headed tests. For almost a year, many of these tests were flaky (didn't pass 100% of the time), meaning that every engineer working on that codebase would experience frequent merge build failures, having to "retry the merge" many times per PR. These tests are unpleasant to debug because they run slowly against a browser, require the backend to be running in a certain configuration, and behave slightly differently when run locally vs. in our CI environment.

Actions taken

Our first attempt to improve the reliability of the test suite was to prioritize flaky test fixing as an item in each pod's roadmap (a pod is a cross-functional team consisting of engineers, a PM, and a designer). For instance, if a pod owned 5 flaky tests at the time of planning, we would estimate and allocate time in their roadmap for the quarter to fix the 5 tests. This required a considerable amount of negotiation to prioritize test suite work against other priorities. Even after the roadmap was in place, fixing tests would typically take a back seat to more urgent product work over the course of the quarter. The test suite remained flaky. Our second approach was inspired by one way in which governments solve tragedies of the commons. A tragedy of the commons is a situation where a resource is shared by a group of parties, and it decays because no one is individually incentivized to maintain it. One approach to solving a tragedy of the commons is to divide the shared resource and privatize ownership of each part, requiring each party to pay a tax proportional to their contribution to the problem. We applied this idea to the test suite. We instituted a policy where, before each sprint, we calculated each pod's "tax" — a number of engineers owed proportional to the number of flaky tests owned by the pod. Those engineers would be required to join the "Technical Health Pod" the following sprint, devoted to fixing flaky tests. This created a skin-in-the-game incentive for each pod to get ahead of flaky test debt to avoid losing engineers and disrupting their main work stream. It worked extremely well, and quickly. Within a month, the flaky tests were almost completely eliminated, all without any micromanagement. No one has ever been pulled into the Technical Health Pod. We've expanded the tax to cover other types of tech debt in shard resources with similar results.

Lessons learned

Roadmapping the work to improve shared resources isn't necessarily effective, especially in a culture of distributed ownership and high autonomy like Blend's. Instead of micromanaging each pod's task list, try creating an incentive such as a tax to drive behavior that better serves the greater good in a systematic, continuous fashion.


Related stories

Pivot to Transform Failures into Opportunities
30 May

Marc LeBrun, VP of Engineering at Flow Kana and a co-creator of the Apple Mac, recalls how one of his early prototypes failed to meet the original intent but how that failure turned into an unanticipated opportunity.

Productivity
Impact
Product
Marc LeBrun

Marc LeBrun

VP Engineering at Flow Kana

Developing High Performing Teams: How To Transform a Software Developer Into a Product Engineer
30 May

Alessandro Pintaudi, Product Management Director at Payfit, comes up with an exciting proposal of transforming software developers into product engineers by establishing cross-functional context analysis and shared objectives.

Cross-functional collaboration
Product Team
Collaboration
Alessandro Pintaudi

Alessandro Pintaudi

Product Management Director at PayFit

Leveraging Diverse Peer Groups for Tighter Feedback
24 May

With both the need for a more supportive team setting and shorter feedback cycles, Marc LeBrun, VP Engineering at Flow Kana, addresses two problems with a single solution.

Team reaction
Cross-functional collaboration
Collaboration
Marc LeBrun

Marc LeBrun

VP Engineering at Flow Kana

Aligning an Ineffective Team to Realize Product Potential
21 May

Kowsheek Mahmood, Principal and CTO at ArchetypeTech, explains how he adapted an ineffective team by determining and implementing team-evaluation processes to better align the team on product delivery.

Team reaction
Managing Expectations
Underperformance
Firing
Internal Communication
Productivity
Reorganization
Kowsheek Mahmood

Kowsheek Mahmood

Principal & CTO at ArchetypeTech

How To Identify the Right Product To Build
19 May

Alessandro Pintaudi, Product Management Director at Payfit, talks about how teams need to focus more time on building the right things and how to keep doing it with scale.

OKR
Managing Expectations
Product Team
Product
Dev Processes
Internal Communication
Roadmap
Alessandro Pintaudi

Alessandro Pintaudi

Product Management Director at PayFit

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.