login


Google Sign inLinkedIn Sign in

Don't have an account? 

Crush Tech Debt with Taxation, not Micromanagement

Collaboration
Productivity
Team processes
OKR

17 January, 2019

Eugene Marinelli describes one failed (roadmap-based) and one successful (taxation-based) approach to eliminating flaky tests that increased the productivity and accountability of the team at Blend.

Problem

Merging changes to our most active codebase requires passing a suite of end-to-end headed tests. For almost a year, many of these tests were flaky (didn't pass 100% of the time), meaning that every engineer working on that codebase would experience frequent merge build failures, having to "retry the merge" many times per PR. These tests are unpleasant to debug because they run slowly against a browser, require the backend to be running in a certain configuration, and behave slightly differently when run locally vs. in our CI environment.

Actions taken

Our first attempt to improve the reliability of the test suite was to prioritize flaky test fixing as an item in each pod's roadmap (a pod is a cross-functional team consisting of engineers, a PM, and a designer). For instance, if a pod owned 5 flaky tests at the time of planning, we would estimate and allocate time in their roadmap for the quarter to fix the 5 tests. This required a considerable amount of negotiation to prioritize test suite work against other priorities. Even after the roadmap was in place, fixing tests would typically take a back seat to more urgent product work over the course of the quarter. The test suite remained flaky. Our second approach was inspired by one way in which governments solve tragedies of the commons. A tragedy of the commons is a situation where a resource is shared by a group of parties, and it decays because no one is individually incentivized to maintain it. One approach to solving a tragedy of the commons is to divide the shared resource and privatize ownership of each part, requiring each party to pay a tax proportional to their contribution to the problem. We applied this idea to the test suite. We instituted a policy where, before each sprint, we calculated each pod's "tax" — a number of engineers owed proportional to the number of flaky tests owned by the pod. Those engineers would be required to join the "Technical Health Pod" the following sprint, devoted to fixing flaky tests. This created a skin-in-the-game incentive for each pod to get ahead of flaky test debt to avoid losing engineers and disrupting their main work stream. It worked extremely well, and quickly. Within a month, the flaky tests were almost completely eliminated, all without any micromanagement. No one has ever been pulled into the Technical Health Pod. We've expanded the tax to cover other types of tech debt in shard resources with similar results.

Lessons learned

Roadmapping the work to improve shared resources isn't necessarily effective, especially in a culture of distributed ownership and high autonomy like Blend's. Instead of micromanaging each pod's task list, try creating an incentive such as a tax to drive behavior that better serves the greater good in a systematic, continuous fashion.


Related stories

How to Effectively Communicate on Slack
6 July

Shridharan Muthu, VP of Engineering at Zoosk, discusses effective communication using Slack including a recommended framework that entails three simple tips to make the most of the tool.

Internal Communication
Remote
Productivity
Shridharan Muthu

Shridharan Muthu

VP of Engineering, Backend Applications at Zoosk

Improving Collaboration Between Engineering and Product Across Time Zones
6 July

Shridharan Muthu, VP of Engineering at Zoosk, describes how to make communication effective between PMs and engineers when they are located in different time zones and have very little overlap.

Collaboration
Internal Communication
Reorganization
Remote
Shridharan Muthu

Shridharan Muthu

VP of Engineering, Backend Applications at Zoosk

Handling a Mistake - Adopting a New Workflow
6 July

Shridharan Muthu, VP of Engineering at Zoosk, describes how he quickly agreed to adopt new workflows, a mistake he later regretted, and how he handled the situation by spending the time to course correct and taking a stab at making things easier for his team.

Team processes
Agile / Scrum
Collaboration
Shridharan Muthu

Shridharan Muthu

VP of Engineering, Backend Applications at Zoosk

Cultivating a Relationship Between Collocated and Remote Teams
3 July

Arjun Rao, Director of Engineering at Place Exchange, highlights three ways that induce a genial, positive, and flourishing atmosphere between collocated teams and their remote, contracted, or outsourced counterparts.

Remote
Collaboration
Company Culture
Arjun Rao

Arjun Rao

Director of Engineering at Place Exchange

What We Learned From Running Open Spaces
30 June

Jeff Foster, Head of Product Engineering, highlights key learnings from his experience of running open spaces and if and how it contributed to an increase in innovation.

Company Culture
Productivity
Impact
Jeff Foster

Jeff Foster

Head of Product Engineering at Redgate

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.