We've just launched plato for individuals

🔥

login


Google Sign inLinkedIn Sign in

Don't have an account? 

Lessons From Leading Total Rewrite Projects

Leadership
Conflict solving
Dev Processes
Ownership
Delegate
Coaching / Training / Mentorship

6 December, 2017

Dmitriy Ryaboy discusses the lessons he has learned from leading total rewrite projects, and discusses the importance of the team being responsible for both maintaining old systems and developing new ones.

Problem

In 2014, Twitter wanted to rewrite its internal A/B testing system. There was a team that had created the first system. However, there was a perception from the company that the system wasn't working very well. When the system said an experiment was underperforming, people tended to blame the testing framework rather than believe the results -- a trust problem. A lot of teams also felt it was difficult to measure the things they wanted to measure. Some of this was based on perception, and some was based in reality. A few senior engineers from other groups came together to form an ad-hoc working group, with the VP Eng's blessing, to develop a new tool from the ground up and started implementing it. The problem was that this created an "us" versus "them" situation, where there was the old team that owned the product, who was too small to create the new tool given all of the challenges it would face, and there was the other team who thought they could do it better, so decided they'd just go around the old team, but lacked context and lessons learned from the old system. This situation created tension and a lack of clarity across the board.

Actions taken

The team who had originally built the tool felt completely sidelined and unappreciated; the new team was perceived as dismissive of their efforts -- yet the burden of keeping the older production system up still landed on the old team. The other side felt that whenever the original team made improvements to their tool, they were attacking the working groups work, as now they had to continually catch up with what the old tool could do, whilst still building the new tool. I realized that:

  • A/B testing was really important for the company
  • the new project wasn't going to succeed the way it was set up; and
  • I was one of the few people that all of the camps trusted I decided to help fix the problem. I argued that a new team with responsibility for A/B testing should be created, staffed with both data scientists and engineers. I merged the two teams, and pulled in people responsible for critical components A/B testing relied on from a few other groups. The single new team was responsible for both maintaining the old system and developing a new one. This really helped to align people behind one goal, and we could more easily agree on how we were going to verify that the new system was correct and how we were going to migrate people off of the old system. Had to build trust and make it clear everyone's contribution was valued, regardless of "old" vs "new" affiliation. For example the lead data scientist from the old team was given decision power over when we switch to the new system in production, and asked to do the analysis to support the decision. That empowered her and prevented her from feeling as though she had been sidelined, while also putting her in a position to discover issues in the old system and converting her from a sceptic to an advocate for change.

Lessons learned

I have led teams that wound up completely replacing a system a number of times. One thing I've seen consistently is that while it is tempting to separate people into a legacy-support team and a fun-new-system team, this approach doesn't work well. Don't create parallel efforts that build resentment, bad blood, lack of context, and misaligned incentives. The team building something new needs to understand practical considerations that went into the old system. There's a reason it has grown awkward warts and convoluted business logic -- this was a response to the business logic often actually being convoluted! Learning all the special cases and gotchas can take a long time, and what seems like a straightforward, clean solution may be flawed because it doesn't account for real-world subtleties. It's also good to be responsible for the old system because you become intimately familiar with its failings, and what exactly it does well, and what it doesn't do so well; you can also discover good incremental ways to migrate off it. The reasons for an old system failing are likely to be similar to the reasons a new system will fail, and it is helpful for people to get to know the old system so they can avoid repeating mistakes. Nothing quite incentivises delivering a new piece of software quickly -- and having it be production-ready and stable -- like having to fix bugs and put out fires in the legacy software until the new stuff is ready. On the other hand, there are few things as demotivating as having your team be told that you have to support something that's seen as outdated and waiting to be turned off, and that you shouldn't even be improving it since that would be wasted effort and scope creep for the new system.


Related stories

How to Help Your Reports Grow and Pursue Their Goals
27 September

Himanshu Gahlot, Director of Engineering at Lambda School, shares how he used his own learnings to support his direct reports and help them grow in their careers.

Coaching / Training / Mentorship
Career Path
Himanshu Gahlot

Himanshu Gahlot

Director of Engineering at Lambda School

Driving Clarity of Charter in a Large Organization
27 September

Jeffrey Wescott, Director of Engineering at Splunk, describes how he introduced clarity on ownership between four disparate teams by drafting a charter that precisely demarcated who owned what.

Ownership
Reorganization
Team reaction
New Manager of Manager
Managing Up
Jeffrey Wescott

Jeffrey Wescott

Director of Engineering at Splunk

What Will Make You a Great Engineering Leader
27 September

Pete Murray, Principal Software Engineer at Electronic Arts, discusses what makes one a great engineering leader and singles out creating opportunities and motivating engineers as two key characteristics.

Leadership
Personal growth
Coaching / Training / Mentorship
Pete Murray

Pete Murray

Principal Software Engineer at Electronic Arts

How to Introduce a New Technology to Your Organization
27 September

Pete Murray, Principal Software Engineer at Electronic Arts, recalls his efforts to introduce a cutting edge technology of that time and how that was intrinsically connected with his personal growth as an engineer.

Dev Processes
Impact
Internal Communication
Convincing
Pete Murray

Pete Murray

Principal Software Engineer at Electronic Arts

Getting From What to Why
27 September

Pete Murray, Principal Software Engineer at Electronic Arts, explains how he successfully solved a long-troubling problem by applying a root cause analysis.

Dev Processes
Productivity
Impact
Pete Murray

Pete Murray

Principal Software Engineer at Electronic Arts

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.