Redefining And Restructuring A Team

Problem

Last year, I inherited a team consisting of two Data Scientists and two Data Engineers. In reality, there were two Backend engineers and two brilliant Physics PhDs short on industry experience. There was an incredible amount of tech debt as people had learned analytics and data warehousing by doing. The difference between Data Engineering and Backend Engineering was also quite confused. At the same time, another new leader came on to deal with business analytics, and the company dealt with some key departures. Which department should own what became really unclear.

Actions taken

I had many discussions with others about how they saw things: stakeholders of course in Engineering and Product but also in Sales, Customer Success, Marketing, and Finance (essentially the whole org) with leaders and individual contributors. I talked to my new team to find out where their skill sets lay and to assess their best fit. Most were feeling a bit fish-out-of-water and were eager for some more clarity. I wrote a big list of systems and responsibilities, and with my team and the leaders of Backend and Business Analytics agreed on who would take what and on what timeline.

"Most were feeling a bit fish-out-of-water and were eager for some more clarity."

I got relatively quick agreement with these partners, but there wasn't much follow through. People who were supposed to take over certain components couldn't pull away from client commitments (and new requirements flooded in), so I was faced with keeping the lights on with the existing technology and a bunch of grand commitments I didn't have resources to execute. To get through this, I went about finding and then hiring really strong Data Engineers who could build the new systems while I worked to stabilize and migrate existing systems.

The Backend engineers I had inherited had the aptitude and desire to stay within the Data Team and learn more data eng & sci. But ultimately to let go of the Backend systems, I had to release those engineers into the Backend organization as well. We eased this transition by giving one engineer a tech lead role (which he had strongly earned) and he embraced, and the other was keen to work on a new tech stack in Backend. They had been promised a path in data, so we needed to make sure they were taken care of and happy.

Lessons learned

I was initially surprised by how quickly I was able to get agreement on things, but this was because the culture supported easy agreement but shallow follow-thru (something caused by years of firefighting and re-planning fatigue).

"The on-going challenge I've been faced with is making the understanding of who does what stick."

The on-going challenge I've been faced with is making the understanding of who does what stick. We've had a lot of turnover, so new folks come in with different understandings and expectations, so it's a continual task to retrain and evangelize at all levels and across all departments. At this point, there are enough people seeded throughout the org, and we have documentation and slide decks ready to go to make this go reasonably smoothly.