Applying Agile to Data Science Teams
31 March, 2020
I joined a company with engineering teams releasing frequently and consistently. However, the software relied on data science models to make predictions and those models and the surrounding infrastructure to access these models was being released infrequently and inconsistently. The data science team was conducting a lot of research which was making it difficult to say when software or models would be released and this eventually affected the whole company’s ability to react quickly.
We decided to create a data science scrum team. Instead of introducing the most common sprint length of two weeks, we decided to go with weekly sprints as the team was more comfortable with a shorter time frame as this allowed them to change plans more frequently based on the result of research spikes. Additionally, they were more confident in their ability to plan for one week instead of two.
After a couple of months, the team became more accustomed to talking about priorities and deciding as a group on what to focus on and what can be accomplished within a week. At this point, we decided to move to two weeks sprints aligning them with other scrum teams and merging some engineering resources onto the team to make them more cross-functional.
One of the main benefits was that Agile encouraged the team to:
- build a deployment pipeline with unit tests and multiple environments instead of having ad hoc releases and processes
- document the release process
- move configuration out of code such that it could be updated without a new release.
More generally, they migrated from a research approach to a hybrid approach where some time was allocated for research spikes and some were allocated to software development each sprint. The product owner, who was well versed in data science, was able to review the results of the research spikes more frequently than before and decide if the team should continue spiking or if priorities had changed.
- Scrum processes can work for data science and other AI optimization teams. You need to provide teams with the ability to conduct research within these processes.
- You get a lot of ancillary benefits along with the transition as you bring more software engineering into data science teams
Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader
Liz Henderson, an Executive consultant at Capgemini, shares her experience hiring a data team with a manager who was difficult to work with.
Executive consultant at Capgemini
Snehal Shaha, Lead Technical Program Manager at Momentive (fka SurveyMonkey), details her short-term technical strategy to unify processes among teams following an acquisition.
Senior EPM/TPM at Apple Inc.
Kamal Qadri, Senior Manager at FICO, drives the importance of setting expectations when optimizing large-scale requirements.
Head of Software Quality Assurance at FICO
Henning Muszynski, Head of Frontend at Doist, promotes his ideas on how documentation ensures consistency, efficiency, and standardization.
Head of Frontend at Doist
Henning Muszynski, Head of Frontend at Doist, talks about the cost of slow and arduous processes that add up over time and how to bring the changes systematically.
Head of Frontend at Doist
You're a great engineer.
Become a great engineering leader.
Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.