Applying Agile to Data Science Teams
31 March, 2020
I joined a company with engineering teams releasing frequently and consistently. However, the software relied on data science models to make predictions and those models and the surrounding infrastructure to access these models was being released infrequently and inconsistently. The data science team was conducting a lot of research which was making it difficult to say when software or models would be released and this eventually affected the whole company’s ability to react quickly.
We decided to create a data science scrum team. Instead of introducing the most common sprint length of two weeks, we decided to go with weekly sprints as the team was more comfortable with a shorter time frame as this allowed them to change plans more frequently based on the result of research spikes. Additionally, they were more confident in their ability to plan for one week instead of two.
After a couple of months, the team became more accustomed to talking about priorities and deciding as a group on what to focus on and what can be accomplished within a week. At this point, we decided to move to two weeks sprints aligning them with other scrum teams and merging some engineering resources onto the team to make them more cross-functional.
One of the main benefits was that Agile encouraged the team to:
- build a deployment pipeline with unit tests and multiple environments instead of having ad hoc releases and processes
- document the release process
- move configuration out of code such that it could be updated without a new release.
More generally, they migrated from a research approach to a hybrid approach where some time was allocated for research spikes and some were allocated to software development each sprint. The product owner, who was well versed in data science, was able to review the results of the research spikes more frequently than before and decide if the team should continue spiking or if priorities had changed.
- Scrum processes can work for data science and other AI optimization teams. You need to provide teams with the ability to conduct research within these processes.
- You get a lot of ancillary benefits along with the transition as you bring more software engineering into data science teams
Shridharan Muthu, VP of Engineering at Zoosk, describes how he quickly agreed to adopt new workflows, a mistake he later regretted, and how he handled the situation by spending the time to course correct and taking a stab at making things easier for his team.
VP of Engineering, Backend Applications at Zoosk
Pierre Bergamin, VP of Engineering at Assignar, outlines some useful tips for decoupling releases from deployment and increasing deployments by a huge factor, speeding up reverts and planning releases in a better way.
VP of Engineering at Assignar
Ben Coats, Solutions Architect at InfoArmor, recalls how he had to adjust his personal working style of non-linear thinking and coding, to the shifting priorities and incremental delivery of Agile development.
Solutions Architect / Principal Engineer at InfoArmor
Stefan Gruber, VP of Engineering at Bitmovin, describes when he decided to introduce Scrum into his organization and the moment he realized that tech items were left off the priority list for engineering.
VP of Engineering at Bitmovin
Marc LeBrun, VP Engineering at Flow Kana, describes how he empowered teams to make decisions and uplevel their process by adopting a low-tech approach to agile practices.
VP Engineering at Flow Kana
You're a great engineer.
Become a great engineering leader.
Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.