Applying Agile to Data Science Teams

Problem

I joined a company with engineering teams releasing frequently and consistently. However, the software relied on data science models to make predictions and those models and the surrounding infrastructure to access these models was being released infrequently and inconsistently. The data science team was conducting a lot of research which was making it difficult to say when software or models would be released and this eventually affected the whole company’s ability to react quickly.

Actions taken

We decided to create a data science scrum team. Instead of introducing the most common sprint length of two weeks, we decided to go with weekly sprints as the team was more comfortable with a shorter time frame as this allowed them to change plans more frequently based on the result of research spikes. Additionally, they were more confident in their ability to plan for one week instead of two.

After a couple of months, the team became more accustomed to talking about priorities and deciding as a group on what to focus on and what can be accomplished within a week. At this point, we decided to move to two weeks sprints aligning them with other scrum teams and merging some engineering resources onto the team to make them more cross-functional.

One of the main benefits was that Agile encouraged the team to:

• build a deployment pipeline with unit tests and multiple environments instead of having ad hoc releases and processes
• document the release process
• move configuration out of code such that it could be updated without a new release.
  

More generally, they migrated from a research approach to a hybrid approach where some time was allocated for research spikes and some were allocated to software development each sprint. The product owner, who was well versed in data science, was able to review the results of the research spikes more frequently than before and decide if the team should continue spiking or if priorities had changed.

Lessons learned

• Scrum processes can work for data science and other AI optimization teams. You need to provide teams with the ability to conduct research within these processes.
• You get a lot of ancillary benefits along with the transition as you bring more software engineering into data science teams