Google Sign inLinkedIn Sign in

Don't have an account? 

Applying Agile to Data Science Teams

Agile / Scrum
Team processes
Data team

31 March, 2020

Jason Fama, VP of Engineering and Head of Innovation Lab, shares his experience of introducing Agile to data science teams emphasizing some of the benefits it brings.


I joined a company with engineering teams releasing frequently and consistently. However, the software relied on data science models to make predictions and those models and the surrounding infrastructure to access these models was being released infrequently and inconsistently. The data science team was conducting a lot of research which was making it difficult to say when software or models would be released and this eventually affected the whole company’s ability to react quickly.

Actions taken

We decided to create a data science scrum team. Instead of introducing the most common sprint length of two weeks, we decided to go with weekly sprints as the team was more comfortable with a shorter time frame as this allowed them to change plans more frequently based on the result of research spikes. Additionally, they were more confident in their ability to plan for one week instead of two.

After a couple of months, the team became more accustomed to talking about priorities and deciding as a group on what to focus on and what can be accomplished within a week. At this point, we decided to move to two weeks sprints aligning them with other scrum teams and merging some engineering resources onto the team to make them more cross-functional.

One of the main benefits was that Agile encouraged the team to:

  • build a deployment pipeline with unit tests and multiple environments instead of having ad hoc releases and processes
  • document the release process
  • move configuration out of code such that it could be updated without a new release.

More generally, they migrated from a research approach to a hybrid approach where some time was allocated for research spikes and some were allocated to software development each sprint. The product owner, who was well versed in data science, was able to review the results of the research spikes more frequently than before and decide if the team should continue spiking or if priorities had changed.

Lessons learned

  • Scrum processes can work for data science and other AI optimization teams. You need to provide teams with the ability to conduct research within these processes.
  • You get a lot of ancillary benefits along with the transition as you bring more software engineering into data science teams

Related stories

Handling a Mistake - Adopting a New Workflow
6 July

Shridharan Muthu, VP of Engineering at Zoosk, describes how he quickly agreed to adopt new workflows, a mistake he later regretted, and how he handled the situation by spending the time to course correct and taking a stab at making things easier for his team.

Team processes
Agile / Scrum
Shridharan Muthu

Shridharan Muthu

VP of Engineering, Backend Applications at Zoosk

Some Useful Tips for Decoupling Releases and Deployments
30 June

Pierre Bergamin, VP of Engineering at Assignar, outlines some useful tips for decoupling releases from deployment and increasing deployments by a huge factor, speeding up reverts and planning releases in a better way.

Agile / Scrum
Dev Processes
Pierre Bergamin

Pierre Bergamin

VP of Engineering at Assignar

Adjusting Your Working Style to Agile Practices and Mindset
15 June

Ben Coats, Solutions Architect at InfoArmor, recalls how he had to adjust his personal working style of non-linear thinking and coding, to the shifting priorities and incremental delivery of Agile development.

Agile / Scrum
Ben Coats

Ben Coats

Solutions Architect / Principal Engineer at InfoArmor

Prioritizing Technical Matters Within Scrum
11 June

Stefan Gruber, VP of Engineering at Bitmovin, describes when he decided to introduce Scrum into his organization and the moment he realized that tech items were left off the priority list for engineering.

Agile / Scrum
Stefan Gruber

Stefan Gruber

VP of Engineering at Bitmovin

Going Low-Tech to Uplevel Agile Practices
24 May

Marc LeBrun, VP Engineering at Flow Kana, describes how he empowered teams to make decisions and uplevel their process by adopting a low-tech approach to agile practices.

Team reaction
Agile / Scrum
Marc LeBrun

Marc LeBrun

VP Engineering at Flow Kana

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.