Back to resources

Applying Agile to Data Science Teams

Team Processes
Data Team
Agile / Scrum

31 March, 2020

Jason Fama
Jason Fama

SVP of Product and Engineering at Talkdesk

Jason Fama, VP of Engineering and Head of Innovation Lab, shares his experience of introducing Agile to data science teams emphasizing some of the benefits it brings.

Problem

I joined a company with engineering teams releasing frequently and consistently. However, the software relied on data science models to make predictions and those models and the surrounding infrastructure to access these models was being released infrequently and inconsistently. The data science team was conducting a lot of research which was making it difficult to say when software or models would be released and this eventually affected the whole company’s ability to react quickly.

Actions taken

We decided to create a data science scrum team. Instead of introducing the most common sprint length of two weeks, we decided to go with weekly sprints as the team was more comfortable with a shorter time frame as this allowed them to change plans more frequently based on the result of research spikes. Additionally, they were more confident in their ability to plan for one week instead of two.

After a couple of months, the team became more accustomed to talking about priorities and deciding as a group on what to focus on and what can be accomplished within a week. At this point, we decided to move to two weeks sprints aligning them with other scrum teams and merging some engineering resources onto the team to make them more cross-functional.

One of the main benefits was that Agile encouraged the team to:

  • build a deployment pipeline with unit tests and multiple environments instead of having ad hoc releases and processes
  • document the release process
  • move configuration out of code such that it could be updated without a new release.

More generally, they migrated from a research approach to a hybrid approach where some time was allocated for research spikes and some were allocated to software development each sprint. The product owner, who was well versed in data science, was able to review the results of the research spikes more frequently than before and decide if the team should continue spiking or if priorities had changed.

Lessons learned

  • Scrum processes can work for data science and other AI optimization teams. You need to provide teams with the ability to conduct research within these processes.
  • You get a lot of ancillary benefits along with the transition as you bring more software engineering into data science teams

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

Hiring a Data Team With a Stubborn Manager

24 May

Liz Henderson, an Executive consultant at Capgemini, shares her experience hiring a data team with a manager who was difficult to work with.

Managing Up
Building A Team
Conflict Solving
Hiring
Data Team
Liz Henderson

Liz Henderson

Executive consultant at Capgemini

Streamlining Product Processes After a Reorganization

16 May

Snehal Shaha, Lead Technical Program Manager at Momentive (fka SurveyMonkey), details her short-term technical strategy to unify processes among teams following an acquisition.

Acquisition / Integration
Product Team
Product
Building A Team
Leadership
Internal Communication
Collaboration
Reorganization
Strategy
Team Processes
Cross-Functional Collaboration
Snehal Shaha

Snehal Shaha

Senior EPM/TPM at Apple Inc.

The Optimization and Organization of Large Scale Demand

4 May

Kamal Qadri, Senior Manager at FICO, drives the importance of setting expectations when optimizing large-scale requirements.

Managing Expectations
Delegate
Team Processes
Prioritization
Kamal Qadri

Kamal Qadri

Head of Software Quality Assurance at FICO

Why Documentation Is the Key to Success

6 April

Henning Muszynski, Head of Frontend at Doist, promotes his ideas on how documentation ensures consistency, efficiency, and standardization.

Alignment
Collaboration
Productivity
Hiring
Team Processes
Henning Muszynski

Henning Muszynski

Head of Frontend at Doist

It's Time to Say 'No' to Manual Business Processes

6 April

Henning Muszynski, Head of Frontend at Doist, talks about the cost of slow and arduous processes that add up over time and how to bring the changes systematically.

Changing A Company
Conflict Solving
Internal Communication
Feedback
Team Processes
Henning Muszynski

Henning Muszynski

Head of Frontend at Doist

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.