Back to resources

Getting Machine Learning Projects Unstuck

Impact
Productivity

27 April, 2021

Philippe Girolami
Philippe Girolami

VP of Engineer at Upflow

Phillipe Girolami, VP of Engineering for Data Engineering and Machine Learning at Dailymotion, speaks of non-engineering factors that determine the success rate of machine learning projects.

Problem

One of my teams does machine learning and we’ve had our share of stuck projects despite having some amazing ML experts on the team and alignment in the company around the need for ML. Many factors determine the success rate of ML projects that are non-engineering in their essence.

Actions taken

The first way one of the ML projects got stuck was by a lack of discussions with Product. If a few key questions aren’t addressed upfront, people weave a net of assumptions and won’t dissect the consequences of choosing one metric or solution over others.

So the first action to take is to talk to the PM for whom we would be delivering features and start asking questions. Here are some arising from our cases; there are surely others that apply to your case:

  • What constraints apply to the product the ML project is for? There are always constraints you can’t wiggle out, whether they are legal, business, or product constraints. If you throw data at a computer, it will find the shortest path, which is not always acceptable. What is the success metric? Everyone should be crystal clear about what we are trying to improve or optimize.
  • What mistakes and error rates are acceptable because an ML model will always make mistakes, and what to do about them?

Once everyone agrees on success metrics, engineers could use them directly, applying them to how they would train the data and measure how the model works, or they would have to find a proxy for it. That would initiate a number of other discussions, but at least nothing would be taken for granted.

Responses about acceptable errors would shape up the kind of solution we would come up with. That would provide engineers with an understanding of tradeoffs and possible alternative solutions. In fact, if something is exceedingly important and errors are not to be tolerated, maybe machine learning would not be the right solution in the first place, and instead, a human should be the one to do it. In some cases, we could accept false positives (something is wrong, but in fact, it isn’t), but not false negatives (everything is fine, but it isn’t). Obviously, no one would accept the same kind of errors if looking at chest X-rays and a recommendation system.

The next step would be to understand how we should remediate mistakes that are not acceptable. There are three most common scenarios: we would reduce them enough to drop them, add a human in the loop, or build the second model. These discussions are often missing, and that is the main cause for projects to get stuck.

The second example of a stuck ML project simply required clarifying the different roles of people involved in the project. Machine learning is a new field, and there can be a lot of confusion about roles and responsibilities. In this particular case, there was confusion about the role of ML engineer as an expert building the model, product analyst, data engineer, and PM.

Lessons learned

  • There is nothing obvious with delivering ML capability in production. It takes learning from anyone involved. You will most likely trip over something but have to learn how to get up and adapt. Discussing constraints is part of that learning and differs for every single project.
  • Be very clear about what you expect different people on the ML team to do. Demarcate clearly the responsibilities of an ML engineer and data scientist/analyst. These two roles are somewhat fuzzy and what they encompass pretty much depends on the company.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

Managing Remotely: Balancing Team Cohesion and Focus Time

26 May

Jonathan Belcher, Engineering Manager at Curative, explains how to balance team cohesion and individual focus time, tapping into his experiences of working remotely for seven years.

Remote
Micromanagement
Meetings
Internal Communication
Productivity
Psychological Safety
Performance
Jonathan Belcher

Jonathan Belcher

Engineering Manager - Patient Experience at Curative

Here to Make a Recognizable Difference: How to Develop Teams

5 May

Eric Merritt, VP of Engineering at Whitepages.com, divulges on the many complexities of developing teams in management by solving problems according to their needs, and empowering teams.

Leadership
Impact
Sharing The Vision
Coaching / Training / Mentorship
Eric Merritt

Eric Merritt

VP of Engineering at Whitepages.com

Balancing Technical Debt Innovation: How Roadmaps for Development Help Your Company Succeed

4 May

Brad Jayakody outlines the roadmap to maintaining a healthy balance between technical debt and team growth. However, just as balancing acts go it is important to have a strong foundation.

Alignment
Leadership
Impact
Roadmap
Tech Debt
Career Path
Brad Jayakody

Brad Jayakody

Director of Engineering at Motorway

The Necessary Structures of Time Management

14 April

Suryakant Mutnal, Engineering Manager at PayPal, discusses the importance of time management and the necessary structures in order to create internal consistency.

Goal Setting
Managing Expectations
Remote
Deadlines
Productivity
Roadmap
Prioritization
Performance
Suryakant Mutnal

Suryakant Mutnal

Engineering manager at PayPal

Why Documentation Is the Key to Success

6 April

Henning Muszynski, Head of Frontend at Doist, promotes his ideas on how documentation ensures consistency, efficiency, and standardization.

Alignment
Collaboration
Productivity
Hiring
Team Processes
Henning Muszynski

Henning Muszynski

Head of Frontend at Doist

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.