Back to resources

Teams Facing Common Machine Learning Problems

Data Team

24 September, 2021

Deepak Kumar Singh
Deepak Kumar Singh

Senior Product Manager at Yelp

Deepak Kumar Singh, Principal Product Manager at Omise, talks about combining different teams to solve one common problem and saving processing time.


It is no surprise that project teams are likely to face the same kind of technical challenges. Similarly, we had several teams facing the same problems on the machine learning side. Our time taken to become life was way more than expected, and it was heavily impacting our deliverables. Initially, we did come up with solutions to fix it in our team, but we realized that many other groups might have faced the same problems. There were plenty of different machine learning teams in the company.

I understood that in order to take our product from level 1 to the next level, I needed more funding and people to work on it. However, it was not the best business case where I could go and ask our CEO or COO to allow me to hire more developers. As per them, it was not the biggest win in terms of the silo. The entire situation was a little twisted because we had a product and a potential, but we did not have a collaborative effort. It was not about hiring some developers or engineers to get the job done; instead, it was about moving the needle.

Actions taken

To begin with, what I did was pretty straightforward and easy: I identified all the teams who might have been using machine learning. I went around to have smaller group discussions with them to recollect the problems they were facing and how they dealt with them. Based on the discussion, I went back to them, talking about what I had currently and if adding a set of features we're going to solve the problem in a big way.

I pitched them the idea that they were not the only teams facing the problem; multiple teams across the organization were facing the same issues. To add to that, if we combined the resources, we would have come up with a quicker solution; instead of having to wait 3 quarters, we could get it done in 1 quarter.

Based on those individuals, the benefits that people were getting, ultimately, we could pull in resources every quarter from other teams that helped us lead to the stage that the product was at a later stage. We knew that if we were working only with our team — with the 2 - 3 developers that we could salvage — it would never be done in less than 2 quarters.

Being involved with work related to machine learning product management, I was heavily involved in that area. I believed in how the machine learning models and their roadmaps worked. Looking at this space, I recognized that the problem was around how the data science team had done the analysis and how we were supposed to put the model online.

While they used offline data points to create the model, it did not work for us in real-time. When we were trying to re-create the model in a production environment, we looked at the real-time events to create the same features that they had. After that, we matched the same features, working in the same way for offline and online. We ended up creating something that we named as the features store, the idea of which was to help the data science team to read and write from there. There was no time lag between reading and writing.

Furthermore, every time they came up with a new model, we ran some minor tests, after which it was ready for production. That cut down our time from 8 weeks to 1 week. Undoubtedly, this was one of the best ways our team had made their biggest win. Theoretically, if they wanted to run 90 tests, they were more than welcome to do so.

Lessons learned

  • Try to proceed one step at a time. Take time, discover the problem, involve other teams (if needed), and then move systematically instead of rushing the process. Combine the synergies and extend the feature store to a feature platform.
  • When the machine learning models do not perform accordingly, look for solutions. In our case, we were able to put an audit on top of all the data that was being written and read. That informed us about any problems, beforehand, about any of the models.
  • If we combine many forces, we would not be doing much redundant work in our silos. For instance, we build something, the other team builds something else, and another team makes another feature, and in the end, it would be seen that we were all building similar features. Instead, combine your vision and resources and build one concrete structure to support everyone in the future.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader

Related stories

Guiding Your Team Through a Data-Driven Transformation

2 June

Philip Gollucci, Director of Cloud Engineering at CareRev, shares his tips for managing people amidst a significant data-driven transformation.

Innovation / Experiment
Changing A Company
Sharing The Vision
Data Team
Philip Gollucci

Philip Gollucci

CEO/Founder at P6M7G8 Inc.

Hiring a Data Team With a Stubborn Manager

24 May

Liz Henderson, an Executive consultant at Capgemini, shares her experience hiring a data team with a manager who was difficult to work with.

Managing Up
Building A Team
Conflict Solving
Data Team
Liz Henderson

Liz Henderson

Executive Advisor at Capgemini

Managing Customer Expectations in the Age of AI

8 February

Adhiguna Mahendra, Chief AI and Product at nodeflux, discusses how to embed an explainable AI system for a better justifiable logical reasoning behind what caused something to happen

Managing Expectations
Dev Processes
Data Team
Adhiguna Mahendra

Adhiguna Mahendra

Chief AI and Product at nodeflux

Preparing Your Team for the Remote Workplace

29 November

Vadim Antonov, Engineering Manager at Meta, dictates how he brought a brand new team into the remote learning process by ramping up onboarding and creating a mentor system.

Internal Communication
Coaching / Training / Mentorship
Data Team
Cross-Functional Collaboration
Vadim Antonov

Vadim Antonov

Engineering Manager at Meta (Facebook, Oculus, & Family of Apps)

How Data-Driven Products Help Customers and Increase Sales

11 November

Richard Maraschi, VP of Data Products & Insights at WarnerMedia, shares his insight on incorporating data science, AI, and product management to overcome slowing growth of the company.

Conflict Solving
Data Team
Richard Maraschi

Richard Maraschi

VP Data Product Management at WarnerMedia