Teams Facing Common Machine Learning Problems

Problem

It is no surprise that project teams are likely to face the same kind of technical challenges. Similarly, we had several teams facing the same problems on the machine learning side. Our time taken to become life was way more than expected, and it was heavily impacting our deliverables. Initially, we did come up with solutions to fix it in our team, but we realized that many other groups might have faced the same problems. There were plenty of different machine learning teams in the company.

I understood that in order to take our product from level 1 to the next level, I needed more funding and people to work on it. However, it was not the best business case where I could go and ask our CEO or COO to allow me to hire more developers. As per them, it was not the biggest win in terms of the silo. The entire situation was a little twisted because we had a product and a potential, but we did not have a collaborative effort. It was not about hiring some developers or engineers to get the job done; instead, it was about moving the needle.

Actions taken

To begin with, what I did was pretty straightforward and easy: I identified all the teams who might have been using machine learning. I went around to have smaller group discussions with them to recollect the problems they were facing and how they dealt with them. Based on the discussion, I went back to them, talking about what I had currently and if adding a set of features we're going to solve the problem in a big way.

I pitched them the idea that they were not the only teams facing the problem; multiple teams across the organization were facing the same issues. To add to that, if we combined the resources, we would have come up with a quicker solution; instead of having to wait 3 quarters, we could get it done in 1 quarter.

Based on those individuals, the benefits that people were getting, ultimately, we could pull in resources every quarter from other teams that helped us lead to the stage that the product was at a later stage. We knew that if we were working only with our team — with the 2 - 3 developers that we could salvage — it would never be done in less than 2 quarters.

Being involved with work related to machine learning product management, I was heavily involved in that area. I believed in how the machine learning models and their roadmaps worked. Looking at this space, I recognized that the problem was around how the data science team had done the analysis and how we were supposed to put the model online.

While they used offline data points to create the model, it did not work for us in real-time. When we were trying to re-create the model in a production environment, we looked at the real-time events to create the same features that they had. After that, we matched the same features, working in the same way for offline and online. We ended up creating something that we named as the features store, the idea of which was to help the data science team to read and write from there. There was no time lag between reading and writing.

Furthermore, every time they came up with a new model, we ran some minor tests, after which it was ready for production. That cut down our time from 8 weeks to 1 week. Undoubtedly, this was one of the best ways our team had made their biggest win. Theoretically, if they wanted to run 90 tests, they were more than welcome to do so.

Lessons learned

• Try to proceed one step at a time. Take time, discover the problem, involve other teams (if needed), and then move systematically instead of rushing the process. Combine the synergies and extend the feature store to a feature platform.
• When the machine learning models do not perform accordingly, look for solutions. In our case, we were able to put an audit on top of all the data that was being written and read. That informed us about any problems, beforehand, about any of the models.
• If we combine many forces, we would not be doing much redundant work in our silos. For instance, we build something, the other team builds something else, and another team makes another feature, and in the end, it would be seen that we were all building similar features. Instead, combine your vision and resources and build one concrete structure to support everyone in the future.