Back to resources

Integrating Machine Learning with Human Knowledge

Innovation / Experiment
New Manager

15 September, 2021

Deepak Kumar Singh
Deepak Kumar Singh

Senior Product Manager at Yelp

Deepak Kumar Singh, Principal Product Manager at Omise, articulates how machine learning and human-in-the-loop can create synergistic results when integrated correctly.

Problem

When everything was running smoothly with the team, I got into a challenging situation where the company wanted to monetize its audience data. The company had merely acquired another company with some manpower and technical expertise, but the challenge posed was how do we scale up the operations? For instance, if today we were doing a certain amount of business, how do we grow 10 times more in 6 months? In such a situation, we could not go through the operations such as hiring 10 times more people to increase the output and revenue.

I had to think of something. The problem space was related to ad exchanges, where we had data about our customers. We knew what our customers wanted and what we wanted. All we had to do was get a better click-through rate. Our best bet was because they were shoppers, they came to our website to buy something. Despite it being my first job and first time landing into such a problem space, I did get plenty of help from my seniors and mentors. I had to scale up on the supply side as well as the operations side while delivering consistent quality results.

Actions taken

For the supply problem, I was just going around to see who the key suppliers we could proceed and integrate with. So it was a pretty straightforward move to go around and merge with some of the big ad exchanges while getting supplies.

During the current space, on the operations side, we had a very manual-driven team. We had some analysts who were responsible for managing and optimizing ongoing campaigns. It was pretty unscalable, where people had to go and check in every day in and out. On top of that, we could only run a certain amount of campaigns.

I talked to my seniors about this, letting them know that it was a solved problem; where Google or Facebook are already doing it, we just needed a machine learning-based solution. After that, I started digging in, trying to figure out if the current data that we had was enough to build an ML-based solution or not.

I researched and found out that the data we had was pretty decent and in a great format for us to use it. So, based on my initial understanding of ML and talking to my seniors about it, we pitched our ideas to the data science team to build a click prediction model. Luckily, the pitch went pretty well, and we were able to allocate a data science person who would make the model.

Coming to the stakeholders, they had feedback for us. Their point of view was that they tried to make it work in the past, but it did not work out. The company we had just merged with did try to build a predictive model, which they were not successful with. What they had to say was the manual approach was much more precise. I had to be cognizant of that. So, cleverly enough, once the machine learning model comes in, we would not immediately replace it; instead, we would experiment, show the results, and then prove it with action.

In that regard, we had pretty encouraging results from the offline model that the data scientists came up with. We did get double the times better click-through rates than what we would get from the analytics team. The best part was that it was all automated — we did not need a lot of operations effort to put into it.

Besides, to prove that it will work on actual data, I went around configuring an experiment. Finally, we uploaded the model online and used it as a shadow performance against what the analysts were doing. That went on for about a month, after which I was able to prove that our predictive model was performing much better than the analysts. However, it created another obvious problem: the analysts thought they would end up losing their jobs.

Our model did have some minor errors. For example, I noticed that even though we were getting a reasonable click-through prediction rate, sometimes the entire ad campaign was not burnt enough. For instance, if it were a 30-day campaign, it would not burn the whole impressions in those 30 days because it was optimized based on clicks. So we tried to develop a solution based on top of this model, which was to give these operations people a couple of tools that they could use and optimize the performance.

In this way, based on the ML model and the tool, we announced that we could successfully scale our operations up to 5 times more than before. It was also noteworthy how we made a breakthrough without any new hires and just with the same amount of time. Safe to say that my first brush with machine learning and utilizing it to scale up operations was a success.

Lessons learned

  • Since it was about 5 years ago, and I had a push back, I realized that you could not fight with opinions. Only data and proven results from experiments can do all the talking. Take bets, run suitable experiments, and demonstrate the value.
  • People can push back when anything comes in you and hear what we should identify is what value they can bring in for the machine. Try to use a synergistic approach with both humans and machines instead of assuming that machines will do all the work. Machine learning plus manual setting gives the optimal results.
  • Get a lot of help whenever and wherever you need it. In the corporate world, your seniors are always willing to help anyone out. All you have to do is gather the courage and go up to them to ask for it.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

The Art of Asking Why: Narrowing the Gap Between Customers and Users

24 May

Jord Sips, Senior Product Manager at Mews, shares his expertise on a common challenge for product managers – finding root causes and solutions.

Customers
Innovation / Experiment
Product
Personal Growth
Leadership
Stakeholders
Users
Jord Sips

Jord Sips

Senior Product Manager at Mews

Navigating Disagreements When It Comes to Priorities

9 May

Pavel Safarik, Head of Product at ROI Hunter, shares his insights on how to deal with disagreements about prioritization when building a product.

Innovation / Experiment
Product Team
Product
Dev Processes
Conflict Solving
Internal Communication
Collaboration
Convincing
Strategy
Prioritization
Pavel Safarik

Pavel Safarik

Head of Product at ROI Hunter

Why You Should Take Technology Risks in Product Development

25 April

Matias Pizarro, CTO and VP of Residents at ComunidadFeliz, recalls a time in his early career when he took a technology risk that had wide-ranging benefits to his product's user experience.

Innovation / Experiment
Product
Scaling Team
Dev Processes
Matias Pizarro

Matias Pizarro

CTO and VP of Residents at ComunidadFeliz

The Challenges of Becoming a New Manager Among Old Peers

22 April

Neha Saha, Manager, Software Development Engineering at Workday, illustrates the challenges of obtaining a position in management with no prior experience and the confidence it takes in order to succeed.

Leadership
New Manager
Neha Saha

Neha Saha

Manager, Software Development Engineering at Workday

Combining User Research with Testing to Build a Successful Product

20 April

Waldo Vanderhaeghen, VP of Product at 3YOURMIND, details an experience in product discovery, analyzing user research, and testing.

Innovation / Experiment
Product
Conflict Solving
Feedback
Users
Waldo Vanderhaeghen

Waldo Vanderhaeghen

VP Product at 3YOURMIND

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.