Back to resources

Integrating Machine Learning with Human Knowledge

Innovation / Experiment
New Manager

15 September, 2021

Deepak Kumar Singh
Deepak Kumar Singh

Senior Product Manager at Yelp

Deepak Kumar Singh, Principal Product Manager at Omise, articulates how machine learning and human-in-the-loop can create synergistic results when integrated correctly.


When everything was running smoothly with the team, I got into a challenging situation where the company wanted to monetize its audience data. The company had merely acquired another company with some manpower and technical expertise, but the challenge posed was how do we scale up the operations? For instance, if today we were doing a certain amount of business, how do we grow 10 times more in 6 months? In such a situation, we could not go through the operations such as hiring 10 times more people to increase the output and revenue.

I had to think of something. The problem space was related to ad exchanges, where we had data about our customers. We knew what our customers wanted and what we wanted. All we had to do was get a better click-through rate. Our best bet was because they were shoppers, they came to our website to buy something. Despite it being my first job and first time landing into such a problem space, I did get plenty of help from my seniors and mentors. I had to scale up on the supply side as well as the operations side while delivering consistent quality results.

Actions taken

For the supply problem, I was just going around to see who the key suppliers we could proceed and integrate with. So it was a pretty straightforward move to go around and merge with some of the big ad exchanges while getting supplies.

During the current space, on the operations side, we had a very manual-driven team. We had some analysts who were responsible for managing and optimizing ongoing campaigns. It was pretty unscalable, where people had to go and check in every day in and out. On top of that, we could only run a certain amount of campaigns.

I talked to my seniors about this, letting them know that it was a solved problem; where Google or Facebook are already doing it, we just needed a machine learning-based solution. After that, I started digging in, trying to figure out if the current data that we had was enough to build an ML-based solution or not.

I researched and found out that the data we had was pretty decent and in a great format for us to use it. So, based on my initial understanding of ML and talking to my seniors about it, we pitched our ideas to the data science team to build a click prediction model. Luckily, the pitch went pretty well, and we were able to allocate a data science person who would make the model.

Coming to the stakeholders, they had feedback for us. Their point of view was that they tried to make it work in the past, but it did not work out. The company we had just merged with did try to build a predictive model, which they were not successful with. What they had to say was the manual approach was much more precise. I had to be cognizant of that. So, cleverly enough, once the machine learning model comes in, we would not immediately replace it; instead, we would experiment, show the results, and then prove it with action.

In that regard, we had pretty encouraging results from the offline model that the data scientists came up with. We did get double the times better click-through rates than what we would get from the analytics team. The best part was that it was all automated — we did not need a lot of operations effort to put into it.

Besides, to prove that it will work on actual data, I went around configuring an experiment. Finally, we uploaded the model online and used it as a shadow performance against what the analysts were doing. That went on for about a month, after which I was able to prove that our predictive model was performing much better than the analysts. However, it created another obvious problem: the analysts thought they would end up losing their jobs.

Our model did have some minor errors. For example, I noticed that even though we were getting a reasonable click-through prediction rate, sometimes the entire ad campaign was not burnt enough. For instance, if it were a 30-day campaign, it would not burn the whole impressions in those 30 days because it was optimized based on clicks. So we tried to develop a solution based on top of this model, which was to give these operations people a couple of tools that they could use and optimize the performance.

In this way, based on the ML model and the tool, we announced that we could successfully scale our operations up to 5 times more than before. It was also noteworthy how we made a breakthrough without any new hires and just with the same amount of time. Safe to say that my first brush with machine learning and utilizing it to scale up operations was a success.

Lessons learned

  • Since it was about 5 years ago, and I had a push back, I realized that you could not fight with opinions. Only data and proven results from experiments can do all the talking. Take bets, run suitable experiments, and demonstrate the value.
  • People can push back when anything comes in you and hear what we should identify is what value they can bring in for the machine. Try to use a synergistic approach with both humans and machines instead of assuming that machines will do all the work. Machine learning plus manual setting gives the optimal results.
  • Get a lot of help whenever and wherever you need it. In the corporate world, your seniors are always willing to help anyone out. All you have to do is gather the courage and go up to them to ask for it.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader

Related stories

Congratulations you're an Engineering Manager! Now What?

29 July

Congratulations, you have just been promoted to an engineering management role. Once you are done celebrating the promotion you have worked hard to earn you might start to ask yourself, now what do I do?

New Manager
AJ St. Aubin

AJ St. Aubin

Director Software Engineering at The RepTrak Company

Team Development Framework for new managers

26 June

Individual Contributors are familiar with a technical development framework that helps them with building products. Managers, especially new managers can leverage a parallel framework to help them build their teams while drawing analogies from an already familiar framework.

Building A Team
Team Processes
New Manager
Viswa Mani Kiran Peddinti

Viswa Mani Kiran Peddinti

Sr Engineering Manager at Instacart

Bootstrapping a Startup While Working Full-Time

23 June

Lucjan Suski, CEO & Co-founder of Surfer, relates how he started a company as a side project and shares his insights on bootstrapping tech startups.

Innovation / Experiment
Lucjan Suski

Lucjan Suski

Co-founder, formerly CTO and CEO at Surfer

Deciding if Management Is Right for You

13 June

Deekshita Amaravadi, Engineering Manager at Justworks, talks about the intricacies of making the decision to switch from an IC role to a management one.

Personal Growth
Career Path
New Manager
Deekshita Amaravadi

Deekshita Amaravadi

Engineering Manager at Justworks

Guiding Your Team Through a Data-Driven Transformation

2 June

Philip Gollucci, Director of Cloud Engineering at CareRev, shares his tips for managing people amidst a significant data-driven transformation.

Innovation / Experiment
Changing A Company
Sharing The Vision
Data Team
Philip Gollucci

Philip Gollucci

CEO/Founder at P6M7G8 Inc.