Back to resources

Integrating Machine Learning with Human Knowledge

Engineering Processes
Transitioning into a New Management Role
Working with Product Teams

15 September, 2021

Deepak Kumar Singh
Deepak Kumar Singh

Senior Product Manager at Yelp

Deepak Kumar Singh, Principal Product Manager at Omise, articulates how machine learning and human-in-the-loop can create synergistic results when integrated correctly.


When everything was running smoothly with the team, I got into a challenging situation where the company wanted to monetize its audience data. The company had merely acquired another company with some manpower and technical expertise, but the challenge posed was how do we scale up the operations? For instance, if today we were doing a certain amount of business, how do we grow 10 times more in 6 months? In such a situation, we could not go through the operations such as hiring 10 times more people to increase the output and revenue.

I had to think of something. The problem space was related to ad exchanges, where we had data about our customers. We knew what our customers wanted and what we wanted. All we had to do was get a better click-through rate. Our best bet was because they were shoppers, they came to our website to buy something. Despite it being my first job and first time landing into such a problem space, I did get plenty of help from my seniors and mentors. I had to scale up on the supply side as well as the operations side while delivering consistent quality results.

Actions taken

For the supply problem, I was just going around to see who the key suppliers we could proceed and integrate with. So it was a pretty straightforward move to go around and merge with some of the big ad exchanges while getting supplies.

During the current space, on the operations side, we had a very manual-driven team. We had some analysts who were responsible for managing and optimizing ongoing campaigns. It was pretty unscalable, where people had to go and check in every day in and out. On top of that, we could only run a certain amount of campaigns.

I talked to my seniors about this, letting them know that it was a solved problem; where Google or Facebook are already doing it, we just needed a machine learning-based solution. After that, I started digging in, trying to figure out if the current data that we had was enough to build an ML-based solution or not.

I researched and found out that the data we had was pretty decent and in a great format for us to use it. So, based on my initial understanding of ML and talking to my seniors about it, we pitched our ideas to the data science team to build a click prediction model. Luckily, the pitch went pretty well, and we were able to allocate a data science person who would make the model.

Coming to the stakeholders, they had feedback for us. Their point of view was that they tried to make it work in the past, but it did not work out. The company we had just merged with did try to build a predictive model, which they were not successful with. What they had to say was the manual approach was much more precise. I had to be cognizant of that. So, cleverly enough, once the machine learning model comes in, we would not immediately replace it; instead, we would experiment, show the results, and then prove it with action.

In that regard, we had pretty encouraging results from the offline model that the data scientists came up with. We did get double the times better click-through rates than what we would get from the analytics team. The best part was that it was all automated — we did not need a lot of operations effort to put into it.

Besides, to prove that it will work on actual data, I went around configuring an experiment. Finally, we uploaded the model online and used it as a shadow performance against what the analysts were doing. That went on for about a month, after which I was able to prove that our predictive model was performing much better than the analysts. However, it created another obvious problem: the analysts thought they would end up losing their jobs.

Our model did have some minor errors. For example, I noticed that even though we were getting a reasonable click-through prediction rate, sometimes the entire ad campaign was not burnt enough. For instance, if it were a 30-day campaign, it would not burn the whole impressions in those 30 days because it was optimized based on clicks. So we tried to develop a solution based on top of this model, which was to give these operations people a couple of tools that they could use and optimize the performance.

In this way, based on the ML model and the tool, we announced that we could successfully scale our operations up to 5 times more than before. It was also noteworthy how we made a breakthrough without any new hires and just with the same amount of time. Safe to say that my first brush with machine learning and utilizing it to scale up operations was a success.

Lessons learned

  • Since it was about 5 years ago, and I had a push back, I realized that you could not fight with opinions. Only data and proven results from experiments can do all the talking. Take bets, run suitable experiments, and demonstrate the value.
  • People can push back when anything comes in you and hear what we should identify is what value they can bring in for the machine. Try to use a synergistic approach with both humans and machines instead of assuming that machines will do all the work. Machine learning plus manual setting gives the optimal results.
  • Get a lot of help whenever and wherever you need it. In the corporate world, your seniors are always willing to help anyone out. All you have to do is gather the courage and go up to them to ask for it.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader

Related stories

My experience shadowing an Engineering Director for a week

28 February

Recently I had the opportunity to remotely job shadow another Engineering Director and Mentor from Plato for a week. The article describes how the week unfolded and some of the stats and my key observations from the shadow program.

Transitioning into a New Management Role
Working and Leading Remote Teams
Mrunal Kapade

Mrunal Kapade

Director of Engineering at Inspire Energy

People, Process & Product

5 February

As an Engineering Manager, what to focus on more? People or Process or Product?

Transitioning into a New Management Role
Kamal Raj Guptha R

Kamal Raj Guptha R

Engineering Manager at Jeavio

"You don't care about quality" A story of single metric bias

3 February

This was not a high point in my career. It's a story of single metric bias, how I let one measure become a 'source of truth', failed to manage up and ended up yelling at one of the most respected engineers in my team.

Conflict Resolution
Working with Product Teams
Alex Shaw

Alex Shaw

Chief Technology and Product Officer at Hive Learning

15 Tips to improve profile discovery on LI

20 January

Recently, I have read the book ‘Linked’ from Omar Garriott and Jeremy Schifeling on audible. The audio book is 7 hours long. If you dont’ have time or need a brief summary, read on

Career Growth
Transitioning into a New Management Role
Hiring, Retaining, or Firing
Praveen Cheruvu

Praveen Cheruvu

Senior Software Engineering Manager at Anaplan

V2 infrastructure project

21 December

Consideration for starting a multi year software infrastructure ( V2 ) project that involves hundreds of globally distributed engineers.

Stakeholder Management
Engineering Processes
Ahsan Habib

Ahsan Habib

VP Software Engineering at human