Integrating Machine Learning with Human Knowledge

Problem

When everything was running smoothly with the team, I got into a challenging situation where the company wanted to monetize its audience data. The company had merely acquired another company with some manpower and technical expertise, but the challenge posed was how do we scale up the operations? For instance, if today we were doing a certain amount of business, how do we grow 10 times more in 6 months? In such a situation, we could not go through the operations such as hiring 10 times more people to increase the output and revenue.

I had to think of something. The problem space was related to ad exchanges, where we had data about our customers. We knew what our customers wanted and what we wanted. All we had to do was get a better click-through rate. Our best bet was because they were shoppers, they came to our website to buy something. Despite it being my first job and first time landing into such a problem space, I did get plenty of help from my seniors and mentors. I had to scale up on the supply side as well as the operations side while delivering consistent quality results.

Actions taken

For the supply problem, I was just going around to see who the key suppliers we could proceed and integrate with. So it was a pretty straightforward move to go around and merge with some of the big ad exchanges while getting supplies.

During the current space, on the operations side, we had a very manual-driven team. We had some analysts who were responsible for managing and optimizing ongoing campaigns. It was pretty unscalable, where people had to go and check in every day in and out. On top of that, we could only run a certain amount of campaigns.

I talked to my seniors about this, letting them know that it was a solved problem; where Google or Facebook are already doing it, we just needed a machine learning-based solution. After that, I started digging in, trying to figure out if the current data that we had was enough to build an ML-based solution or not.

I researched and found out that the data we had was pretty decent and in a great format for us to use it. So, based on my initial understanding of ML and talking to my seniors about it, we pitched our ideas to the data science team to build a click prediction model. Luckily, the pitch went pretty well, and we were able to allocate a data science person who would make the model.

Coming to the stakeholders, they had feedback for us. Their point of view was that they tried to make it work in the past, but it did not work out. The company we had just merged with did try to build a predictive model, which they were not successful with. What they had to say was the manual approach was much more precise. I had to be cognizant of that. So, cleverly enough, once the machine learning model comes in, we would not immediately replace it; instead, we would experiment, show the results, and then prove it with action.

In that regard, we had pretty encouraging results from the offline model that the data scientists came up with. We did get double the times better click-through rates than what we would get from the analytics team. The best part was that it was all automated — we did not need a lot of operations effort to put into it.

Besides, to prove that it will work on actual data, I went around configuring an experiment. Finally, we uploaded the model online and used it as a shadow performance against what the analysts were doing. That went on for about a month, after which I was able to prove that our predictive model was performing much better than the analysts. However, it created another obvious problem: the analysts thought they would end up losing their jobs.

Our model did have some minor errors. For example, I noticed that even though we were getting a reasonable click-through prediction rate, sometimes the entire ad campaign was not burnt enough. For instance, if it were a 30-day campaign, it would not burn the whole impressions in those 30 days because it was optimized based on clicks. So we tried to develop a solution based on top of this model, which was to give these operations people a couple of tools that they could use and optimize the performance.

In this way, based on the ML model and the tool, we announced that we could successfully scale our operations up to 5 times more than before. It was also noteworthy how we made a breakthrough without any new hires and just with the same amount of time. Safe to say that my first brush with machine learning and utilizing it to scale up operations was a success.

Lessons learned

• Since it was about 5 years ago, and I had a push back, I realized that you could not fight with opinions. Only data and proven results from experiments can do all the talking. Take bets, run suitable experiments, and demonstrate the value.
• People can push back when anything comes in you and hear what we should identify is what value they can bring in for the machine. Try to use a synergistic approach with both humans and machines instead of assuming that machines will do all the work. Machine learning plus manual setting gives the optimal results.
• Get a lot of help whenever and wherever you need it. In the corporate world, your seniors are always willing to help anyone out. All you have to do is gather the courage and go up to them to ask for it.