The Importance of Empowering Data Engineers to See a Company’s Bigger Priorities

Problem

"I started seeing a pattern from the data scientists and students that I work with on 'artificial intelligence' or 'machine learning' projects, which is that the better and more experienced an engineer is, the more tools he has to work with. Because they're also so passionate about their knowledge and their work, they get excited to apply these tools whenever the chance arises. So when the engineer is tasked with creating an AI product for a business, they often spend time perfecting this technical engineering product, trying many advanced tools, without having perspective into the company's problem this will ultimately be solving. This leads to engineers spending a lot of time and energy to select and configure frameworks before they even spent time with the business problem."

This is a huge inefficiency problem for the company because sometimes when the machine learning product is finally delivered it doesn't actually work plus there is no progress until very late. I think this is because most of the product managers treat machine learning differently than other engineering projects. They normally work very agile with developers, by separating the project into smaller tasks and giving background on the tasks that are being assigned. But when it comes to machine learning, very often the manager lets the scientist work however he wants and isn't able to challenge him or to break the problem down.

Actions taken

"From this trend I've noticed, I try to teach this lesson to students at university, to product managers and to ML engineers. In an exercise, I forbid them to use any fancy frameworks, solutions, or algorithms. I first make them spend time with the underlying business problem with no computer, only a sheet of paper and a pen. I ask them for the most basic solution they can come up with, like approximating, guessing or brute forcing. Think of this model after the brain teasers you hear about in Google interviews. I do this because I want them to think of a first quick solution for which no advanced engineering tools are required. I then ask for a way to measure how good or bad their solution is. Then they can start improving and measuring improvements. These steps in the exercise take maybe a few hours."

Lessons learned

"The outcome for students/product managers/ML engineers is to see that in a very short time, by thinking and implying domain knowledge, they can come up with ways to quickly improve a simple version and manage to get early on into a pretty well working ballpark where a solution is already usable, depending what the task is. Having them see that starting with an almost primitive or naive solution instead of immediately applying complex tools help them to focus on the bigger picture of what their solution will be doing and where it will fit into the environment. Also thinking of a way to measure the quality helps to figure out on what actually is important. Based on that they can improve or increase the complexity of their solution. Meanwhile, the rest of the world or company has already something they can use!"