Why Standardizing the Interview Process

Problem

For a while, we were concerned with our interview process. We were continuously missing our hiring targets because we were not getting the signal that we needed during the interview. We decided to re-evaluate our whole hiring process from sourcing to the evaluation of candidates and determine where the gaps were. We particularly focused on standardizing our interview questions as we noticed that randomness and inconsistency in that area affected significantly the quality of our hires.

Actions taken

Before we tackled the interview questions, we wanted to make sure that we were getting enough of the right candidates. Therefore, we reevaluated the messaging recruiters were using along with diversifying our sourcing. We worked with recruiting on tailoring the job descriptions to the specific skill sets that we were interested in. In this particular case, we were focused on machine learning and experimented with the phrasing of the requirements that were causing people to not apply. We relied extensively on feedback we were receiving from past candidates and had integrated their responses in job descriptions. We also reviewed where the candidates were applying from in the terms of sourcing and discussed using additional sources such as LinkedIn.

Standardizing the interview questions

We decided to introduce a specific technical interview that would take place before any other assessment. In the past, we were bringing people for onsite and they were failing the technical interview in high numbers. Having five or six people for a full day was rather expensive and the results were not worthwhile. Moving the technical interview earlier on, we were able to reduce costs significantly.

Our main problem, however, was that we were not receiving a good signal on problem-solving ability through coding. By standardizing these questions we were hoping to get the right people for our team; that is to say, to better understand and be able to quantify their problem-solving ability. Together with a colleague of mine, I worked to come up with several questions that I shared with the team. Together we calibrated them, and then picked two or three specific technical questions that would allow us to well evaluate their problem-solving ability.

We calibrated questions at different difficulty levels. We would have an easy read and a medium level read. Level one would require a person to be proficient in the coding language -- which for us is Python -- and be able to solve the challenge at the most fundamental level. The medium difficulty level would have three layers and should be significantly more complex. In terms of coding, a candidate should be able to solve a couple of different problems that have more machine learning /data science relevance whereas the first level would be about general problem-solving.

At that time we were trying to hire for multiple open roles and didn’t have enough people familiar with interviewing and simplifying the process down to a couple of questions made them comfortable faster. Since we introduced technical remote interviews we could be more open to who we were interviewing without trying to coordinate a specific day with everybody’s schedule and we could move quickly through the pipeline.

Once we added a medium level difficulty and calibrated it with people on the team, we knew what would be the main challenges for the candidates and we were able to get a stronger read on that part of the interview and have more confidence in terms of what someone’s strengths would be.

Lessons learned

• You should go through your interview questions and try to solve them yourself in a time-limited fashion and level set your expectations on how much time is reasonable to solve them. Also, it would allow you to identify the main obstacles and prepare clarifications ahead of time.
• Experiment with different difficulty levels. Solve the questions yourself and give them to candidates to see how the signal you will get from different difficulty levels is accurate and how that would affect your likeliness to make an offer.
• People should shadow first. They should do the questions offline and sit in on some interviews, then do reverse shadow, and only after that, they should be going to the interview and do it with confidence.