Why Standardizing the Interview Process
26 November, 2020

Head of Applied Research / Data Science at Scribd
Problem
For a while, we were concerned with our interview process. We were continuously missing our hiring targets because we were not getting the signal that we needed during the interview. We decided to re-evaluate our whole hiring process from sourcing to the evaluation of candidates and determine where the gaps were. We particularly focused on standardizing our interview questions as we noticed that randomness and inconsistency in that area affected significantly the quality of our hires.
Actions taken
Before we tackled the interview questions, we wanted to make sure that we were getting enough of the right candidates. Therefore, we reevaluated the messaging recruiters were using along with diversifying our sourcing. We worked with recruiting on tailoring the job descriptions to the specific skill sets that we were interested in. In this particular case, we were focused on machine learning and experimented with the phrasing of the requirements that were causing people to not apply. We relied extensively on feedback we were receiving from past candidates and had integrated their responses in job descriptions. We also reviewed where the candidates were applying from in the terms of sourcing and discussed using additional sources such as LinkedIn.
Standardizing the interview questions
We decided to introduce a specific technical interview that would take place before any other assessment. In the past, we were bringing people for onsite and they were failing the technical interview in high numbers. Having five or six people for a full day was rather expensive and the results were not worthwhile. Moving the technical interview earlier on, we were able to reduce costs significantly.
Our main problem, however, was that we were not receiving a good signal on problem-solving ability through coding. By standardizing these questions we were hoping to get the right people for our team; that is to say, to better understand and be able to quantify their problem-solving ability. Together with a colleague of mine, I worked to come up with several questions that I shared with the team. Together we calibrated them, and then picked two or three specific technical questions that would allow us to well evaluate their problem-solving ability.
We calibrated questions at different difficulty levels. We would have an easy read and a medium level read. Level one would require a person to be proficient in the coding language -- which for us is Python -- and be able to solve the challenge at the most fundamental level. The medium difficulty level would have three layers and should be significantly more complex. In terms of coding, a candidate should be able to solve a couple of different problems that have more machine learning /data science relevance whereas the first level would be about general problem-solving.
At that time we were trying to hire for multiple open roles and didn’t have enough people familiar with interviewing and simplifying the process down to a couple of questions made them comfortable faster. Since we introduced technical remote interviews we could be more open to who we were interviewing without trying to coordinate a specific day with everybody’s schedule and we could move quickly through the pipeline.
Once we added a medium level difficulty and calibrated it with people on the team, we knew what would be the main challenges for the candidates and we were able to get a stronger read on that part of the interview and have more confidence in terms of what someone’s strengths would be.
Lessons learned
- You should go through your interview questions and try to solve them yourself in a time-limited fashion and level set your expectations on how much time is reasonable to solve them. Also, it would allow you to identify the main obstacles and prepare clarifications ahead of time.
- Experiment with different difficulty levels. Solve the questions yourself and give them to candidates to see how the signal you will get from different difficulty levels is accurate and how that would affect your likeliness to make an offer.
- People should shadow first. They should do the questions offline and sit in on some interviews, then do reverse shadow, and only after that, they should be going to the interview and do it with confidence.
Discover Plato
Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader
Related stories
26 May
Hiring 10x engineers is hard for most companies. It’s a tough battle out there for talent. So how should most companies approach building their team?

Vaidik Kapoor
VP Engineering - DevOps & Security at Grofers
26 May
Philip Gollucci, Director of Cloud Engineering at CareRev, describes a new method for hiring in a market climate that favors candidates instead of recruiters.

Philip Gollucci
CEO/Founder at P6M7G8 Inc.
25 May
Vimal Patel, Founder and CTO at iMORPHr, shares how he retained all of his employees since beginning his software development company in 2019.

Vimal Patel
Director of Engineering at iMORPHr
24 May
Liz Henderson, an Executive consultant at Capgemini, shares her experience hiring a data team with a manager who was difficult to work with.

Liz Henderson
Executive consultant at Capgemini
11 May
Tom Hill, Engineering Manager at Globality, Inc., shares how he works with a culturally diverse team based within a thirteen-hour time gap.

Tom Hill
Engineering Manager at Torii
You're a great engineer.
Become a great engineering leader.
Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.
