Back to resources

Why Standardizing the Interview Process

Hiring

26 November, 2020

Kevin Perko
Kevin Perko

Head of Applied Research / Data Science at Scribd

Kevin Perko, Head of Applied Research and Data Science at Scribd, explains how standardizing the interview process -- and particularly the interview questions -- resulted in better hires and skill set fits.

Problem

For a while, we were concerned with our interview process. We were continuously missing our hiring targets because we were not getting the signal that we needed during the interview. We decided to re-evaluate our whole hiring process from sourcing to the evaluation of candidates and determine where the gaps were. We particularly focused on standardizing our interview questions as we noticed that randomness and inconsistency in that area affected significantly the quality of our hires.

Actions taken

Before we tackled the interview questions, we wanted to make sure that we were getting enough of the right candidates. Therefore, we reevaluated the messaging recruiters were using along with diversifying our sourcing. We worked with recruiting on tailoring the job descriptions to the specific skill sets that we were interested in. In this particular case, we were focused on machine learning and experimented with the phrasing of the requirements that were causing people to not apply. We relied extensively on feedback we were receiving from past candidates and had integrated their responses in job descriptions. We also reviewed where the candidates were applying from in the terms of sourcing and discussed using additional sources such as LinkedIn.

Standardizing the interview questions

We decided to introduce a specific technical interview that would take place before any other assessment. In the past, we were bringing people for onsite and they were failing the technical interview in high numbers. Having five or six people for a full day was rather expensive and the results were not worthwhile. Moving the technical interview earlier on, we were able to reduce costs significantly.

Our main problem, however, was that we were not receiving a good signal on problem-solving ability through coding. By standardizing these questions we were hoping to get the right people for our team; that is to say, to better understand and be able to quantify their problem-solving ability. Together with a colleague of mine, I worked to come up with several questions that I shared with the team. Together we calibrated them, and then picked two or three specific technical questions that would allow us to well evaluate their problem-solving ability.

We calibrated questions at different difficulty levels. We would have an easy read and a medium level read. Level one would require a person to be proficient in the coding language -- which for us is Python -- and be able to solve the challenge at the most fundamental level. The medium difficulty level would have three layers and should be significantly more complex. In terms of coding, a candidate should be able to solve a couple of different problems that have more machine learning /data science relevance whereas the first level would be about general problem-solving.

At that time we were trying to hire for multiple open roles and didn’t have enough people familiar with interviewing and simplifying the process down to a couple of questions made them comfortable faster. Since we introduced technical remote interviews we could be more open to who we were interviewing without trying to coordinate a specific day with everybody’s schedule and we could move quickly through the pipeline.

Once we added a medium level difficulty and calibrated it with people on the team, we knew what would be the main challenges for the candidates and we were able to get a stronger read on that part of the interview and have more confidence in terms of what someone’s strengths would be.

Lessons learned

  • You should go through your interview questions and try to solve them yourself in a time-limited fashion and level set your expectations on how much time is reasonable to solve them. Also, it would allow you to identify the main obstacles and prepare clarifications ahead of time.
  • Experiment with different difficulty levels. Solve the questions yourself and give them to candidates to see how the signal you will get from different difficulty levels is accurate and how that would affect your likeliness to make an offer.
  • People should shadow first. They should do the questions offline and sit in on some interviews, then do reverse shadow, and only after that, they should be going to the interview and do it with confidence.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

Effective Hiring Practices: Asking the Right Questions

23 June

Josef Starychfojtu, VP of Engineering at Mews, delves into his interviewing tactics for recruiting the best-suited candidates.

Building A Team
Hiring
Josef Starychfojtu

Josef Starychfojtu

VP of Engineering, Platform at Mews

10x engineer or 10x impact?

26 May

Hiring 10x engineers is hard for most companies. It’s a tough battle out there for talent. So how should most companies approach building their team?

Building A Team
Leadership
Hiring
Coaching / Training / Mentorship
Vaidik Kapoor

Vaidik Kapoor

Consulting CTO/VPE at Self Employeed

How to Streamline Your Recruitment Process for Quick and Effective Hiring

26 May

Philip Gollucci, Director of Cloud Engineering at CareRev, describes a new method for hiring in a market climate that favors candidates instead of recruiters.

Scaling Team
Building A Team
Hiring
Philip Gollucci

Philip Gollucci

CEO/Founder at P6M7G8 Inc.

How to Maximize Employee Retention in Engineering Teams

25 May

Vimal Patel, Founder and CTO at iMORPHr, shares how he retained all of his employees since beginning his software development company in 2019.

Building A Team
Company Culture
Hiring
Retention
Psychological Safety
Vimal Patel

Vimal Patel

Director of Engineering at iMORPHr

Hiring a Data Team With a Stubborn Manager

24 May

Liz Henderson, an Executive consultant at Capgemini, shares her experience hiring a data team with a manager who was difficult to work with.

Managing Up
Building A Team
Conflict Solving
Hiring
Data Team
Liz Henderson

Liz Henderson

Executive Advisor at Capgemini