Plato Elevate Winter Summit has been announced (Dec 7th-8th)


Back to resources

Why Standardizing the Interview Process


26 November, 2020

Kevin Perko
Kevin Perko

Head of Applied Research / Data Science at Scribd

Kevin Perko, Head of Applied Research and Data Science at Scribd, explains how standardizing the interview process -- and particularly the interview questions -- resulted in better hires and skill set fits.


For a while, we were concerned with our interview process. We were continuously missing our hiring targets because we were not getting the signal that we needed during the interview. We decided to re-evaluate our whole hiring process from sourcing to the evaluation of candidates and determine where the gaps were. We particularly focused on standardizing our interview questions as we noticed that randomness and inconsistency in that area affected significantly the quality of our hires.

Actions taken

Before we tackled the interview questions, we wanted to make sure that we were getting enough of the right candidates. Therefore, we reevaluated the messaging recruiters were using along with diversifying our sourcing. We worked with recruiting on tailoring the job descriptions to the specific skill sets that we were interested in. In this particular case, we were focused on machine learning and experimented with the phrasing of the requirements that were causing people to not apply. We relied extensively on feedback we were receiving from past candidates and had integrated their responses in job descriptions. We also reviewed where the candidates were applying from in the terms of sourcing and discussed using additional sources such as LinkedIn.

Standardizing the interview questions

We decided to introduce a specific technical interview that would take place before any other assessment. In the past, we were bringing people for onsite and they were failing the technical interview in high numbers. Having five or six people for a full day was rather expensive and the results were not worthwhile. Moving the technical interview earlier on, we were able to reduce costs significantly.

Our main problem, however, was that we were not receiving a good signal on problem-solving ability through coding. By standardizing these questions we were hoping to get the right people for our team; that is to say, to better understand and be able to quantify their problem-solving ability. Together with a colleague of mine, I worked to come up with several questions that I shared with the team. Together we calibrated them, and then picked two or three specific technical questions that would allow us to well evaluate their problem-solving ability.

We calibrated questions at different difficulty levels. We would have an easy read and a medium level read. Level one would require a person to be proficient in the coding language -- which for us is Python -- and be able to solve the challenge at the most fundamental level. The medium difficulty level would have three layers and should be significantly more complex. In terms of coding, a candidate should be able to solve a couple of different problems that have more machine learning /data science relevance whereas the first level would be about general problem-solving.

At that time we were trying to hire for multiple open roles and didn’t have enough people familiar with interviewing and simplifying the process down to a couple of questions made them comfortable faster. Since we introduced technical remote interviews we could be more open to who we were interviewing without trying to coordinate a specific day with everybody’s schedule and we could move quickly through the pipeline.

Once we added a medium level difficulty and calibrated it with people on the team, we knew what would be the main challenges for the candidates and we were able to get a stronger read on that part of the interview and have more confidence in terms of what someone’s strengths would be.

Lessons learned

  • You should go through your interview questions and try to solve them yourself in a time-limited fashion and level set your expectations on how much time is reasonable to solve them. Also, it would allow you to identify the main obstacles and prepare clarifications ahead of time.
  • Experiment with different difficulty levels. Solve the questions yourself and give them to candidates to see how the signal you will get from different difficulty levels is accurate and how that would affect your likeliness to make an offer.
  • People should shadow first. They should do the questions offline and sit in on some interviews, then do reverse shadow, and only after that, they should be going to the interview and do it with confidence.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader

Related stories

Demystifying the Cult of the Founding Engineer: Talking to Customers and Discovering “Hidden” Talent

23 November

Albert Lie, former Founding Engineer and Tech Lead at Xendit, didn’t know what it takes to become an early engineering hire and not a lot of people around him experienced this unknown and arcane path. He had to learn it the hard way from the pitfalls he encountered along the way and he has been creating numerous frameworks to measure his growth and keep burgeoning in this role since then. He codifies and expresses the systems he put in place surrounding the balance of customer inquiry to product building and growing the engineering team.

Albert Lie

Albert Lie

Former Tech Lead at Xendit

How to Build a Software Team from the Ground Up

12 November

Deepesh Makkar, Sr Director of Engineering at SunPower Corporation, shares his experience transitioning his organization from contractors to a 50/50 split of full-time employees and international vendors.

Cross-Functional Collaboration
Agile / Scrum
Deepesh Makkar

Deepesh Makkar

Sr Director of Engineering at SunPower Corporation

Gaining the Technical Confidence to Crush an Interview

2 November

Chris Sellek, Staff Software Engineer at WillowTree, describes his troublesome process to land a job when he graduated college and how he gained the technical confidence to perform well during interviews.

Managing Expectations
Career Path
Chris Sellek

Chris Sellek

Staff Software Engineer at WillowTree Apps

Creating New Hiring Methods to Find Qualified Candidates

2 November

Robin Gupta, Engineering Manager at athenahealth, shares his most recent challenge of remote hiring in an oversaturated technical market.

Scaling Team
Different Skillsets
Robin Gupta

Robin Gupta

Engineering Manager at athenahealth

Mentoring Freshly Recruited Employees

28 October

Yamini Choudhary, Business Strategy at Verizon, unveils some of the right tactics toward coaching fresh graduates.

Coaching / Training / Mentorship
Yamini Choudhary

Yamini Choudhary

Business Strategy at Verizon

You're a great engineer.
Become a great engineering leader.

Plato ( is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.