Overhauling An Interview Process

Problem

"When I began working at Segment.com, I was hired as an Engineering Manager but for what was essentially a team of zero. I needed to hire quickly. However, our engineers were not very good at conducting interviews, leading to poor candidate experiences. In addition, during debriefs we were not able to make conclusive yes or no decisions, due to poor signals because of our interview process. We had a few engineers at the company, but barely enough to put together an interview loop. Due to this, I was doing quite a lot of the phone screening calls and on-site interviews."

Actions taken

"We had two recruiters, and I decided to partner very closely with them and to spend a lot of time talking about what our strategy was, and what worked and didn't work in our interview processes. We used Greenhouse, a piece of recruiting and applicant tracking software, extensively and set up scorecards. We gathered up the raw data on the interview process we had and worked to ensure we were documenting the process as it existed at that point. Next, we decided on a new interview loop. To do this, we looked at the interviews in our loop to determine whether they were adding value, what signal each interview should give us, and whether the interview was well set up to give that signal. We removed interviews that weren't relevant. For technical exercises, we clearly documented the exercises from the perspective of the interviewer, so it was clear to anyone asked to interview a candidate what a good, average and poor performance looked like. Over time, we have also created documents we share with interviewees in order to prepare them for what to expect during our interviews. This is especially useful for interviews where we will expect candidates to write code. Next, we dived into the technical exercises in more detail. We decided to move away from theoretical exercises and towards testing people on things they would do on a day-to-day basis. To do this, we gave applicants take-home tests. What we found was that people who were really invested in working for Segment did a great job, people who were shopping around at multiple companies didn't bother, and people that weren't good at managing their own time were screened out. This helped to save us time, as we then didn't have to bring them onsite to interview. However, there should be a balance and the take-home exercise shouldn't be so long as to make candidates lose interest. We also performed two major technical interviews onsite - the first was a pair programming exercise that took two hours. Candidates would write code with our engineers in a manner similar to the way they would when working and our Engineers would ask questions throughout. This was extremely useful, and even with very senior candidates, it gave us a good sense of whether someone would be able to guide or lead a technical discussion in their team and whether they'd be able to help their team if it got stuck."

Lessons learned

"It's extremely important to partner very closely with your recruiters. This is one of the most important relationships you can build, so I spend a lot of time with our recruiters talking about sourcing, candidates coming onsite, and debriefing. You should also look critically at your interview process - you can always make it a little bit better. You shouldn't be afraid to experiment, but ensure you have a plan in terms of what information you want to gather. It can be incredibly valuable to look for practical skills rather than focusing on theoretical tests."