Evaluating technical abilities when hiring

Problem

"Evaluating the technical abilities of a candidate you are interviewing is difficult. Tests tend to be subjective, and somebody with three years of experience evaluating technical skills may not be any better than someone who has done it just once. In addition, hiring is often about interpersonal skills, and an engineer may be technically skilled but may not interview well. People will often resort to presenting candidates with brain problems, coding problems to be completed right then and there, or take-home tests that are fairly unreliable."

Actions taken

"As an alternative, my company developed a very reliable technique for doing technical interviews. And because it's easily repeatable we have gotten better and better at it, and we are now able to evaluate engineers in 5-10 minutes with a fairly high level of accuracy."

"Our test is called the 'dog-walking problem', and was designed to test technical abilities that are important for us (relational databases). The problem goes like this: You are an entrepreneur and you want to have a start-up where you will be walking dogs by yourself. We ask the engineer to design a schema on how they would design a database to walk their dogs. It's a fairly simple exercise, which takes about five minutes and it shows how they would model scheduling. After this, we increase the exercise's difficulty by saying that a friend wants to join the company so now there will be two dog walkers. Then we say there are now two locations, with one on the east coast and one on the west coast. The problem gets harder and harder. A question like, 'Can you write a SQL query to find me which time slot is the most popular' reveals inadequacies in their scheduling schema, especially across time zones? Can an engineer re-factor well based on this wrinkle? The problem eventually goes on to cover transactions, demand prediction, internationalization and scaling. Just when you think you figure it out, we expand to become a cat walking business."

"One of the main reasons this test is so powerful is that we use the same problem again and again for every single candidate. Every person in our engineering team has gone through this problem, and as we have been using this problem since 2003, we've seen close to 15 years of candidates run through the same problem. Because we aren't nearly the size of Google, the problem has never been published, and even if it was I don't think you can easily solve it."

"We also have other engineers sit in the room while the candidate works through the problem. This allows our staff to see other engineers go through the problem. This improves their understanding of the problem, and it leads to staff having respect for one another as they can see each other go through the problem. It also allows staff to develop their interviewing skills for the future."

Lessons learned

"Everybody interviews differently, so you have to really think about how to technically evaluate your engineers, rather than throwing brain problems at candidates and judging them when they can't work out an answer. The tests should be repeatable, scalable, and ideally improve in an iterative way. This allows you to be objective in how you interview talent."

"Spend time thinking about a problem that can progressively increase in difficulty, and that can be run again and again, regardless of the type of engineering that the candidates specialize in."