Whiteboard interviews, how to and how not to do it

Problem

A few years ago in my previous company, we decided that we want to do whiteboard interviews. I live in France, not in the bay area, so by us, whiteboarding is not a common practice. By having engineers which did not have experience either doing or answering to whiteboard, we instantly refused all the candidates in the pipeline. Engineers would pull the wildest problems out of their sleeves. Problems which were "free form" and have one single solution. Of course, candidates were horrified, and scared from this. We had horrible feedback, and lost a lot of good people.

Actions taken

We first realized that we have an issue, and paused whiteboard interviews until we get our bearings. We then did some training, and a risky one, some of our seniors (the original team that was trusted to do this) were (voluntarily) allowed to do interviews with other companies just so that they experience the outside world. Then they had to share the problem, and the experience with the core team and the other engineers. We also did a lot of research about good and bad practices and decided to devise rules for whiteboard. Something in these lines:

• A question has to be approved by at least 3 people first to get in the pool for whiteboard.
• A question has to have at least two level answers (easy and advanced)
• A question must be incomplete (our questions were tailored to spike questions)
• Each question which will be proposed has to be fully explained to the team, why is the question important, what will it reveal, which routes can it take, which possible solutions are there. As for the reviewers they need to go through a process:
• Each interviewer has to "shadow" at least 3 interviews with before performing one.
• A new interviewer has to be shadowed by a senior before he can shadow junior interviewers. Before an interview we send a very detailed email with sample questions, videos and examples of questions we do not ask. As we want our candidates to be fully prepared and we don't like to do surprises.

Lessons learned

We defined that the whiteboard is a way for us to solve a problem together, not to prove the candidate can't do quicksort from scratch. As I read somewhere, a good interview question is like an onion. You can peel the question layer by layer and there is still one layer below it. We structured the questions in a fashion where they would spike another question in the candidate. With this we make sure the candidate is thorough when analyzing a problem. Once they start we monitor:

• ability to create a structured approach.
• ability to explain their approach in a quick and simple fashion
• ability to sketch out their solution and map their thoughts We also try to get the candidate to somewhere where they have to say "I don't know" and we really appreciate candidates which do that. As we're sure they will do it again, and we won't have to worry about improvisation. After everything our recruitment efforts were back on track, we successfully hired good engineers, and got very positive grades from people we rejected. To our surprise these candidates knew they did not perform well and asked if they can have a second chance in the future.