The Ins and Outs of Evaluating Engineers - Elevate 2019

Steve Ballmer may be best remembered for his “developers, developers, developers” chant but he wasn’t wrong. Finding, hiring, and retaining strong engineers is crucial to building a successful tech company.

At Plato Elevate, our recurring summit for Engineering and Product Leaders, Ammon Bartram welcomed three guests on stage to talk about the ins and outs of evaluating engineers. Ammon is the cofounder of TripleByte, a company that helps startups improve technical interviewing and hiring.

On stage, he played host to 3 seasoned engineering executives:

• Denali Lumma, Director of Engineering at Robinhood. She previously held similar positions at Box, Uber, and Salesforce.
• Tim Wagner, VP Engineering at Coinbase. He is known for starting AWS Lambda and leading the VB and C# teams at Microsoft in the past.
• Prachi Gupta, Director of Engineering at YouTube. She handles the Red and Growth Platform, as well as YouTube Premium. She has been at Google for 12 years and is on the steering committee for women working at YouTube.

Elevate Conference — Panel: "The ins and outs of evaluating engineers"

On Evaluating Engineers

‍ Ammon jumped right in and asked his guests how they evaluate engineers, how they decide who to interview, and how the process has evolved over time.

Denali faced big challenges when she arrived at Box, mainly ensuring products and features were shipped on time. The first thing she did was to reorg from horizontal to vertical units, in order to limit the number of dependencies and to deliver value as quickly as possible.

She created a program called 10x Staff to fix a hiring problem and meet the team’s hiring goals. At the time, she felt like hiring was only a part of the problem, one side of the coin. The other side was retention. If you ever meet your hiring goals and aren’t able to retain engineers afterward, it’s a bit like filling a leaky bucket. If you churn, you lose and the end result is bad. 

She also figured out that she needed to change the mindset of Engineering Managers, who believed the recruiting team was failing to hire. Engineering Managers are actually key in hiring and retaining employees and should be evaluated on it. Denali started giving weekly quotas to EMs in terms of sourcing, set up a performance evaluation on their ability to hire. They thus expanded their sourcing efforts and started using TripleByte as well.

Tim Wagner agreed that HR and EMs should work together. When he started his career, he thought it was somebody else’s problem while the responsibility actually sits with both and works best if there is a strong partnership. EMs should  be involved from the top of the funnel to help find the best candidates. Increasing professionalism of tech recruiting challenges using TripleByte or Karat is also a critical piece of the cake to him.

On the other hand, Prachi Gupta focused on hiring senior leaders, which has its own set of challenges. During her career, Prachi has hired for positions from entry-level all the way to senior engineers. A lot of it is not just about technicality, but about the fit within the role, the team, and the company.

To ensure she got the best candidates, she empowered her managers, asking them to hire as if it was for their own companies. Doing so helped them find candidates that would fit the company beyond their initial role, and forced them to think long-term. This means they had to build a relationship and create an environment where people can grow and thrive.

In the long term, Prachi figured out it had many positive externalities for the company, exporting talent beyond her own team or department. It’s crucial to tell managers to be rooted in reality, while keeping in mind that the future is close. They’ll have to think about where the company is going, and what their new hires will do. They have to ask themselves what kind of tasks they’ll do in the future if they are ready and, if not, how to prepare them.

On Skills Needed to Be A Good EM

Ammon then asked to what extent it’s important for Engineering Managers to retain strong IC skills.

To Prachi, it is highly dependent on the role they will have in the project. Some EMs have rusty tech skills with great leadership. In teams, you would need all kinds of people and not everyone needs to be coding. EMs just need to have a good understanding of what their employees do, and know about the latest tech.

One of the principles Denali put in place when she arrived at Box was to hire people for their strengths rather than pinpoint their weaknesses. It feels very random to her that some people get offers from big companies and get rejected by other companies in the same tier. It doesn’t follow any logic. Instead, it shows that a lot of tech interviewing is arbitrary. 

She did a little experiment, asking some people to interview multiple times with different interviewers. The outcome was completely random: people either recommended to hire them or to reject them. This showed the bias and influence of the types of questions asked, the evaluation process, the people you meet or from which team they are. Denali believes that diverse teams are higher-performing and hiring for people’s strengths and looking for ways of augmenting the team with different perspectives and attributes will lead to the best outcome.

Tim thinks people tend to forget their job is to compose highly performing teams no matter how technically skilled they are. Bringing diverse people with different strengths enables him to improve the team’s productivity. His litmus test for Engineer Managers is just to ensure the EM is capable of auditing the work of its subordinates. Can they ask the right questions? What are their service-level objectives? How are they complementary to the rest of the team, even if they can't write the latest or greatest piece of code?

Besides this, Denali added that bringing diversity to an organization is difficult. To her, the industry will succeed at it only when people will come to recognize the truth: truly diverse teams are higher-performing. It will need studies and tools to correlate diversity with business outcomes, profitability or technical excellence.

To sum up, diversity, complementarity, and objectivity are sometimes amiss in the hiring process, and there might be a sort of selection bias.

On The Interview Process

Ammon Bertram concluded that research on interviews shows them to be significantly noisier, and wondered: should we be doing interviews at all? Or fall back on other ways of evaluating engineers?

Prachi answered that the key to a good hire lies in evaluation and that is hard. Instead of throwing everything away and starting from scratch, she believes recruiters can learn and evolve along the way. They should try to see where the gaps are, how they can fix them, and start doing data-driven, rubrics-based interviews. The important thing is to have a constant feedback loop between research and practice to ensure you improve over time.

At Google, her team would look for skills, but also other important things, such as care for the team, getting the job done, putting the user first, bringing value quickly, and openness to feedback. Thus, the interview process should have its own evaluation process. The challenge is to ensure teams are diverse enough. As soon as you write a job description, you already have an impact on the kind of people who will apply. How do people feel? Are they comfortable? Will non-traditional backgrounds apply? How do we prepare them for a marathon they may not be used to? Hiring committees have to make panels diverse and make sure new hires will evolve in an environment that enables them to thrive.

Tim believes in making interviews data-driven. It would feel strange if people pursued a product or a feature set without data or success criteria. The same applies to interviews. Interview data has to be studied, correlated with feedback, and built back into the process. Some tests need to be more customized or nuanced to get a sense of the interviewee’s skills. Interviewers should also be trained to know what a good, average, or bad answer to a specific question is.

Denali thinks the randomness of the output can be addressed with more project-based interviews with people working on a project together, rather than just doing a coding interview. Interviewers should also be trained to decrease noise, identify their own biases in reviews, rubrics or a specific skillset the company is looking for in a candidate. This will help them control their seat-of-the-pants gut feeling, and evaluate engineers based on technological aids or formal theory instead of personal experience and judgment.

Underrated or overrated?

Ammon was inspired by Tom Willerer’s “Underrated or overrated?” game in a previous panel and decided to play it with his guests too.

Take-home projects that take half a day to a day to finish as product interviews

• Denali: Underrated, as it is nice way to move forward in the industry, giving people the option of the format of the recruitment process: meeting, ad hoc coding tests, tech interviews or working privately at home, with your own computer and IDE. Just give people the option!
• Tim: Underrated. There is a strong caveat in these though: we need to be respectful of people, their family time, the fact they would be doing it on top of an existing job, etc. It’s important to find the right balance.
• Prachi: Underrated, but it really depends on the function. For UX interviews, for instance, we believe such case studies are useful. But again, as Tim said, we need to find a balance of how many teams people take at home.

Algorithmic problems as a way to measure people’s knowledge and technical skills in an interview

• Denali: Overrated as they can be studied and practiced. With tools like HackerRank, you can “cheat” your way and be trained on how to pass a tech interview, instead of being trained to actually do the job.
• Tim: Underrated, as there is value in it, ensuring people have a basic comprehension of computer science, not getting a wash when things start to get complicated.
• Prachi: It depends on the role and the team set up. Sometimes it’s crucial to have that tech knowledge, for instance when you work on the architecture. Sometimes, it’s less important to have them, especially when you can add another subset of skills that will be useful to the team, like UX or Data Science.

Thanks to Ammon Bartram, Prachi Gupta, Denali Lumma and Tim Wagner for their time and insights on the evaluation process of engineers.