login


Google Sign inLinkedIn Sign in

Don't have an account? 

Think you know how to conduct a technical interview? Think again.

24 June, 2020

By Shannon Hogue, Global Head of Solutions Engineering, Karat (https://karat.com/)

We’ve all been talking about fairness and bias a lot these past few weeks -- and the depth at which it's currently being discussed is long overdue. At Karat, we have been conducting research to identify sources of bias, inconsistency, and other challenges that relate to technical interviewing.

Back in January, we partnered with Harris Poll to conduct a survey of 253 engineering leaders and hiring decision-makers. We used those results as benchmarks to compare to a second survey of 100 additional leaders last month to gauge the impact of COVID-19 on hiring trends. You can read the full Interview Gap report here (no signup required): https://bit.ly/2Z3oLQs

A lot of the directional findings are pretty intuitive in a world turned upside down by a global pandemic.

  • Software engineers are more valuable than ever. In our May survey, 90% of engineering leaders agree that “software engineers are more valuable than capital, up from 80% in January.

  • It’s easy to understand why there are more software engineers looking for jobs compared to the start of the year, but the shifting labor dynamics are making it more difficult to find the best engineers available. 45% of engineering leaders now strongly agree that “while there are a lot of software engineers available, it is hard to find the right ones,” compared to just 25% in January.

Do you see eye to eye with your boss?

One interesting trend is how perceptions have changed across levels in an organization, and how rapidly the C-suite has shifted to expose some differences between them and the VP, Director, and Manager levels.

The good news is that your C-suite is happy with your performance. 71% of C-level leaders are “very satisfied” with the performance of their software engineers, compared to just 37% other engineering leaders.

The bad news--especially as you are looking to grow as organizations emerge from the economic downturn--is that they think you suck at conducting interviews. 84% of c-level agree or strongly agree that “very few people at my company know how to conduct a technical interview,” compared to just 56% of managers.

Why the difference?

I’ve conducted a lot of technical interviews. I used to think I was pretty good at them. I’m personable. I can remember questions I’ve been asked previously (count the number of lowercase a’s in a null-terminated string, anyone?), and I treat all of my candidates fairly. But when we’re talking about systemic issues of bias, or functional issues of predictiveness, we need to look at the full system, not just our individual opinions. And that’s something the C-suite does.

Do you have a sense of how your interviews impact the company’s bottom line? Do you know how many interviews your company conducts per open software engineering role? Do you track detailed hiring funnel metrics?

The answer for the majority of us, is “no,” or at least “not to the extent of my C-suite.”

  • Nearly half of C-level executives (47%) track detailed hiring funnel metrics like the number of interviews, candidates, and onsite to offer ratios compared to just 25% of managers.

  • This means that 75% of engineering managers, don’t understand the inner workings of their own hiring system.

Think you know how many interviews you conduct per open SWE role?

  • Managers say they conduct an average of 17.9 technical interviews per SWE hire.

  • C-level execs, who are looking at fuller data sets and hiring funnel metrics, say that number is 24.7.

So it’s no surprise that 79% of C-level execs agree that “Interviewing software engineering candidates is a financial drain on my company” compared to just. 50% of VPs, directors, and managers.

Planning to ship some major features this fall during the thick of the upcoming university recruiting season?

  • 55% of C-level leaders recognized that “reaching hiring targets while delivering product features is challenging,” compared to just 32% across other levels.

How can we get better?

There are three areas that organizations can focus on to right their interviewing ships: capacity, consistency, and measurement.

  • Capacity constraints make it difficult to conduct enough interviews to reach hiring targets. 61% of c-level execs agreed that “reaching hiring targets would be easier if we had more people qualified to interview” compared to 37% of VPs, directors, and managers. You can increase capacity by training more engineers to conduct interviews.

  • Consistency is another way to improve your hiring program. 55% of the c-suite strongly agree that “the typical technical interview fails to predict the performance of software engineers,” compared to 24% of the rest of us. Create a structured interview process that includes dedicated interviewers and performance reviews. Use standardized, battle-tested questions to evaluate specific competencies. 85% of engineering leaders who were very confident in reaching hiring targets used a structured process compared to just 73% of less-confident leaders.

  • Measurement is critical to reducing bias. Tracking detailed hiring funnel metrics is something that 73% of companies that are “very satisfied” with their engineers’ performance do, compared to just 57% of other organizations. In addition to looking across the entire funnel, use a structured scoring rubric for every interview to reduce opportunities for individual interviewer bias to affect your hiring process. One of the most common questions I get at events and meetups is about our structured scoring rubric. Here’s a quick guide on how to create a structured scoring rubric that my team pulled together: https://bit.ly/2VfxWvR

If you’re interested in learning more about hiring and interviewing software engineers, check out our recent live Q&A (https://www.youtube.com/watch?v=CkdkO96JmqM), or check out the full report from Karat (https://bit.ly/2Z3oLQs).

By Shannon Hogue, Global Head of Solutions Engineering, Karat (https://karat.com/)

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.