Investing in "Keep-the-Lights-On" Initiatives
26 November, 2020
A couple of months ago I joined my current company and one thing I observed rather quickly was that the reliability efforts on the team were understaffed. As a result, we had a number of problems with system reliability that affected the morale of our backend and infrastructure teams. These teams were responsible for keeping our systems healthy and our service up for our customers. One of the key challenges I faced was balancing the prioritization of keep-the-lights-on initiatives with our product initiatives.
We were spending more time building things for customers which made systems buggy and difficult to manage while at the same time burning people on the team out.
First and foremost, I wanted to understand what the situation was on the ground and build empathy with the team. I wanted to genuinely acknowledge the problems they were grappling with and as part of that, I decided to participate in the on-call rotation.
I tracked various different KPIs that captured the on-the-ground situation such as the number of pages people were getting, the actual system reliability, the frequency of incidents, and when they were occurring (inside/outside business hours). Putting those metrics in place helped me understand what was really happening and at what pace we were improving.
Evangelizing "keep-the-light-on initiatives"
I went to the CEO to explain to them what the problem was and why it was important for us to staff. Product initiatives drive new money and it is easy to explain that persuasively. However, I had to demonstrate to executives that keep-the-lights-on initiatives are just as important for our business.
Reaping the rewards
In the end, I had to go through everything and improve the KPIs based on allocating staff on it. Once we got the staff on it, we did weekly retrospectives to do continuous improvements to our metrics.
- Keeping the lights is one of the most important pieces of running the business. The role of an engineering manager is to represent that value to the business and hold the line on the staffing requirements. If the engineering management is not advocating for that investment then the business will falter. You have to own that responsibility as a core function of what you do.
- Building empathy and putting yourself in the team’s shoes is not a show but a useful management and learning technique. It helps you direct the team’s efforts to improve systems and thoughtfully communicate why investments are important because you will have an on-the-ground understanding of the situation.
- Repetition is key to evangelism. Advocacy is a continuous exercise and I would make sure to circulate the numbers -- once I had them -- every single week and repeat why we need to improve.
Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader
William Bajzek, Director of Engineering at Sapphire Digital, compares and contrasts a team structure that utilized siloed skill sets and one where everybody’s duties overlap at the edges.
Director of Engineering at Sapphire Digital
Neelima Annam, Sr Director Information Technology at Outmatch, shares how something as minor as collaboration tools can be a BIG issue during mergers and acquisitions.
Sr. Director Information Technology at Outmatch HCM
James Engelbert, Head of Product at BT, shares how managing up is all about being an excellent manager to bring the best out of a team.
Head of Product at BT
Piyush Dubey, Senior Software Engineer at Microsoft, shares his journey of climbing up the career ladder through awkward times dealing with an introverted manager.
Senior Software Engineer at Microsoft
Piyush Dubey, Senior Software Engineer at Microsoft, shares how to understand the stakeholder communication process better and why it is essential.
Senior Software Engineer at Microsoft
You're a great engineer.
Become a great engineering leader.
Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.