Elevate Spring Summit has been announced (Thu, Mar 11th)

🔥


Don't have an account? 

Investing in "Keep-the-Lights-On" Initiatives

Managing Up
Dev Processes
Collaboration

26 November, 2020

Matt Pillar, VP of Engineering at OneSignal, shares how he improved the reliability of high scale systems by securing investment in infrastructure and on-call services.

Problem

A couple of months ago I joined my current company and one thing I observed rather quickly was that the reliability efforts on the team were understaffed. As a result, we had a number of problems with system reliability that affected the morale of our backend and infrastructure teams. These teams were responsible for keeping our systems healthy and our service up for our customers. One of the key challenges I faced was balancing the prioritization of keep-the-lights-on initiatives with our product initiatives.
 

We were spending more time building things for customers which made systems buggy and difficult to manage while at the same time burning people on the team out.
 

Actions taken

Empathy building
First and foremost, I wanted to understand what the situation was on the ground and build empathy with the team. I wanted to genuinely acknowledge the problems they were grappling with and as part of that, I decided to participate in the on-call rotation.
 

Establishing measurements
I tracked various different KPIs that captured the on-the-ground situation such as the number of pages people were getting, the actual system reliability, the frequency of incidents, and when they were occurring (inside/outside business hours). Putting those metrics in place helped me understand what was really happening and at what pace we were improving.
 

Evangelizing "keep-the-light-on initiatives"
I went to the CEO to explain to them what the problem was and why it was important for us to staff. Product initiatives drive new money and it is easy to explain that persuasively. However, I had to demonstrate to executives that keep-the-lights-on initiatives are just as important for our business.
 

Reaping the rewards In the end, I had to go through everything and improve the KPIs based on allocating staff on it. Once we got the staff on it, we did weekly retrospectives to do continuous improvements to our metrics.
 

Lessons learned

  • Keeping the lights is one of the most important pieces of running the business. The role of an engineering manager is to represent that value to the business and hold the line on the staffing requirements. If the engineering management is not advocating for that investment then the business will falter. You have to own that responsibility as a core function of what you do.
  • Building empathy and putting yourself in the team’s shoes is not a show but a useful management and learning technique. It helps you direct the team’s efforts to improve systems and thoughtfully communicate why investments are important because you will have an on-the-ground understanding of the situation.
  • Repetition is key to evangelism. Advocacy is a continuous exercise and I would make sure to circulate the numbers -- once I had them -- every single week and repeat why we need to improve.

Related stories

Why a Risk-Driven Product Roadmap
20 January

Aram Grigoryan, Product Manager at Qualtrics, explains how a risk-driven product roadmap can help with building a product that bridges the gap between the digital and physical world.

Product
Collaboration
Roadmap
Aram Grigoryan

Aram Grigoryan

Product Manager at Facebook

Managing Cross-Team Dependencies
28 December

Frances Li, Lead Product Manager at Carta, tells of her recent experience of managing dependencies with another team and how ensuring transparency is a prerequisite for resolving them.

Cross-functional collaboration
Managing Expectations
Collaboration
Frances Li

Frances Li

Lead Product Manager at Carta

Bringing Tangible Results Within the First Quarter
28 December

Frances Li, Lead Product Manager at Carta, recalls how a sudden change of direction and decision to bring tangible results within the first quarter impacted a product she was building.

Collaboration
Roadmap
Prioritization
Frances Li

Frances Li

Lead Product Manager at Carta

Building Internal Tools and Balancing Different Stakeholders’ Requirements
28 December

Frances Li, Lead Product Manager at Carta, describes how she created a unique framework that helps her balance different stakeholders’ requirements when building internal tools.

Managing Expectations
Collaboration
Team processes
Prioritization
Frances Li

Frances Li

Lead Product Manager at Carta

Building the Automated Testing Culture
25 December

Sean Calarco, Director of Engineering at The Predictive Index, recalls how he successfully built the automated testing culture despite the initial resistance he encountered.

Dev Processes
Team processes
Sean Calarco

Sean Calarco

Director of Engineering at The Predictive Index

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.