Plato Elevate Winter Summit has been announced (Dec 7th-8th)

🔥

Back to resources

Investing in "Keep-the-Lights-On" Initiatives

Managing Up
Dev Processes
Collaboration

26 November, 2020

Matt Pillar
Matt Pillar

VP Engineering at OneSignal

Matt Pillar, VP of Engineering at OneSignal, shares how he improved the reliability of high scale systems by securing investment in infrastructure and on-call services.

Problem

A couple of months ago I joined my current company and one thing I observed rather quickly was that the reliability efforts on the team were understaffed. As a result, we had a number of problems with system reliability that affected the morale of our backend and infrastructure teams. These teams were responsible for keeping our systems healthy and our service up for our customers. One of the key challenges I faced was balancing the prioritization of keep-the-lights-on initiatives with our product initiatives.

We were spending more time building things for customers which made systems buggy and difficult to manage while at the same time burning people on the team out.

Actions taken

Empathy building
First and foremost, I wanted to understand what the situation was on the ground and build empathy with the team. I wanted to genuinely acknowledge the problems they were grappling with and as part of that, I decided to participate in the on-call rotation.

Establishing measurements
I tracked various different KPIs that captured the on-the-ground situation such as the number of pages people were getting, the actual system reliability, the frequency of incidents, and when they were occurring (inside/outside business hours). Putting those metrics in place helped me understand what was really happening and at what pace we were improving.

Evangelizing "keep-the-light-on initiatives"
I went to the CEO to explain to them what the problem was and why it was important for us to staff. Product initiatives drive new money and it is easy to explain that persuasively. However, I had to demonstrate to executives that keep-the-lights-on initiatives are just as important for our business.

Reaping the rewards In the end, I had to go through everything and improve the KPIs based on allocating staff on it. Once we got the staff on it, we did weekly retrospectives to do continuous improvements to our metrics.

Lessons learned

  • Keeping the lights is one of the most important pieces of running the business. The role of an engineering manager is to represent that value to the business and hold the line on the staffing requirements. If the engineering management is not advocating for that investment then the business will falter. You have to own that responsibility as a core function of what you do.
  • Building empathy and putting yourself in the team’s shoes is not a show but a useful management and learning technique. It helps you direct the team’s efforts to improve systems and thoughtfully communicate why investments are important because you will have an on-the-ground understanding of the situation.
  • Repetition is key to evangelism. Advocacy is a continuous exercise and I would make sure to circulate the numbers -- once I had them -- every single week and repeat why we need to improve.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

Specialization vs. Wearing Many Hats

23 November

William Bajzek, Director of Engineering at Sapphire Digital, compares and contrasts a team structure that utilized siloed skill sets and one where everybody’s duties overlap at the edges.

Internal Communication
Collaboration
William Bajzek

William Bajzek

Director of Engineering at Sapphire Digital

Mergers and Acquisitions: Collaboration tools hold a key to bringing cultures together

23 November

Neelima Annam, Sr Director Information Technology at Outmatch, shares how something as minor as collaboration tools can be a BIG issue during mergers and acquisitions.

Acquisition / Integration
Internal Communication
Collaboration
Neelima Annam

Neelima Annam

Sr. Director Information Technology at Outmatch HCM

The art of managing up

19 November

James Engelbert, Head of Product at BT, shares how managing up is all about being an excellent manager to bring the best out of a team.

Mission / Vision / Charter
Managing Up
Internal Communication
Strategy
Stakeholders
Cross-Functional Collaboration
James Engelbert

James Engelbert

Head of Product at BT

How to Build Rapport With an Introverted Manager

17 November

Piyush Dubey, Senior Software Engineer at Microsoft, shares his journey of climbing up the career ladder through awkward times dealing with an introverted manager.

Managing Expectations
Internal Communication
Collaboration
Coaching / Training / Mentorship
Juniors
Piyush Dubey

Piyush Dubey

Senior Software Engineer at Microsoft

The Benefits of Stakeholder Communication

17 November

Piyush Dubey, Senior Software Engineer at Microsoft, shares how to understand the stakeholder communication process better and why it is essential.

Meetings
Internal Communication
Collaboration
Ownership
Stakeholders
Piyush Dubey

Piyush Dubey

Senior Software Engineer at Microsoft

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.