Acting Quickly To Resolve an Incident
9 April, 2021

Engineering Manager at Carta
Problem
Recently we had an internal incident that required a prompt response. Our long-standing customer notified us that they found a bug on our platform. They were under a tight deadline to deliver some documents to authorities by means of our platform. The bug disabled them to locate the documents, let alone to send them.
Actions taken
A couple of our engineers were tasked to investigate what happened and why. While it was a matter of the greatest urgency, they were stuck for a few hours, unable to understand what had happened. I was called in to help them out.
I had a strong guess what the problem was. Being calm and composed allowed me to coordinate our efforts in the right direction. I organized the team in no time; I had someone checking on one piece of the application, someone else looking at the logs while I was talking to our point of contact that was communicating with the client. At that moment, I understood that having the most accurate pieces of information was critical; the more accurate the information, the more precisely we could detect the problem. I instructed the team to report to me frequently about what was going on. After collecting enough information, I was confident that my original guess was correct. We were able to identify what happened to our customer’s documents and merely had to recover them.
By quickly resolving the incident, we managed to turn a disadvantageous situation in our favor and further strengthen the relationship with the client.
In addition, I made sure to turn this experience into institutional knowledge. In a situation of great urgency, when every second counts, one can’t often share what they are doing. But I seized the first opportunity after the incident was resolved to reach out to engineers who were originally tasked to fix the problem and offered to explain what I did step-by-step. Other people also joined and were appreciative of my efforts to make our actions the collective learning experience.
Lessons learned
- By acting quickly to resolve an incident, one can turn the disadvantageous situation into a great opportunity. Not only did I fix the problem promptly without jeopardizing the relationship with the customer, but our rapid response and commitment to be at our customer’s service strengthen our relationship.
- I was able to share step-by-step what I did, so next time when someone on the team encounters the same problem, we can be even quicker to resolve the incident.
Discover Plato
Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader
Related stories
26 May
Elwin Lau, Director of Software at Jana, advocates the importance of maintaining culture within a company when scaling teams.

Elwin Lau
Director of Software at JANA Corporation
26 May
Elwin Lau, Director of Software at Jana, advocates the importance of maintaining culture within a company when scaling teams.

Elwin Lau
Director of Software at JANA Corporation
26 May
Hiring 10x engineers is hard for most companies. It’s a tough battle out there for talent. So how should most companies approach building their team?

Vaidik Kapoor
VP Engineering - DevOps & Security at Grofers
24 May
Jord Sips, Senior Product Manager at Mews, shares his expertise on a common challenge for product managers – finding root causes and solutions.

Jord Sips
Senior Product Manager at Mews
19 May
Jonathan Belcher, Engineering Manager at Curative, shares an unknown side of synchronous communication tools and advises managers on how to handle a team that’s spread across the globe.

Jonathan Belcher
Engineering Manager - Patient Experience at Curative
You're a great engineer.
Become a great engineering leader.
Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.
