Back to resources

Navigating a Crisis Situation

Feelings Aside
Conflict Solving
Ownership

15 October, 2019

Greg Weinger
Greg Weinger

VP of Product Management at SheerID

Being in charge of a critical situation without the experience to handle it can happen to anyone. Greg Weinger, Head of Product Management at SheerID, highlights his learnings from a career-changing moment early on in his career.

Problem

We had a custom client implementation that involved the migration of a popular, existing website to our platform. We had performed such migrations before, but we didn't realize until late in the game their site was a couple orders of magnitude larger than we had ever handled before.

The migration took far longer than expected, so our QA time period was squeezed. With a massive effort, we loaded the data and validated functionality just in time for a highly publicized launch date. There was a lot of pressure to make that date. Load testing was nowhere on our radar. It had never been necessary before, and no one on our team--including me--had the experience to recognize what was about to come. We launched, and the software failed spectacularly. First, it slowed, and then it died completely within the hour. We had to shut it down and revert to the old site. It was a major PR problem for the customer, and I had a huge, very public failure on my hands.

Actions taken

My immediate reaction was "I'm going to be fired." That might have happened at another company, but what happened instead, though I didn't realize it at the time, was that I was given an opportunity to lead us out of this mess. A crisis is an emotional thing; everybody from the internal team to the customer is upset and fearful. I wish I could say I completely rose to the challenge, or that I knew what to do. Several people involved, including the customer, prompted us to a series of actions that became my blueprint for handling crisis situations over my career. This blueprint has helped me both in preventing and gracefully handling situations that go sideways: Deliver the News Immediately after the incident, we owed the customer and their management team a full and transparent accounting of what happened. This had to make sense at the high level, with enough supporting detail to satisfy them they were dealing with a competent partner who had nonetheless made a mistake. If we had tried to hide or minimize anything, we never would have regained their trust. Accept Responsibility There also needed to be a moment of "falling on your sword" and accepting responsibility, both personally and as a company for what happened. The customer's attitude towards us shifted perceptibly once this happened. Increase Communications The lynchpin to navigating this crisis situation was radically increasing the transparency and frequency of communications with the customer. Initially, the customer required meetings at the start and end of each day, including weekends. It felt like micromanagement, and it was, but it was necessary in order to regain their trust. As we worked through the coming weeks it lessened to once a day, and eventually a weekly cadence. Align Internally Beyond communications, what we needed to do was actually solve the problem. For that, we had to get buy-in with our own internal management that this was a top priority. The company understood the situation and committed the necessary resources for us to solve the problem. Form a Solid Plan After that, we merely had a series of challenging technical issues to solve! We had to establish new ways of measuring performance and systematically work on multiple parts of the system to achieve our performance goals. These issues ended up taking a couple of months to solve. But we re-established trust over time by building a credible plan and showing daily progress toward it. Work the Plan and Keep Communicating The customer was fairly technical and wanted to see detailed load testing and DB metrics of our progress. We worked with engineering to produce the baseline and track improvements. Sometimes we improved, and sometimes there were setbacks, but the key to it all was the ongoing communication. Eventually, we re-launched. You may have even used the software at one point!

Lessons learned

  • Rely on mentors
  • I didn't have mentors at the time I could turn to. Everyone here is doing well on that score - someone experienced, sympathetic, and removed from the situation can be a great support and provide perspective.
  • Accept your mistakes. Resist perfectionism.
  • Everyone makes mistakes. Publicly accepting responsibility when things go wrong is part of leadership, but internally, it's a different story. It's important to accept responsibility for your part, but recognize that many events and people played a role in the outcome.
  • Blame may be flying around, but try not to get caught up in blaming others or the situation, so you can approach it with a clear mind. This is much easier if you've cut yourself some slack.
  • If you're a perfectionist, you may be intolerant of mistakes, and you'll have a tendency to overdo it with self-recrimination; recognize that no one benefits from that, you least of all.
  • Avoid surprises.
  • Foresight comes with experience, but it's good practice to try to anticipate what will go wrong. I conduct pre-mortem meetings before major launches, where a large team gets together and thinks of everything that could go wrong. Then we make sure we've got a plan to deal with it.
  • Identify & communicate risks. When risks arise, it's important to make stakeholders aware of them. This gives leaders and stakeholders the full information they need to make a decision to move forward or not.
  • Trust your gut.
  • Pay attention to the warning signs even when there is external pressure to ignore them. Very public deadlines, for example.
  • For us, the fact that the data import job was taking a lot longer than we expected and experiencing failures was a risk I could have recognized and raised.
  • Be willing to deliver the bad news.
  • In that situation, it felt like the prospect of having to delay launch was the worst possible news you could deliver, but in reality, it was much worse: you could actually launch and fail! I learned that the hard way.
  • It takes some courage, and may not be well received at the time, but in the end, people will appreciate and value your honesty & transparency, and efforts to warn them. If not, you're better off somewhere else!

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

I was passed for Promotion. What now ?

26 January

Passing for promotion happens to everyone in their career lifespan. If someone does not had to go through the situation, consider them they are unique and blessed. Managing disappointment and handling situations in professional setting when things don’t pan out, is an important life skill.

Changing A Company
Handling Promotion
Feelings Aside
Feedback
Coaching / Training / Mentorship
Fairness
Career Path
Praveen Cheruvu

Praveen Cheruvu

Senior Software Engineering Manager at Anaplan

DevSecOps: Why, Benefits and Culture Shift

29 November

Why DevSecOps matter and what's really in it for you, the team and the organisation?

Innovation / Experiment
Building A Team
Leadership
Ownership
Stakeholders
Cross-Functional Collaboration
Vikash Chhaganlal

Vikash Chhaganlal

Head of Engineering at Xero

The Growth Mindset in Modern Product Engineering

28 November

The impact you can have with a Growth Mindset' and the factors involved in driving orchestrated change.

Building A Team
Leadership
Collaboration
Feedback
Ownership
Stakeholders
Vikash Chhaganlal

Vikash Chhaganlal

Head of Engineering at Xero

Strategies for Hiring

12 October

Recruiting right people is the single most important decision for the company. Building a great platform, product and company is hard. Getting the right people into the company is twice hard.

Building A Team
Company Culture
Ownership
Hiring
Praveen Cheruvu

Praveen Cheruvu

Senior Software Engineering Manager at Anaplan

Leading A (Distributed) Team? Foster "Above the Line" Behaviors.

12 July

No online tool will address your team's ability to connect, collaborate, and deliver results if the individuals don't bring the right mindset to work.

Changing A Company
Building A Team
Company Culture
Leadership
Ownership
Ram Singh

Ram Singh

Principal / Founder at id8 inc