Surviving an All-Hands Fire with No One On-Site

Problem

I was called in as an Escalation Manager for a fire issue at a US datacenter during a banking holiday in France. As you can imagine, it was not an easy task for me to reach everyone. Additionally, these were all remote workers. Regardless of the impossible circumstances, I was determined to solve the issue and take control of the situation.

Actions taken

My very first task was to produce internal communication about the incident and orchestrate people to do things in the correct order. For example, there could be server restarts if the network had not been not checked. I developed a roadmap for this issue which laid out the steps people should take. Most engineers were in France it was not possible to get people directly in the US datacenter. Luckily, one network guy was there and available to operate. Once people had actions and the order, all was clear and quick. The issue was handled efficiently and in as timely manner.

Lessons learned

This was the first time in my career that I had to lead all people involved remotely for this type of critical issue. To make things worse, I was not able to not send people to the office. I had to accept the situation. The context was different and thus, so was the method. This failure made me think a lot and I realized - among other things - that both onsite workers and those working remotely can be efficient and productive. I found the remote workers to be less stressful than if we'd been at the office (maybe due to my presence). Ultimately, the result was considered a success at the postmortem. From this story, I have learned to never ask to come back to the office during such an issue.