Handling A Security Breach

Problem

"I joined Datadog in October 2015 as the first designated security person. As Chief Security Officer, I was trying to hire for security roles, but hiring for security roles is really difficult, so even by July 2016 we only had a team of three. Unfortunately, on July 6th of 2016, I woke up to an encrypted PGP email from an individual saying they had seen unusual activity in their AWS account, which was tied back to the AWS credentials they had supplied to us. The only way this was possible was if either Datadog or that organization had leaked the key."

Actions taken

"I immediately called Alexis, our CTO, and we started assessing the issue. We realized one of the worst possible things that could have happened to us had happened. An attacker had breached our external security controls to gain access to our central data stores, which included all of our customers' credentials for their third-party services such as AWS. This was a fundamental breach of trust for our customers."

"Our security of team had very little experience in incident response and forensics and while we had very capable technology professionals who did what they could in terms of security up until that point, they weren't security experts. Our system was really complex, with thousands of machines running in Amazon, complex services and lots of credential management issues, and we had to work out what happened, whether the attackers were still present, how we were going to respond, and what we would do for our customers in order to minimize harm."

"We pulled all of the people who were central to our security practice at Datadog into a room together and quickly determined which systems had been compromised. Once we had this information, we started to examine and close off the routes the attacker had used. I then took a step back from the response effort and contacted a number of people I trusted in the security industry to identify resources we could bring in to support us. Through the efforts of the Datadog staff, it took us 24 hours of solid work to get to a point where on Saturday morning we realized the attacker probably didn't still have a foothold in our network. At that point, we had a hard decision to make in terms of notifying our customers and how much we were going to tell them."

"We decided to be more transparent than other organizations that had suffered similar incidents had been. I authored a blog post explaining the scope of the incident in excruciating detail and dealing with other implementation details for Datadog. However, I also discussed how we had been storing passwords correctly, so users were assured that their password details hadn't been accessed, only their third-party credentials were at risk. Because of the popular media narrative at the time, of big internet companies not storing passwords properly, a lot of the media reporting focussed on that, which helped our company's narrative."

"By the start of the next week, while the terror had ended, there was still a lot of technical work keeping us busy and our whole perspective had changed. Things that had been acceptable on July 5th were no longer acceptable. Over the next three months, we put a huge amount of engineering effort into changing the way people access AWS accounts and how we handle credentials."

Lessons learned

"When the breach first happened, I felt like it was cataclysmically awful and that it was going to destroy us. But often when things look bleak, they aren't as bleak as you think they are. Keep looking for the next step and moving forward and you'll often find your way through."

"While I didn't have an incident response team or forensic experts, I did have Site Reliability Engineers and Software Engineers, and by using their skills we were able to quickly identify issues and start working on fixing holes in our security. Don't let the fact that you don't have the specialist expertise hold you back from pursuing whatever your goal is."