Responding To Customer Issues Without Heroics

Problem

"I joined a medium-sized startup with a enterprise data analytics offering about two years prior, as the Director of Quality Engineering (QE). Although the company had a Customer Success group, at the time, we didn't have an official process for escalating customer issues into Engineering. If there was an issue that needed to go to Engineering, tickets would be created, and these tickets would occasionally be reviewed by a product manager and were sometimes prioritized and sometimes missed. This ad hoc process resulted in slow turn-around for escalations and didn't live up to our 'User Impact First' company value."

Actions taken

"At the time, we didn't have a shared engineering team understanding of how quickly we wanted to react to customer issues, how much they mattered, how many resources we should spend on them, and how this would fit in with our larger quality strategy. I decided to lead by example, take on the tickets, and open up a direct channel to pass the tickets to my QE team to triage them, figure out a reproduction scenario for the bug, and potentially unblock the customer if we knew a workaround. If the issue needed a code fix, only then we would escalate it to development (with a great deal of debugging already done), and reach out to the product manager for help with prioritization. The big win was in building up trust between the customer success org and engineering. At first, we experienced a spike in the number of tickets being sent, as people realized that issues were being worked on and addressed. Over time we managed to smooth the flow of issues, keep up the responsiveness, and make issues visible. By doing so we reduced our average response time to mere hours, and time to fix to 7-14 days, an 8x reduction. 'User Impact First' indeed! We worked like this for close to a year, and became seen as 'the heroes' by the business side of the company, since they could trust QE to attend to their escalations. The downside was that my team truly took on a bit of a hero mentality, a feeling that it's up to us to fix everything. If product managers were too slow, and if developers were too overworked and couldn't take on triage, we would do it better. This was not a healthy attitude, and not a sustainable mode of operation. We quickly became overloaded and my team turned into a defacto customer escalation team. We became entirely reactive and had no time to work on strategic quality issues, testing, or building frameworks. Once I realized this problem, I worked to establish a proper on-call rotation to take care of the escalations, with developer buy-in and participation, because we were also shielding the developers from learning. They only knew what we had already figured out and passed on to them, and they didn't see the customers' pain points. Over the course of another year, we slowly started to take ourselves out of the picture. Eventually, we arrived at a mature on-call rotation, where developers triaged customer's questions and handled tickets, with QE organization in a supporting role rather than on the frontline. Having developers do this had an impact that QA could not. Our developers could see when there were a lot of issues coming from one area of code, and could then work to raise the quality for that entire area of code. My team's bandwidth freed up and we were able to work on some big tooling projects, and better prepare for the new product we were building."

Lessons learned

"When looking back on this journey, I still think I did the right thing by leading by example. This helped to show what could be done and how we could fix the situation. However, we spent too long in that role and we should have moved much faster to having developers respond to the tickets. You should show people how you can fix issues, and then step back to let them take control."