Proactive Alerting System

Problem

Being customer obsessed is key to a company’s success. But in B2B environments, in particular, managing customer escalations can be extremely disruptive as they can consume a sizable fraction of the product development capacity. As a head of product, I struggled to make time reliable commitments on my product roadmap because of these unpredictable impacts on the product and engineering team’s work caused by customer escalations. The challenge was to address customer issues, bugs, and malfunctions and be as organized as possible while doing it. The solution was to be proactively aware of those issues, rather than reactive, which did not solve everything but it definitely made things a lot easier.

Actions taken

We built an infrastructure that included two functions. One, it automatically monitored the state of a customer’s deployment, and two it gave each of those deployments a score. Everyday we would run the system and proactively detect changes in the level of engagement. Day by day and week by week, we would track the engagement of that customer and whether it was going up or down. Using a statistical approach we determined anomalies over a timed series, settling on whether a variation was the norm or if it was unusual. Any drastic abnormal changes would set off an internal alert.

This alerting system enabled timely proactive communication on emerging issues. Once the algorithm pronounced the engagement as strange behavior, it internally alerted our team so that individuals were aware of the situation before the customer had a chance to call. Based on automated detection of relative drops and surges in engagement over time, our team was sometimes able to solve the problem without the customer realizing it was even there. On the execution side, we created a lightweight alerting workflow compatible with existing CRM processes. We also built an MVP with Slack before considering integration with SalesForce.

Lessons learned

• Reducing time to awareness has its yin and yang effects. It’s soothing and brings you peace of mind because you know of problems as they happen, before your customer calls-in. As a result, you no longer encounter a surprise situation that escalates up the ladder until fingers are pointed at product with glaring eyes. You’re aware early on and that is extremely beneficial. But at the same time it’s stressful because now you have the information so you must do something with it. This increases the accountability aspect, which is another advantage in and of itself.
• Think globally - Act locally. Engage stakeholders early on in customer support, customer success, data engineering, product quality, product, and devops. This is a real project. You can MVP this and do it in stages but it is important to have the stakeholders in place in order for it to succeed. Think about the initiative, get the right people involved, and have a sustained resource plan in place. That’s how you’ll get it to work.
• Focus on both drops & surges. At first we were solely focusing our efforts on drops because that’s where we thought our attention was needed the most. However, you don’t want to always have to call the customer because of something that broke. Instead, also draw attention to the surges. A surge means that a feature or product is working, and that’s good news for all. This gives you the opportunity to talk with customers about the value the product is bringing, rather than concentrating only on the negative aspects. You can keep them informed and congratulate them on the use of the product. They’ll be happy to know how much you care.