Creating a Metric to Measure Progress in an Area of Ambiguity

Problem

Sometimes, everybody has a different idea of what they want to do in a given situation. Deciding which way of approaching a problem is best can be difficult to discern at times.

I was working for a company that sold real estate. We hired agents to sell homes and a big portion of our bottom line consisted of the commission that we took from each successfully-closed deal. My role involved converting visitors from passively browsing our listings to actually contacting one of our agents in order to visit the homes and eventually purchasing one.

The main way that this would happen was by scheduling a tour. The best thing that we ever did to get any of those millions of users to actually reach out in this way was by streamlining the scheduling process, making it fast and easy. They no longer had to pick up the phone and leave a bunch of messages. They pressed a couple of buttons and the appointment would be arranged.

One challenge that this presented was trying to determine how many of these appointments would lead to an actual sale, and how many of them were less serious in their intentions. One of the biggest costs that we had was paying the agents for their time, so the more appointments that were made frivolously, the more it would be cutting into our final profit. Not only that: it was frustrating for our agents to take time out of their days to waste time with these appointments.

It became difficult to figure out if the problem was systematic. How big of a deal was it, and what could we do on our end to minimize it?

Actions taken

We reached out to our agents in order to define a quality customer from their standpoint - somebody already pre-approved, for example. What we discovered was that this way of thinking was not in alignment with what our business was trying to achieve. We realized that part of our job was helping people to try to get to that point in the first place. Then, the task became figuring out where along that spectrum between both extremes we wanted to be. More barriers meant a surer sale, but would come at the cost of growth.

At a conceptual level, what exactly defines a customer that we want to serve? What is worth an agent’s time? What we decided on was that our ideal customer was somebody who was going to buy a home at some point, whether it was with our company or not. It didn’t have to be in a short timeframe, either.

We were then able to pull data on that, comparing the information on the people who would schedule a tour with us, and comparing that to property records that would show whether they had bought a home at any time afterward. We found that there were plenty of people who were scheduling tours and then never went on with the process at all, even several years down the line.

We weren’t sure, however, if that came as a result of the customers that we were bringing in, that sort of customer quality factor, or if it was because our agents weren’t reaching them in some way, making it more of a service problem. Seeing what our customers were doing after their tours was really how we started to pull those things apart. The next thing that we had to do was to come up with something that would be a good predictor of what that long-term behavior was going to be like. We considered a bunch of ideas from agents and the one that we ultimately landed on was looking at how often that person who had scheduled a tour came back to our website after the fact.

We saw that if they did so frequently in that first week or so after touring with us, they were really likely to actually go on and to buy a home, again, whether through our service or through somebody else. That got us to a place where we actually had this metric that we could evaluate through the course of the process. We could make a change to our online booking flow.

In the past, we’d always been able to see whether that got us more people going on tour or fewer, but in the case where we had fewer people going on tour, it was hard to say whether that was a good thing or a bad thing. It could be a really bad thing, because it could indicate that there was friction preventing people from cutting through, or it could be a good thing, because we were only retaining those higher-quality customers. Now, we had a solid metric with which to evaluate this difference.

Once we had found that, we were able to make so much more progress on this contact quality problem over the course of a couple of months, which followed years of intermittent effort before. We had turned a very ambiguous problem into something that we could measure and improve over time as a result.

Lessons learned

• In trying to evaluate what metrics were important, we learned that it was really valuable to gain qualitative insights from our agents who were actually in the field. We were able to ask them: how do you know whether or not a customer is actually serious about making a transaction? They were able to give us reliable indicators of serious interest, where before we were relying on more abstract metrics found at our end alone. We gained an understanding of why this metric was so consistent, which made it much easier to align the rest of the plan with their observations.
• When you have many employees bringing something to your attention, you can’t just ignore this problem that they all seem to be seeing across all of these different channels. It will keep bubbling up as a source of frustration. I’m a big believer in a sort of triangulation of incoming sources — here’s qualitative information coming from agents, and here’s some stuff from customers about it, too. Combining these things with your qualitative data will show where every source of insight is pointing, leading you to the parts of the problem that you should probably be caring about.
• This is how you are able to identify the issues that are taking the greatest toll on the organization overall; that was one of the biggest benefits that I was able to take away from this experience. It was pretty easy to tie this episode back to the financial impact that it was having on the company. The biggest cost to the company was always the cost of each agent’s time, so if they’re finding themselves wasting hours with customers who they have no chance of winning over, that’s going to really matter to the company’s bottom line.