How Metrics Can Help to “Disagree and Commit”

Problem

On the path to building the best online apartment rental marketplace, my company decided to differentiate from the competition by focusing on a five-star renter experience. We introduced a customer service call center: whenever a renter was trying to contact a property they would actually be connected to our customer service representative who would help them: answer questions, check availability, and eventually help get into the transaction and sign the lease. We hired 70+ call center reps, my team built custom software to make them efficient, and we were even searching for real estate in cheaper parts of the country to grow it further.

However, to someone with a technology background like myself, this strategy didn’t make sense for a few reasons. First, most modern companies tend to solve the problems in a scalable way with technology, while we were solving it with people - the most expensive asset. Second, our call center employees who were based in San Francisco couldn’t really answer real questions about the apartment community in, let’s say, Texas, and would have to forward the call to the property. Finally, cultural issues also started to emerge: a lean company was becoming bloated with a fast-growing headcount of call center employees, with all the company systems (HR, payroll, IT, security, etc) being stressed under this growth.

Actions taken

As a head of engineering initially, I took a “disagree and commit” stance and diligently executed on a strategy laid out by the leadership team. However, I also couldn’t shed the feeling that we are not doing the right thing. I knew that in order to bring up this existential question with the leadership team, I needed a lot more than just my opinion -- I needed to find data that corroborates it. Inspired by a book Freakonomics I started looking for a natural A/B test in our existing corpus of data. I spent a lot of time -- mostly after hours -- digging through the various metrics about our call center interaction in our data warehouse and trying to come up with a metric that would quantify the impact of our customer service center on the whole business model. My SQL skills were getting better day after day…

Finally, I had my “Aha!” moment. I was able to find a small subset of properties where renters were bypassing the call center. It allowed me to build a dashboard that compared transaction rates across these two classes of properties. It turned out that the transaction rate through the call center was not better, but actually slightly worse than for this set of properties. Data seemed to point out that the extra layer of interaction we forced on our renters was not helpful for the business. It was a start!

That was an interesting data set that I was able to bring to our leadership team and start a conversation. It was no longer just an opinion, it is a lot more difficult to argue with actual data. Collectively we designed several more A/B tests that allowed us to segment properties better, and understand how we can actually help renter where needed, and remove extra friction where not. Eventually, the call center program was scaled down a lot: we no longer needed large real estate, or custom call center software to maintain. The impact on the company path to the profitability of this change can not be understated.

Lessons learned

• Data wins arguments: I was really glad that I was able to pivot company strategy with the right metrics while maintaining or improving relationships with other leaders in the company.
• As a head of engineering, I could carry on executing the roadmap. However, I learned that leadership spans beyond your department. Engineering should play an important part in setting business and product vision. Without that, we risk working tirelessly on something that should not be done at all.
• I learned the true meaning of “disagree and commit”. If you passionately believe that the company direction is wrong, there are many constructive ways to bring it up (using data is one of them). You do not need to “forever hold your peace”.