Optimizing for Results Instead of Velocity

Problem

There was a point where we were introducing a new product to the market. Our first product was selling well and we wanted to expand to bring value to a wider demographic of users. We had a roadmap of things that would get people excited around this product, but, after a couple of quarters, we were struggling to make progress.

In general, the team was really good technically, but they were also overwhelmed by trying to balance enhancements to the existing product with our desire to scale and to build new functionality. The business was concerned that we were not moving fast enough. They came to the engineering team with questions concerning velocity. At the same time, the engineering team was feeling overwhelmed with all of this roadmap work. It was a chaotic time.

Actions taken

We did a couple of things in order to ensure the success of this product launch. It’s very difficult to optimize what you can’t measure. As a start-up, we weren’t perfect at recording all the work being done into tidy, categorized JIRA tickets, and we weren’t about to change any time soon. We introduced strategies for keeping track of how our engineering team spent their time based on Dropbox’s model, splitting work between non-elective necessities (fires, security issues, etc.) and elective initiatives (new features, existing features, and the like).

Every week, we surveyed every team to break down their week and used this information to understand the patterns that we noticed. This gave us a sense of how much time we were spending building our existing product. This creates the right type of conversation to have. It’s no longer about velocity. It’s about how much investment we should be spending on new things versus existing things.

The second strategy was introducing OKRs. Although OKRs are pretty universal at this point, they are also widely misunderstood. One thing that was really key in our implementation was that product and engineering shared their OKRs and measured their progress in relation to our desired business outcomes, as opposed to with product or engineering outcomes. Creating a new product or simply delivering to the roadmap are not good goals to be setting. What we cared more about were top-line items like revenue and product usage.

Our third initiative was being crazy about vertical iterations. Can we split the feature into smaller components, so that every component can then be shipped and measured for viability? With those versions, you can be really, really creative.

For example, when we were introducing new insights to increase usage into a part of our system, we realized building those insights into our product would take several weeks. We wanted to validate whether these insights would be useful at all before building them, and we came up with the idea of sending them by email and measuring the click-through rate as an easier way to get feedback from users. Being obsessive about being able to measure OKR results after every weekly iteration gave us critical feedback to end up delivering insights that our customers actually cared about, much more quickly.

After a couple of quarters, we had created the most successful product that we had ever built up until that point . We’ve been able to use this process as a template for how we bring new products to the table.

Lessons learned

• Time allocation, OKRs, and vertical iterations make up our playbook instead of velocity.
• Having this framework allows our employees to have candid conversations about our goals as we work. Every week, how are we getting closer to where we need to be?
• There is a time and a place for JIRA tickets and for crunching the numbers on how many features you were able to ship for the quarter. The OKRs show us what we are actually accomplishing on a weekly basis. So many engineers consider OKRs only in terms of their usefulness when planning for the long-term, which is also another really good way to use them. We take the notion to the extreme, bringing things down to the weekly scale, as opposed to the yearly scale.