Outcomes Before Outputs: Measuring Engineering Performance

Problem

As an EM who is managing a large number of teams, you are often struggling to follow up closely on how each team is doing whether it is about the pull requests or code reviews. EMs tend to measure various indicators, including the number of closed tickets, the number of commits, or the number of lines of code. In the end, none of those numbers will tell you how well the team is performing. Rather than measuring particular activities engineers are performing, you should be measuring outcomes or more specifically the number of epics and features delivered.

In addition, it would be helpful to create a system that would allow engineers to see how well they are performing at any given moment. This system would also provide a solid argument if and when other departments are complaining about the poor performance of the engineering team.

Actions taken

I established a system that measures the key five indicators based on real-time data. In a second, looking only at a dashboard, I could grasp if a team is doing well or not.

The first thing that I would measure would be a number of epics completed by a team in a month’s time. The reason I had chosen to measure a number of epics would be that they are high-level features listed in a roadmap.

Then I would assess how many epics are in progress for each and every team because if a team works on more than two epics at the same time, there is a lot of context switching and consequently, a lot of wasted time. Ideally, a team should be working on one, exceptionally two epics at the same time.

Following that, I would look at the lead time of features; lead time measures the amount of time that passes from feature planning to the deployment in production. Lead time can be measured in days, for example, the team’s lead time is 5 days on average. I would prefer to keep the lead time to 3 to 6 days. If lead time is constantly increasing that indicates the team’s inefficiency and may indicate a possible delay in feature delivery.

The next indicator is called a roadmap contribution ratio. It is displayed in a percentage; for example, a team has a 38 percent roadmap contribution ratio. That means that only 38 percent of tickets are tied to roadmap-related features, while the rest of the work goes to fixing bugs, or supporting customers. I try to keep a roadmap contribution ratio at about 55 percent in general.

Many managers are still using a number of pull requests or lines of codes to measure performance, yet I don’t see how these metrics can tell you anything about things you should care about the most -- epics and features. We shouldn’t be measuring the output of a team or individual engineers, but the outcome of the whole team.

In addition, I wanted to find a way to transparently put on view how well each team was doing. A dashboard I created would provide an accurate overview in less than five seconds. The board is used across the company but every department can choose the metrics they care about the most. We have a specific product engineering dashboard that shows the outcomes of our own team.

Lessons learned

• Initially, I experienced some pushback from engineers who questioned the system and particularly indicators I selected, often complaining that it turns them into mere numbers. I tried to explain that those numbers are neither a tool to measure our success nor would be used to bully someone, but they serve as guardrails to streamline the team’s focus and stay on track. For example, if we would go off track, the dashboard would signal that by displaying different colors.
• The system also changed how the engineering department was perceived within the company. It advanced transparency and it allowed any single person -- from top management to people from other departments -- to have a detailed insight into what the engineering team was doing.