Back to resources

Outcomes Before Outputs: Measuring Engineering Performance

Impact
Productivity

14 September, 2020

Marian Kamenistak
Marian Kamenistak

VP of Engineering at Mews

Marian Kamenistak, VP of Engineering at Mews, explains why EMs shouldn’t be measuring the output of a team or individual engineers, but the outcome of the whole team.

Problem

As an EM who is managing a large number of teams, you are often struggling to follow up closely on how each team is doing whether it is about the pull requests or code reviews. EMs tend to measure various indicators, including the number of closed tickets, the number of commits, or the number of lines of code. In the end, none of those numbers will tell you how well the team is performing. Rather than measuring particular activities engineers are performing, you should be measuring outcomes or more specifically the number of epics and features delivered.

In addition, it would be helpful to create a system that would allow engineers to see how well they are performing at any given moment. This system would also provide a solid argument if and when other departments are complaining about the poor performance of the engineering team.

Actions taken

I established a system that measures the key five indicators based on real-time data. In a second, looking only at a dashboard, I could grasp if a team is doing well or not.

The first thing that I would measure would be a number of epics completed by a team in a month’s time. The reason I had chosen to measure a number of epics would be that they are high-level features listed in a roadmap.

Then I would assess how many epics are in progress for each and every team because if a team works on more than two epics at the same time, there is a lot of context switching and consequently, a lot of wasted time. Ideally, a team should be working on one, exceptionally two epics at the same time.

Following that, I would look at the lead time of features; lead time measures the amount of time that passes from feature planning to the deployment in production. Lead time can be measured in days, for example, the team’s lead time is 5 days on average. I would prefer to keep the lead time to 3 to 6 days. If lead time is constantly increasing that indicates the team’s inefficiency and may indicate a possible delay in feature delivery.

The next indicator is called a roadmap contribution ratio. It is displayed in a percentage; for example, a team has a 38 percent roadmap contribution ratio. That means that only 38 percent of tickets are tied to roadmap-related features, while the rest of the work goes to fixing bugs, or supporting customers. I try to keep a roadmap contribution ratio at about 55 percent in general.

Many managers are still using a number of pull requests or lines of codes to measure performance, yet I don’t see how these metrics can tell you anything about things you should care about the most -- epics and features. We shouldn’t be measuring the output of a team or individual engineers, but the outcome of the whole team.

In addition, I wanted to find a way to transparently put on view how well each team was doing. A dashboard I created would provide an accurate overview in less than five seconds. The board is used across the company but every department can choose the metrics they care about the most. We have a specific product engineering dashboard that shows the outcomes of our own team.

Lessons learned

  • Initially, I experienced some pushback from engineers who questioned the system and particularly indicators I selected, often complaining that it turns them into mere numbers. I tried to explain that those numbers are neither a tool to measure our success nor would be used to bully someone, but they serve as guardrails to streamline the team’s focus and stay on track. For example, if we would go off track, the dashboard would signal that by displaying different colors.
  • The system also changed how the engineering department was perceived within the company. It advanced transparency and it allowed any single person -- from top management to people from other departments -- to have a detailed insight into what the engineering team was doing.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

From Big Tech to Startup: Adding Value From Day 1

19 January

Angel Jamie, Chief Product Officer at Yayzy, recalls his transition from a well-established tech company to a sustainability startup, and the major differences he experienced.

Dev Processes
Company Culture
Impact
Team Processes
Cross-Functional Collaboration
Changing Company
Career Path
Performance
Angel Jaime

Angel Jaime

CPO at yayzy

How to Spark Sales-Driven Change

19 January

Nani Nitinavakorn, Sr Product Owner at Revolut, recalls her experience initiating a structural change to optimize her entire company.

Customers
Innovation / Experiment
Leadership
Meetings
Impact
Users
Nani Nitinavakorn

Nani Nitinavakorn

Sr Product Owner at Revolut

Analyzing a Problem for Real Causes and Coming to a Pragmatic Solution

7 January

Ranadheer Velamuri, Director of Engineering at Tesco, shares how he increased productivity by analyzing his problem and determining the best solution.

Alignment
Conflict Solving
Internal Communication
Productivity
Prioritization
Ranadheer Velamuri

Ranadheer Velamuri

SVP of Engineering at Locus Technologies

Decreasing Distractions During the Remote Workday

7 January

Ross Bruniges, Engineering Manager at Atlassian, shares his tips for a successful work-life balance, creating boundaries to decrease social distractions.

Salary / Work Conditions
Personal Growth
Productivity
Health / Stress / Burn-Out
Performance
Ross Bruniges

Ross Bruniges

Engineering Manager at Atlassian

Using Documentation to Increase Efficiency in the Remote Workplace

7 January

Kiran Bondalapati, VP Engineering at Snorkel A, describes his transition into the remote working environment at his previous startup and the challenges he overcame.

Remote
Meetings
Collaboration
Productivity
Feedback
Onboarding
Kiran Bondalapati

Kiran Bondalapati

VP Engineering at Snorkel AI

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.