Back to resources

How to measure Engineering Productivity?

Productivity
Prioritization
Performance

30 November, 2022

Ramkumar Sundarakalatharan
Ramkumar Sundarakalatharan

VP - Engineering at ITILITE Technologies

When you grow fast, its normal to focus on Value delivery aka "Feature Releases". Too many releases too soon will inevitably lead to piling tech debts and before you know, inefficiencies creep in, performances goes down, and ultimately any new release takes too long. Sounds familiar? Then read on..

What is Engineering Productivity?

As with anything you’re trying to improve, it starts with measuring the right data. So, you can actually track the right metrics. This data will form the basis of your analysis and baseline. I strongly recommend you don’t change anything about your current engineering process before you can collect sex weeks’ worth of data about your processes. If you start working on processes, you could end up with a Survivorship Basis.

You should have sufficient historical data to make comparisons. On top of that, most teams work in sprints of two weeks, so six weeks of data allows you to collect data for at least three different sprints. This will give you the allowances for any spikes and eliminate any unusual stress or slack on the execution.

Next, you should make gradual changes to the engineering process to see what improves or impedes the value delivery. It’s ideal to only implement one change at a time, so you can see the effect of each change, with all other things being equal. (it never is :D)

For example, if your engineering squads suffer from significant technical debt, you may want to build an additional stub related to feature completion. Every time an engineer completes a new feature, they must document the new feature. This could mean describing the feature, how is it built, what are the outcomes, how it interacts with other functions and the reasoning behind the design decisions.

By continuously measuring engineering productivity metrics, you can determine if this change has positively impacted the developers’ productivity.

How Is Engineering Productivity Measured?

There are potentially 100s of metrics you can measure for an Engineering Org. Here are four key metrics that will help you to get started with measuring engineering productivity. And I have consciously excluded the Sprint Velocity.

The "Prime Directives" of Engineering Metrics

1. The One Metrics to rule them all metrics – Cycle Time 

Software development cycle time measures the amount of time from work started to work delivered. It is a metric “borrowed” from lean manufacturing, and it is one of the most important metrics for software development teams. In plain speak, cycle time measures the amount of time from the first commit to production release.

2. The Oracle of an Engineering Leader – Release Frequency 

You should measure how often you deploy new changes to your customers (production). In addition, you can track deployments to various branches/instances, such as feature branches, hotfix branches, or QA branches. This data would show you how long it takes for a feature/fix to move through the different development stages. In addition, the Release Frequency reflects the throughput of your team. It’s a good stand-in replacement for Agile Velocity, so you don’t spook your Engineers and you are not blind as well.

3. The Guardrail – Number of Bugs

You should definitely track the number of bugs that your team has to resolve within 2 sprints of releasing a feature. This metric helps you to understand the quality of your code better. Higher-quality code should display fewer bugs after feature deployment.

While there are derivative and more evolved metrics like Defect Density, Mean Time to Detect (MTtD), Mean Time to Resolve (MTrR) and Code coverage, those onces makes sense after you’ve taken stock of and address the prime metric “ No: of Bugs” first.

If you want a more detailed list, methodology of QA metrics, refer the links given below. 

4. What is your “Blocker” – Review to Merge Time (RTMT)

This may look like a zoom-in on “Cycle time” metric we discussed earlier. But, in fact it is very different. In fact, it is an interesting metric suggested by GitLab’s development handbook. 

You should measure the time between asking for a pull request (PR) review and merging the PR. Ideally, you want to reduce the time a feature spends in the review state (or pending review state). A high RTMT prevents developers from progressing while they wait for feedback and encourages context-switching between different issues/features.

Arguably, Context-Switching is the highest productivity killer and should be avoided as much as possible

So, why would you measure all these engineering productivity metrics?

Why Is Measuring Engineering Productivity Important?

When you’re a “fast-growing startup”, it’s important to keep an eye on engineering productivity. It happens that these startups favour growth through feature delivery at the cost of effectively scaling the engineering team and ensuring the team’s efficiency.

I hear your question.

But, why does my CEO/VP/MD not understand?

Answer is simple

Assume you have to manage multiple VP’s expectations and outcomes (Sales, Marketing, Support etc), Company’s OKRs, and investors (or) board, will you have more time to dedicate to Engineering Productivity? Thats where you come in as an #EngineeringLeader, you'll have to balance it.

Any way, In these cases, technical debt can quickly grow, which will slowly kill your team’s productivity. Technical debt can have many negative consequences:

  • More bugs for your team to fix
  • Lower code quality—not only bugs but also worse code design
  • Harder to debug code
  • Scalability issues
  • A decline in overall happiness and job satisfaction

To avoid all of these scenarios, you should measure the engineering team’s efficiency and avoid technical debt buildup. Avoiding these problems before they occur is an excellent Occam’s razor. But addressing them head-on will have a significant impact on your organisation, both materially and culturally. 

In addition to preventing your team’s productivity from going down, the engineering productivity approach allows you to experiment with various approaches to try and improve throughput & efficiency. 

So, the goal is to improve the engineering process itself. For example, introducing new tools or applying new techniques. Next, you can measure the impact of these changes on your team’s productivity.

In the next part, I will write down on how can measurement improve engineering productivity, Stay Tuned!

References:

  1. Survivorship Bias. 
  2. https://www.masterclass.com/articles/survivorship-bias
  3. https://en.wikipedia.org/wiki/Survivorship_bias 
  4. Cycle Time
  5. https://tulip.co/blog/cycle-vs-lead-vs-takt
  6. Release Frequency
  7. https://community.atlassian.com/t5/DevOps-articles/Why-should-we-start-measuring-the-Release-Frequency/ba-p/1786430 
  8. Detailed QA Metrics to ponder (in addition to No: of bugs)
  9. https://reqtest.com/agile-blog/agile-testing-metrics/ 
  10. Review to Merge Time
  11. https://about.gitlab.com/handbook/engineering/development/performance-indicators/#review-to-merge-time-rtmt 
  12. Context Switching 
  13. https://pacohq.com/blog/guide/the-high-price-of-context-switching-for-developers/

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

"You don't care about quality" A story of single metric bias

3 February

This was not a high point in my career. It's a story of single metric bias, how I let one measure become a 'source of truth', failed to manage up and ended up yelling at one of the most respected engineers in my team.

Product Team
Productivity
Team Reaction
Alex Shaw

Alex Shaw

Chief Technology and Product Officer at Hive Learning

Myth Busting

10 December

Supporting principles on why being data led (not driven) helps with the story telling.

Alignment
Managing Expectations
Building A Team
Leadership
Collaboration
Productivity
Feedback
Psychological Safety
Stakeholders
Vikash Chhaganlal

Vikash Chhaganlal

Head of Engineering at Xero

How to measure Engineering Productivity?

30 November

When you grow fast, its normal to focus on Value delivery aka "Feature Releases". Too many releases too soon will inevitably lead to piling tech debts and before you know, inefficiencies creep in, performances goes down, and ultimately any new release takes too long. Sounds familiar? Then read on..

Productivity
Prioritization
Performance
Ramkumar Sundarakalatharan

Ramkumar Sundarakalatharan

VP - Engineering at ITILITE Technologies

Mindsets of High Performance team

14 October

Teams have tremendous impact on the products on they build. T.E.A.M definition - Together Everybody Achieves More is true. A collaborative and empowered team builds great product versus the good ones.

Innovation / Experiment
Mission / Vision / Charter
Building A Team
Productivity
Feedback
Motivation
Praveen Cheruvu

Praveen Cheruvu

Senior Software Engineering Manager at Anaplan

How I failed at my startup

14 October

There are nine specific building blocks and functional areas every org/company need to work to launch the product and provide services to customers. How effectively founders tackle them determine the destiny of the company.

Mission / Vision / Charter
Scaling Team
Building A Team
Impact
Strategy
Prioritization
Praveen Cheruvu

Praveen Cheruvu

Senior Software Engineering Manager at Anaplan