Back to resources

How Different Systemic Incentives Are Making Cross-Departmental Cooperation More Challenging

Dev Processes
Cross-Functional Collaboration

30 September, 2020

Brian Guthrie

Brian Guthrie

VP of Engineering at Meetup

Brian Guthrie, VP of Engineering at Meetup, recalls the difficulties he endured due to a lack of cross-departmental cooperation caused by different systemic incentives teams were given to.

Problem

Years ago, I was tasked to develop software for a financial news company and we were required by contract to release our software every two weeks. I was engaged as a consultant and this particular company didn’t allow consultants to touch any production services, including automating the setup.

Every two weeks we had to place our Rails application in a zip file, create a trouble ticket in the system for the operations team and add it as an attachment to the ticket that had time displayed on it. At that exact time, an employee of the company who didn’t know how to deploy the software but had the correct access rights would double-click six icons on their open terminal windows and put them into each individual server that we had to deploy to. Then, we would double-click to get the access to all servers and, then again six times unzip them and six times have them run our setup script which is a hand-rolled shell script.

The process would take an hour and for an hour the entire site was down most of the time and sometimes more when we would have to manually reconcile database issues for hours.

Actions taken

After going through this three or four times and improving each time a bit of script, I knew we had to take another route. I approached my counterpart on the client’s side to put me in touch with the operations team because we were uncertain who was running the policies that were preventing us from automating the process.

We set up a meeting, sat down, and tried to explain our trouble with the software deployment system. They were surprised to learn that we were doing it the way we did and explained how they had a scripting language and a number of other things that would allow us to bypass the complications we entangled ourselves in.

We came expecting disagreement because the operations teams are usually incentivized for stability whereas development teams are usually incentivized for features delivery. Those different systemic incentives would often result in mistrust and disagreement.

However, we learned that they solved all those problems years ago and that we simply weren’t talking to each other. We were both coming to the conversation with the exact same presumption that the other side was doing something in a more complicated and unreasonable way. We both knew the process was broken and wanted to make it better, but we have never sat down and initiated any kind of collaboration to move things forward.

At that time the notion of continuous deployment was still not popular and this client worked on one to two months’ release cycle. The operations team was still not bought into the idea that more frequent releases would lead to more stable systems. So, we had to build trust in the frequency of the release cycle -- which was every two weeks -- and to figure out how to employ tools they had available to smooth out the release cycle.

It took more work than we anticipated but we did it. It was a gradual process and they were showing us what were available tools to automate the setup, how to use the deployment of the systems in their servers, and how we could work to smooth out the release process. Gradually, they bought into the notion of deploying every two weeks, because we were able to show them not only that the release cadence was high, but our release cycles were faster.

Lessons learned

  • Sometimes you need to bypass the email communication and go physically talk to a fellow human being -- if possible -- and create understanding rather than assuming that people on the other side of the fence are your adversaries who don’t know what they are doing. Oftentimes you will have to be that first person to step up and build that bridge of understanding.
  • Our perceptions are often shaped by the systemic incentives that we are given. No one wants to do a bad job but they are incentivized by different problems and rewarded in different ways. This shapes the way we approach our work and a large amount of cross-departmental mistrust has to do with the impact of systemic incentives that teams are given and not the impact of personalities and skills. For example, if you are a QA engineer you are incentivized not to let a bug through, you will naturally tend to build processes that avoid bugs getting into production by all costs. If you are on the operations team and you are incentivized to avoid getting paged you will build processes that prioritize stability at all costs. If you are a developer you will be incentivized to ship all the time and will not worry if there is a bug or someone gets paged.
  • This is how the notion of cross-disciplinary teams emerged. Through my own experience, I learned that DevOps is not automation plus operations -- it is about building trustful relationships across people with different systemic incentives to achieve better team outcomes.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

From Big Tech to Startup: Adding Value From Day 1

19 January

Angel Jamie, Chief Product Officer at Yayzy, recalls his transition from a well-established tech company to a sustainability startup, and the major differences he experienced.

Dev Processes
Company Culture
Impact
Team Processes
Cross-Functional Collaboration
Changing Company
Career Path
Performance
Angel Jaime

Angel Jaime

CPO at yayzy

Change Management in Engineering Organization

18 January

Joëlle Gernez, Vice President, Engineering at Pinger, shares how she collaborated her engineering team with the designers to bring about a change in the processes.

Architecture
Deadlines
Collaboration
Cross-Functional Collaboration
Joëlle Gernez

Joëlle Gernez

Vice President, Engineering at Pinger

The Role of Testing in the Engineering Product Development Process

18 January

Joëlle Gernez, Vice President, Engineering at Pinger, shares how diligently she made the existing developers a part of the testing process instead of bringing in new hires.

QA Team
Collaboration
Cross-Functional Collaboration
Joëlle Gernez

Joëlle Gernez

Vice President, Engineering at Pinger

Effective Cross-Functional Collaboration Between Technical and Product Leaders

4 January

Manu Gurudatha, Senior Director, Software Engineering at PagerDuty, shares how he established a trust-based relationship between product and engineering, which led to many successful product deliveries.

Product
Leadership
Collaboration
Ownership
Coaching / Training / Mentorship
Cross-Functional Collaboration
Manu Gurudatha

Manu Gurudatha

Senior Director, Software Engineering at PagerDuty

What an Engineering Manager Should Be Aware of in a SaaS-Based B2B Company

4 January

Manu Gurudatha, Senior Director, Software Engineering at PagerDuty, discusses how he rallied the teams to pivot their product development in a different direction at the right time successfully.

Product
Internal Communication
Collaboration
Team Processes
Cross-Functional Collaboration
Manu Gurudatha

Manu Gurudatha

Senior Director, Software Engineering at PagerDuty

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.