How Different Systemic Incentives Are Making Cross-Departmental Cooperation More Challenging

Problem

Years ago, I was tasked to develop software for a financial news company and we were required by contract to release our software every two weeks. I was engaged as a consultant and this particular company didn’t allow consultants to touch any production services, including automating the setup.

Every two weeks we had to place our Rails application in a zip file, create a trouble ticket in the system for the operations team and add it as an attachment to the ticket that had time displayed on it. At that exact time, an employee of the company who didn’t know how to deploy the software but had the correct access rights would double-click six icons on their open terminal windows and put them into each individual server that we had to deploy to. Then, we would double-click to get the access to all servers and, then again six times unzip them and six times have them run our setup script which is a hand-rolled shell script.

The process would take an hour and for an hour the entire site was down most of the time and sometimes more when we would have to manually reconcile database issues for hours.

Actions taken

After going through this three or four times and improving each time a bit of script, I knew we had to take another route. I approached my counterpart on the client’s side to put me in touch with the operations team because we were uncertain who was running the policies that were preventing us from automating the process.

We set up a meeting, sat down, and tried to explain our trouble with the software deployment system. They were surprised to learn that we were doing it the way we did and explained how they had a scripting language and a number of other things that would allow us to bypass the complications we entangled ourselves in.

We came expecting disagreement because the operations teams are usually incentivized for stability whereas development teams are usually incentivized for features delivery. Those different systemic incentives would often result in mistrust and disagreement.

However, we learned that they solved all those problems years ago and that we simply weren’t talking to each other. We were both coming to the conversation with the exact same presumption that the other side was doing something in a more complicated and unreasonable way. We both knew the process was broken and wanted to make it better, but we have never sat down and initiated any kind of collaboration to move things forward.

At that time the notion of continuous deployment was still not popular and this client worked on one to two months’ release cycle. The operations team was still not bought into the idea that more frequent releases would lead to more stable systems. So, we had to build trust in the frequency of the release cycle -- which was every two weeks -- and to figure out how to employ tools they had available to smooth out the release cycle.

It took more work than we anticipated but we did it. It was a gradual process and they were showing us what were available tools to automate the setup, how to use the deployment of the systems in their servers, and how we could work to smooth out the release process. Gradually, they bought into the notion of deploying every two weeks, because we were able to show them not only that the release cadence was high, but our release cycles were faster.

Lessons learned

• Sometimes you need to bypass the email communication and go physically talk to a fellow human being -- if possible -- and create understanding rather than assuming that people on the other side of the fence are your adversaries who don’t know what they are doing. Oftentimes you will have to be that first person to step up and build that bridge of understanding.
• Our perceptions are often shaped by the systemic incentives that we are given. No one wants to do a bad job but they are incentivized by different problems and rewarded in different ways. This shapes the way we approach our work and a large amount of cross-departmental mistrust has to do with the impact of systemic incentives that teams are given and not the impact of personalities and skills. For example, if you are a QA engineer you are incentivized not to let a bug through, you will naturally tend to build processes that avoid bugs getting into production by all costs. If you are on the operations team and you are incentivized to avoid getting paged you will build processes that prioritize stability at all costs. If you are a developer you will be incentivized to ship all the time and will not worry if there is a bug or someone gets paged.
• This is how the notion of cross-disciplinary teams emerged. Through my own experience, I learned that DevOps is not automation plus operations -- it is about building trustful relationships across people with different systemic incentives to achieve better team outcomes.