When Solving Problems for Others Is Wrong

Problem

Some time ago, I was leading a team that had a strong dependency on another team. We were using one of their components that was causing intermittent failure. We were spending much time doing post-mortems of our incidents, documenting A to Z all things that would go wrong. The catch was that our problem was not a priority for the other team. Since they were not rushing to fix it, my team decided that we should do it for them.

Now, we have a company-wide culture where each team is responsible for its own tech debt. The organization leans towards being engineering-dominated rather than product-dominated from the perspective of who gets to decide on priorities. Yes, we could have fixed it, but it would go against everything we as the company stood for.

Actions taken

I decided to veto my team’s decision. I wanted to prevent them from fixing the other team’s problem. Instead, I escalated the whole thing, and by escalating it, I caused some friction between people on the other team and me. Eventually, it was inserted into their backlog, and they committed to fixing it.

The problem was that our team felt bad because we were measured against the failure of our components which were dependent on that other component. The pressure was mounting on us to fix the problem because everything downstream from us had a dependency on that faulty component. But the root cause of the problem laid with the other team.

I had to subject the team to a higher pressure than what they felt comfortable with because we needed to uncover that our dependency on the faulty component -- which we diligently documented -- was not something we should deal with. We were merely one link in that chain, and the pressure on us through downstream teams felt uneasy. That was the most challenging part: withstanding the pressure that was coming from downstream teams. We had to explain to them that we would not be able to move forward until the other team fixed the problem.

Given the unrest, the other team committed to fixing the problem based on the diagnosis we provided. For them, it was a matter of priority as they didn’t understand the effects their problem was causing to other teams. We could have fixed the problem and have a better outcome from a short-term perspective. But sometimes, you have to let these things happen because a workaround would cause more problems down the road.

Conway's law (of architecture) states that organizations’ design systems mirror their own communication structure. Essentially, your architecture will mimic your communication. When one crosses organizational boundaries and takes upon fixing other people’s problems, they are blurring those boundaries, which would cause more severe problems in the long run.

We don’t know what could have happened if we had fixed the problem. But the overall organization needs to learn to respect the boundaries. This is especially true for the fast-growing organizations where it could cause a mess in production. In the early startup mode, everyone tends to solve all the problems, resulting in a less clean architecture and design and problems waiting to escalate. To mature, one needs to learn to respect boundaries.

Lessons learned

• Be firm and withstand the pressure as long as you are convinced that what you are doing is right. Put your money where your mouth is; commit and hold the line. Even if it looks like causing pain to the organization, you are helping it mature and grow. You will have to withstand both internal and external pressure and some short-term pain for a more considerable gain.
• I didn’t stop my team from making a diagnosis because the other team was not doing it. It was important to understand and document what was happening and have that at hand; otherwise, it wouldn’t feel comfortable making that decision. Once the evidence is there that showcases the problem, you want to make sure that whoever is responsible for honoring the boundaries has a complete understanding.
• If you want to scale an organization and have as little as possible negative impact stemming from mutual dependencies, you need to honor a contract. Make people aware of how Conway’s law works in practice. If you blur the boundaries in communication, you will blur them in architecture too.