Improving Peer Review Practice

Problem

A few years ago, at another company, I was working on a project where my team was building a training system for a piece of military equipment. It was a high-fidelity simulator for training the operators, and we were combined with another team responsible for building the engineering simulator, which is a low-fidelity platform for developing the software used in the system that we were simulating. We were building similar products, both for a simulated environment, but we were approaching the problem from different perspectives. My team was coming from the angle that we had to have a high-quality device that we could successfully train repeatedly and deploy customers and support over a long product lifetime. On the other hand, their team was focused on building something quick that they could use to easily test out changes to the software as it was being integrated. Our teams were combined because we were trying to build a single system that could do both of those things.

As our teams came together, we experienced a strong clash of cultures. My team had significant product experience and would put a lot of emphasis on building with high quality for the long run and maintainability, because we learned the cost of having low quality when we had to sustain a project over a long period of time. Their team’s concern was that a strong focus on quality practices would slow down development and get in the way of doing the rapid prototyping that everybody needed to do. We had to talk with their engineering leadership to figure out how to go forward. Their engineering leads were on board with our goals of introducing rigorous peer review practices, but their team was not.

We had to figure out how we could change the culture of their team to buy in to the level of quality review necessary for the product we're building for the long term. If we had two sets of standards and were working together, our collaboration would be doomed to fail. At the same time, we had to determine whether any of our quality practices were too rigorous and constrained and look back at what we were doing to make sure that we were not over-optimizing for quality.

Actions taken

Changing team culture can be hugely challenging, especially if it's something that the other team is resisting, unlike introducing something new that they've never tried before. Since there was active resistance to the practices we were trying to promote, we focused first on establishing a peer review culture and clarifying how engineers review each other's code to make sure that it would meet our shared standards. We approached it with an intent to optimize more for maintainability and shared ownership of the code.

Another difference between our two cultures was that our original team was a team building a product together, and we all had a shared sense of code ownership. We felt fairly comfortable working in all parts of the code and didn't feel very protective over any particular way that it was built. The other team tended to be a lot more focused on individual parts of the simulator. For example, one person would do the audio system, and few other people would understand it and be able to explain how it worked. They felt a lot of personal ownership, and that is always an obstacle to doing high-quality peer reviews. Anyone who has that high level of personal ownership (vs. shared ownership) is going to be very defensive whenever someone starts to make suggestions. Therefore, it was important to make sure that those things were built in a way that was more understandable for the whole team and well documented.

To address this problem, we did four things:

Embedding our team

We started by deeply integrating our two teams. Even though we had different goals, we were working together as a team, and we intentionally placed engineers from the two teams together so that they were working closely. We embedded our more senior engineers with their more experienced engineers so that we had a platform for promoting cultural change. They were working together on the same parts of the product and thus had a vested interest in sustaining good relationships.

Publicly demonstrating the good review practice

The next thing we did was to publicly demonstrate a good peer review practice. There are a number of things that make peer reviews effective, but the most important one is the willingness to say what you think about someone else's code. Equally important is the willingness to receive that feedback and process it productively, which means accepting suggestions and entering a discussion about how something was done, etc.

We were very intentional to do that in public for the team. My partner and I -- both working on a particular subsystem -- were very explicit and demonstrative with our peer reviews. “Here's the change that I'm going to suggest,” I would state, and then they would very vocally accept it, or disagree and discuss in the open. We had meetings with the team to go through those peer reviews to teach them how to do it and help them to see that a good peer review is not about someone telling someone else what to do or pushing their weight around but it's about building a common sense of what the right approach is.

We worked through a couple of weeks using several different examples to model good peer reviews and show them what shared ownership of the code implies and how feedback could be delivered graciously and graciously incorporated into the work. At the same time, we showed the value and the heightened quality standards that we were aiming for, in terms of having good documentation, being careful to check edge cases, etc.

Emulating the model

We tried to emulate the model that we demonstrated at the ground level. We wanted the other team -- that was still new to those practices -- to see that it was coming from the engineering tech leads and not just from the people they were working with on a day-to-day basis. We hoped that it would help them understand how our efforts were part of the bigger picture of how we were handling quality as a team.

The other tech leads and I were carefully observing how these things were playing out and very intentionally praising improvements and good practices. We were addressing cases privately when things weren't going very well so that people weren't put to shame for doing a bad job.

Emphasizing the value of good reviews

After we had let these practices soak in and practice them for some time, we started carefully observing and pointing out how our process had changed and improved because of our better quality. We were able to show the other team that we could move a lot faster now and absorb changes more rapidly. We emphasized to the team the value of good reviews and reinforced that they were worth doing, which helped culture changes to stick over time.

 

Over the course of the project, we saw how cultural change made a decisive difference in what we were able to deliver.

 

Lessons learned

• Understanding where the team was coming from and addressing the root cause of their past approach was critical. We could have just come in and said, “We're doing it this way. Everybody, get on board or get out,” but that would inevitably result in resistance, and we would not have been successful. We recognized that the team culture needed to change and not just the team practice, and we approached that from a cultural transition point of view instead of a practice or process imposition point of view.
• The continual reinforcement of the improvement made cultural changes stick. We praised improved practice as we went along and tried to connect it with the overall improvement of quality. Commending the team for the specific element of the work helped culture changes stay.