Plato Elevate Winter Summit has been announced (Dec 7th-8th)

🔥

Back to resources

Getting From What to Why

Dev Processes
Impact
Productivity

27 September, 2020

Pete Murray
Pete Murray

Principal Technical Director at Electronic Arts

Pete Murray, Principal Software Engineer at Electronic Arts, explains how he successfully solved a long-troubling problem by applying a root cause analysis.

Problem

Some years ago, at my previous job where I was a Head of System Software Engineering, I was asked to review a project that was not happening as expected. It was a defense GIS project and the company was doing the hardware, software, and all the integration. The project was running two months late and the defect S-curve typical for any project was not appearing.

We did a lot of technical investigation where changes were happening and defects being introduced, but it wasn’t until we asked the Why and did a root cause analysis that we understood what the problem was about. We chose the technology that people couldn’t build on properly and eventually, ended up changing the technology and rewriting the whole software.

Actions taken

We delved into the source code -- we looked at all the changes and all the defects and then, linked all the changes to all the defects to identify the most vulnerable areas of the code. What we found out was that there was no vulnerable area of the code and that all the code was vulnerable. That contradicted the common understanding that all the defects should happen in a high complexity area. If that was the case, I would simply introduce a tighter control on that piece of software. But that was not the case; all the software was breaking.

The reason behind it was that the software architecture that was chosen -- called Dojo -- a client-side technology that created the system in a browser while the back-end was an API and all the aggregation was happening in a device. It was fragile and depended on a weakly typed javascript framework and building anything on a browser-based technology was like building on quicksand. Anytime you would change anything in this largely uncontrollable environment, you would get unpredictable results on the device.

I inspected closely all changes done in the last three months and chatted with people who did most of the changes on the project. I was curious about why people were making all those changes in the first place but their answers were reasonable and convincing. Then I looked for the point of fragility or the weakest link and after finding the one, I would usually either make it strong or re-architect it. But there was no weak link in the software, the weak link was in people and the fact that everyone was making the changes.

How did I do that? I went through the Five Whys Method, an interactive investigation method used to explore the cause-and-effect relationships underlying a particular problem. I would start with defining the problem and then successively asking Why it is happening. It was a very rudimentary form of root cause analysis, but it would get you to the root cause of the problem before completing all five Whys. In this particular case, the root cause was people. They were highly trained individuals, but they had chosen the wrong technology to work on.

Then, I had to prove what I discovered by using the Five Whys method.
I took one component of the code, brought in a developer who was not working on that particular project and asked him to rewrite that component in Java and Google Web Toolkit (GWT). With evidence of how easy it was to use different technology and ramp up the quality of the product, I went back to the GM who tasked me with solving the problem.

Having the Why conversation with a CEO or a VP usually starts in the middle, not at the beginning, and delivering Five Whys to busy and often impatient people can be a challenge itself.

In the end, we rewrote a project in a month that was dragging for a year and a half.

Lessons learned

  • Five Whys forces you to articulate a problem and find a root cause, but also identify all the steps you need to undertake. Knowing how you got to the solution can serve as a path for decision-making and communicating your proposal to your superiors. Without securing buy-in from my GM, I wouldn’t be able to go back to the team and tell them that we had to change the technology after a year and a half. To convince my GM I had nothing but my own credibility and a very powerful method that could turn a very complex explanation into an elevator pitch.
  • As an architect or principal engineer you should not choose the best architecture to build on principle, but the best architecture for your people to build things on.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

The Right Way to Ship Features in a Startup

11 November

Matt Anger, Senior Staff Engineer at DoorDash, shares how he took the risk and shipped features in a startup.

Alignment
Product
Dev Processes
Matt Anger

Matt Anger

Senior Staff Engineer at DoorDash

Finding the Career Path for You

4 November

Joey Lei, Principal Product Manager at Kasten, shares how he reached the realization that he needed to pivot in his career path and changed course.

Managing Expectations
Personal Growth
Impact
Career Path
Joey Lei

Joey Lei

Principal Product Manager at Kasten

The Problem-Solving Process: A Modern, Data-Driven Approach for Engineering leaders

28 October

Sudheer Bandaru, CEO at Insightly Analytics, recalls how he formed a company for carrying out data-driven solutions to daily engineering problems.

Dev Processes
Team Processes
Sudheer Bandaru

Sudheer Bandaru

CEO at Insightly Analytics

Creating an Engineering Vision

25 October

Mustafa Furniturewala, VP of Engineering, uncovers how he created a succinct engineering vision with their company's values in mind.

Alignment
Mission / Vision / Charter
Goal Setting
Productivity
Mustafa Furniturewala

Mustafa Furniturewala

VP Of Engineering at Coursera

Taking The Lead As A Manager In Crisis

14 October

James Tobias, Senior Product Manager at Mapware, unveils a riveting journey to build a product from ground zero successfully.

Product Team
Product
Dev Processes
James Tobias

James Tobias

Senior Product Manager at Mapware

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.