We've just launched plato for individuals

🔥

login


Google Sign inLinkedIn Sign in

Don't have an account? 

Getting From What to Why

Dev Processes
Productivity
Impact

27 September, 2020

Pete Murray, Principal Software Engineer at Electronic Arts, explains how he successfully solved a long-troubling problem by applying a root cause analysis.

Problem

Some years ago, at my previous job where I was a Head of System Software Engineering, I was asked to review a project that was not happening as expected. It was a defense GIS project and the company was doing the hardware, software, and all the integration. The project was running two months late and the defect S-curve typical for any project was not appearing.
 

We did a lot of technical investigation where changes were happening and defects being introduced, but it wasn’t until we asked the Why and did a root cause analysis that we understood what the problem was about. We chose the technology that people couldn’t build on properly and eventually, ended up changing the technology and rewriting the whole software.
 

Actions taken

We delved into the source code -- we looked at all the changes and all the defects and then, linked all the changes to all the defects to identify the most vulnerable areas of the code. What we found out was that there was no vulnerable area of the code and that all the code was vulnerable. That contradicted the common understanding that all the defects should happen in a high complexity area. If that was the case, I would simply introduce a tighter control on that piece of software. But that was not the case; all the software was breaking.
 

The reason behind it was that the software architecture that was chosen -- called Dojo -- a client-side technology that created the system in a browser while the back-end was an API and all the aggregation was happening in a device. It was fragile and depended on a weakly typed javascript framework and building anything on a browser-based technology was like building on quicksand. Anytime you would change anything in this largely uncontrollable environment, you would get unpredictable results on the device.
 

I inspected closely all changes done in the last three months and chatted with people who did most of the changes on the project. I was curious about why people were making all those changes in the first place but their answers were reasonable and convincing. Then I looked for the point of fragility or the weakest link and after finding the one, I would usually either make it strong or re-architect it. But there was no weak link in the software, the weak link was in people and the fact that everyone was making the changes.
 

How did I do that? I went through the Five Whys Method, an interactive investigation method used to explore the cause-and-effect relationships underlying a particular problem. I would start with defining the problem and then successively asking Why it is happening. It was a very rudimentary form of root cause analysis, but it would get you to the root cause of the problem before completing all five Whys. In this particular case, the root cause was people. They were highly trained individuals, but they had chosen the wrong technology to work on.
 

Then, I had to prove what I discovered by using the Five Whys method.
I took one component of the code, brought in a developer who was not working on that particular project and asked him to rewrite that component in Java and Google Web Toolkit (GWT). With evidence of how easy it was to use different technology and ramp up the quality of the product, I went back to the GM who tasked me with solving the problem.
 

Having the Why conversation with a CEO or a VP usually starts in the middle, not at the beginning, and delivering Five Whys to busy and often impatient people can be a challenge itself.
 

In the end, we rewrote a project in a month that was dragging for a year and a half.
 

Lessons learned

  • Five Whys forces you to articulate a problem and find a root cause, but also identify all the steps you need to undertake. Knowing how you got to the solution can serve as a path for decision-making and communicating your proposal to your superiors. Without securing buy-in from my GM, I wouldn’t be able to go back to the team and tell them that we had to change the technology after a year and a half. To convince my GM I had nothing but my own credibility and a very powerful method that could turn a very complex explanation into an elevator pitch.
  • As an architect or principal engineer you should not choose the best architecture to build on principle, but the best architecture for your people to build things on.

Related stories

AI Products Are Data Products
19 October

Deepak Paramanand, Product Lead at Hitachi, shares how he built three different AI projects that all had one thing in common -- the ability to create or input data.

Product
Impact
Deepak Paramanand

Deepak Paramanand

Product Lead at Hitachi

Introducing Processes for Continuous Improvement
30 September

Peter Berg, Founder and CTO at Forward, recounts how he introduced processes for continuous improvement and thus creating a more psychologically safe working environment.

Impact
Productivity
Coaching / Training / Mentorship
Peter Berg

Peter Berg

Founder / CTO at Forward

The Quick Fix to a Slow Team: A Consultant’s Perspective
30 September

Peter Berg, Founder and CTO at Forward, describes how he helped ramp up a slow-moving team by applying his simple, yet expert approach.

Team processes
Delegate
Productivity
Agile / Scrum
Peter Berg

Peter Berg

Founder / CTO at Forward

Networking for Product Leaders: A Giving Mindset Approach
30 September

Caroline Parnell, previously managed product teams at O2 and Vodafone, emphasizes the importance of networking for product leaders and giving in return some value to her peers.

Personal growth
Collaboration
Impact
Caroline Parnell

Caroline Parnell

Most recently Head of New Product Innovation at Previously O2 and Vodafone

Ensuring Diversity of Thinking through Design Thinking Techniques
30 September

Caroline Parnell, previously managed product teams at O2 and Vodafone, shares some of the techniques she applied with her team to ensure diversity of thinking during product discovery workshops.

Product
Productivity
Internal Communication
Caroline Parnell

Caroline Parnell

Most recently Head of New Product Innovation at Previously O2 and Vodafone

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.