Plato Elevate Winter Summit has been announced (Dec 7th-8th)

🔥

Back to resources

Why Rearchitecting Might Not Always Be the Best Idea

Deadlines
New Manager
Performance

8 September, 2021

Alfredo Fernandez Acuna
Alfredo Fernandez Acuna

Engineering Manager at The Workshop

Alfredo Fernandez Acuna, Engineering Manager, shares how he incrementally developed and improved the system, instead of rearchitecting it.

Problem

We identified that one of our systems could only handle a low number of transactions per second 一 we had about 30% of the performance that we needed. It was far off from what the Product expected and what our clients wanted from the system. The success among this was that we made small incremental changes to the system, slowly winning performance improvements as opposed to rearchitecting the whole system. We had strong deadlines to work towards, which meant no time for working on the system from scratch. The only option left was to incrementally deliver those improvements.

Actions taken

To begin with, I was new to the team and did not have adequate knowledge of the system. I started by asking the team to document the main flows of the system, which helped in understanding the different components of the system, and the messages being exchanged between them, and whether they were synchronous or asynchronous. This allowed us to start discussions around what the main bottlenecks of the system are and how easy it is to improve them.

Following that, we created scripts for load testing, clearly identifying what was being simulated and what was being mocked. We ran load tests gradually incrementing the traffic levels in order to benchmark the system to know what the system could support on a daily basis. Also, we created stress tests that would tell us the breaking point of the system. After running the load tests we were able to analyse the results and determine in practice which was the real bottlenecks. These load tests also helped us get quick feedback after implementing any improvements.

Thanks to this work we found some low-hanging fruit whereby we could have significant performance gains by simply adding more infrastructure, and by changing some tuning parameters of our Databases. We also identified a layer of caching at the application level that stores pre-computed results that would give us significant performance gains, as we would avoid repeated computations and also avoid disk access.

Lessons learned

  • Avoid the temptation to perform a massive re-architecture of the system or to re-design the system from scratch unless you have data to back it up. Rearchitecting the system costs a lot in terms of effort and also kills time for the client to see any value.
  • Have measurable and actionable goals. If the system does not meet its performance expectations, be sure to know what those expectations are in numbers.
  • Be pragmatic and evidence-driven. As engineers, our instincts are important, but you should always gather evidence or data to back your instincts up.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

Delegate successfully as a first time manager of Product Managers

24 November

Andrew Tsui, a Product Leader, works to build great teams that are independent, demonstrate mastery of their domain, and fully commit to their purpose.

Scaling Team
Building A Team
Delegate
Coaching / Training / Mentorship
Psychological Safety
Cross-Functional Collaboration
New Manager
Andrew Tsui

Andrew Tsui

Director of Product at Startup

Working From the Ground Up

23 November

Adam Hawkins, Site Reliability Engineer at Skillshare, uprooted an entire product and built it back up again with the help of his team.

Deadlines
Toxic Atmospheres
Adam Hawkins

Adam Hawkins

Site Reliability Engineer at Skillshare

Why Overloading Product Teams Never Work

23 November

Adi Purwanto Sujarwadi, VP of Product at Evermos, shares how he identified the symptoms of his overworked product team and worked towards defining conflicting priorities.

Managing Expectations
Product Team
Deadlines
Stakeholders
Adi Purwanto Sujarwadi

Adi Purwanto Sujarwadi

VP of Product at Evermos

Building trust as a new Manager

23 November

Neelima Annam, Sr Director Information Technology at Outmatch, shares her insight into her growth path of evolving her management style to build trust.

Alignment
Personal Growth
Conflict Solving
Coaching / Training / Mentorship
New Manager
Neelima Annam

Neelima Annam

Snr Director Information Technology at Outmatch HCM

Managing Team Collaboration After an Acquisition

10 November

Han Wang, Director of Engineering at Sonder Inc., shares the ins and outs of working successfully with the other half of the team after a merger.

Acquisition / Integration
Large Number Of Reports
Company Culture
Performance
Han Wang

Han Wang

Director of Engineering at Sonder Inc

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.