Back to resources

Why Rearchitecting Might Not Always Be the Best Idea

Deadlines
New Manager
Performance

8 September, 2021

Alfredo Fernandez Acuna
Alfredo Fernandez Acuna

Engineering Manager at ComplyAdvantage

Alfredo Fernandez Acuna, Engineering Manager, shares how he incrementally developed and improved the system, instead of rearchitecting it.

Problem

We identified that one of our systems could only handle a low number of transactions per second 一 we had about 30% of the performance that we needed. It was far off from what the Product team expected and what our clients wanted from the system. The success among this was that we made small incremental changes to the system, slowly winning performance improvements as opposed to rearchitecting the whole system. We had strong deadlines to work towards, which meant no time for working on the system from scratch. The only option left was to incrementally deliver those improvements.

Actions taken

To begin with, I was new to the team and did not have adequate knowledge of the system. I started by asking the team to document the main flows of the system, which helped in understanding the different components of the system, and the messages being exchanged between them, and whether they were synchronous or asynchronous. This allowed us to start discussions about what the main bottlenecks of the system were, and to assess the complexity in trying to improve those bottlenecks.

Following that, we created scripts for load testing, clearly identifying what was being simulated and what was being mocked. We ran load tests gradually incrementing the traffic levels in order to benchmark the system to know what the system could support on a daily basis. Also, we created stress tests that would tell us the breaking point of the system. After running the load tests we were able to analyse the results and determine in practice which were the real bottlenecks. These load tests, in particular the stress test, also helped us get quick feedback after implementing any improvements.

Thanks to this work we found some low-hanging fruit whereby we could have significant performance gains by simply adding more infrastructure, and by changing some tuning parameters of our Databases. We also identified a layer of caching at the application level that stores pre-computed results that would give us significant performance gains, as we would avoid repeated computations and prevent unnecessary disk access.

Lessons learned

  • Avoid the temptation to perform a massive re-architecture of the system or to re-design the system from scratch unless you have data to back it up. Rearchitecting the system costs a lot in terms of effort and could also delay your time-to-market significantly.
  • Have measurable and actionable goals. If the system does not meet its performance expectations, be sure to know what those expectations are in numbers.
  • Be pragmatic and evidence-driven. As engineers, our instincts are important, but you should always gather evidence or data to back your instincts up, and especially to convince stakeholders to your way of thinking.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

Managing Remotely: Balancing Team Cohesion and Focus Time

26 May

Jonathan Belcher, Engineering Manager at Curative, explains how to balance team cohesion and individual focus time, tapping into his experiences of working remotely for seven years.

Remote
Micromanagement
Meetings
Internal Communication
Productivity
Psychological Safety
Performance
Jonathan Belcher

Jonathan Belcher

Engineering Manager - Patient Experience at Curative

Leading Your Team in Stressful Situations

27 April

David Kormushoff, Director at Koho, recalls how he galvanized his team to tackle a time-sensitive problem, sharing his tips on how to shift chaos into calm.

Goal Setting
Leadership
Conflict Solving
Deadlines
Collaboration
Motivation
Strategy
Health / Stress / Burn-Out
David Kormushoff

David Kormushoff

Director at Koho

How to Foster and Reinforce Your Company Culture

25 April

Alex Bekker, Former VPE at Cresta and HackerOne, shines a light on how to preserve company culture throughout a growth phase and shares actionable insights on reinforcing your core values.

Mission / Vision / Charter
Company Culture
Meetings
Sharing The Vision
Performance
Alex Bekker

Alex Bekker

ex VP of Engineering at Cresta

The Challenges of Becoming a New Manager Among Old Peers

22 April

Neha Saha, Manager, Software Development Engineering at Workday, illustrates the challenges of obtaining a position in management with no prior experience and the confidence it takes in order to succeed.

Leadership
New Manager
Neha Saha

Neha Saha

Manager, Software Development Engineering at Workday

The Necessary Structures of Time Management

14 April

Suryakant Mutnal, Engineering Manager at PayPal, discusses the importance of time management and the necessary structures in order to create internal consistency.

Goal Setting
Managing Expectations
Remote
Deadlines
Productivity
Roadmap
Prioritization
Performance
Suryakant Mutnal

Suryakant Mutnal

Engineering manager at PayPal

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.