Why Rearchitecting Might Not Always Be the Best Idea
8 September, 2021
We identified that one of our systems could only handle a low number of transactions per second 一 we had about 30% of the performance that we needed. It was far off from what the Product team expected and what our clients wanted from the system. The success among this was that we made small incremental changes to the system, slowly winning performance improvements as opposed to rearchitecting the whole system. We had strong deadlines to work towards, which meant no time for working on the system from scratch. The only option left was to incrementally deliver those improvements.
To begin with, I was new to the team and did not have adequate knowledge of the system. I started by asking the team to document the main flows of the system, which helped in understanding the different components of the system, and the messages being exchanged between them, and whether they were synchronous or asynchronous. This allowed us to start discussions about what the main bottlenecks of the system were, and to assess the complexity in trying to improve those bottlenecks.
Following that, we created scripts for load testing, clearly identifying what was being simulated and what was being mocked. We ran load tests gradually incrementing the traffic levels in order to benchmark the system to know what the system could support on a daily basis. Also, we created stress tests that would tell us the breaking point of the system. After running the load tests we were able to analyse the results and determine in practice which were the real bottlenecks. These load tests, in particular the stress test, also helped us get quick feedback after implementing any improvements.
Thanks to this work we found some low-hanging fruit whereby we could have significant performance gains by simply adding more infrastructure, and by changing some tuning parameters of our Databases. We also identified a layer of caching at the application level that stores pre-computed results that would give us significant performance gains, as we would avoid repeated computations and prevent unnecessary disk access.
- Avoid the temptation to perform a massive re-architecture of the system or to re-design the system from scratch unless you have data to back it up. Rearchitecting the system costs a lot in terms of effort and could also delay your time-to-market significantly.
- Have measurable and actionable goals. If the system does not meet its performance expectations, be sure to know what those expectations are in numbers.
- Be pragmatic and evidence-driven. As engineers, our instincts are important, but you should always gather evidence or data to back your instincts up, and especially to convince stakeholders to your way of thinking.
Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader
When you grow fast, its normal to focus on Value delivery aka "Feature Releases". Too many releases too soon will inevitably lead to piling tech debts and before you know, inefficiencies creep in, performances goes down, and ultimately any new release takes too long. Sounds familiar? Then read on..
VP - Engineering at ITILITE Technologies
Parallels between Work and Sport.
SVP Engineering at Trustly Group AB
Congratulations, you have just been promoted to an engineering management role. Once you are done celebrating the promotion you have worked hard to earn you might start to ask yourself, now what do I do?
AJ St. Aubin
Director Software Engineering at The RepTrak Company
Individual Contributors are familiar with a technical development framework that helps them with building products. Managers, especially new managers can leverage a parallel framework to help them build their teams while drawing analogies from an already familiar framework.
Viswa Mani Kiran Peddinti
Sr Engineering Manager at Instacart
Łukasz Biedrycki, VP of Engineering at BlockFi, talks about the importance of building on your strengths and finding your passions to maximize your impact. He dives into the tactics that managers can use to support their teammates in this pursuit.
Head of Engineering at Spectral Finance