Back to resources

What Failures Can Teach Us: Migrating to Cloud-Native Microservices

Dev Processes
Embracing Failures

16 August, 2021

Richard Li
Richard Li

Senior Software Engineer Manager at ASSURANCE IQ

Richard Li, Senior Software Engineer Manager at ASSURANCE IQ, speaks of his failed attempt to move a monolith product to microservices infrastructure, as he outlines some valuable lessons.

Problem

This was my first time working on a cloud-based project where we had to move a monolith product to a microservices infrastructure. In hindsight, I could tell that we approached it too aggressively. We knew we had to break those monoliths down into pieces, but we didn’t know how small those pieces should be. Lacking the experience but being determined to deliver, we splitted the product based on functionalities into 14 different services. This looked fine until we realized how complicated and laggy it was when we standed up these services and tried to orchestrate them. And with no doubt this project failed due to the complication and unbearable long response time. But that experience was not only a valuable lesson in technology but in patience too.

Actions taken

“You don’t know what you don’t know” is a saying widely popular among engineers. Not surprisingly. We approached the migration with the best of intentions and with the knowledge we had at that moment. As a result, we ended up breaking microservices into each of the function pieces that were too small to be hosted as a service. The problem was that breaking them down into smaller pieces added significant complexity to the whole project. We were perplexed how we could manage and orchestrate those services. Even though we were using a lot of orchestration software like Kubernetes, we still ran into a lot of troubles.

In hindsight, I believe we went with over-engineering of the project. The project suffered from lagging because data was stored at different places, and it would take a lot of roundtrips to get some data, which made the whole optimization a nightmare. Also, the speed significantly decreased. Everything was simply too slow and with too many bottlenecks to maneuver around. It made it hard to manage, not to mention that we spent over six months working on that. We rushed into it, but as it seemed, we took a detour.

Knowing what I know now, the first thing I wished I did was consider the whole picture. I should have planned two years from now to be able to understand where we wanted to be and then assess if our current setup was going to take us there. I would be more patient and take more time to reflect and see if we were moving in the right direction. I felt that we should deliver fast, and thus I solely focused on implementation.

Lessons learned

  • Microservice infrastructure is a great thing, but it is difficult to do it right if you don’t know what exactly you need. I hear many people saying that moving to microservice infrastructure only takes mapping the functions right. I don’t think that it is a feasible approach because it doesn’t match your needs. You have your own needs and understanding the big picture of where you are and where you should be in the future is critical. This is what we should have done at the beginning instead of being impatient and rushing into implementation.
  • The pain point of microservice infrastructure is that it is far more complex to optimize it than what people think. Also, you should always utilize existing tools and infrastructure instead of reinventing the wheel.
  • Don’t over-engineer. I keep telling this to myself as I often, like any other engineer, get carried away. The rule of thumb would be: if you need microservices -- go with it; if you don’t -- don’t do it. Don’t do it for the sake of doing it and because someone labeled it best practice.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

A Day in the Life of a Product Lead in FinTech – A Series

31 January

Discover the daily struggles, challenges, and moments of delight encountered when delivering banking products around the world. I will share my story candidly and honestly, without filter as much as I am allowed, and offer insights into my approach while providing retrospectives of the results.

Strategy
Embracing Failures
Cultural Differences
Career Path
Loussaief Fayssal

Loussaief Fayssal

Director of CX at FLF PRODUCT DESIGN

The Not-So-Easy Guide on How to grow and develop an Amazing A-Team

5 December

Your Org Team may as well be a Sports team. Let's explore how this cohesive, multi-skilled team can be optimized for Great Group Playoff.

Alignment
Building A Team
Company Culture
Sharing The Vision
Embracing Failures
Team Processes
Jaroslav Pantsjoha

Jaroslav Pantsjoha

Google Cloud Practice lead at Contino

How to Successfully Rebuild Your Product

6 June

Adir Nashawi, Senior Product Manager at Hibob, shares his insight and experience from rebuilding a product to handle many feature requests and offerings.

Customers
Product
Dev Processes
Users
Prioritization
Adir Nashawi

Adir Nashawi

Senior Product Manager at Hibob

Navigating Disagreements When It Comes to Priorities

9 May

Pavel Safarik, Head of Product at ROI Hunter, shares his insights on how to deal with disagreements about prioritization when building a product.

Innovation / Experiment
Product Team
Product
Dev Processes
Conflict Solving
Internal Communication
Collaboration
Convincing
Strategy
Prioritization
Pavel Safarik

Pavel Safarik

Head of Product at ROI Hunter

Why You Should Take Technology Risks in Product Development

25 April

Matias Pizarro, CTO and VP of Residents at ComunidadFeliz, recalls a time in his early career when he took a technology risk that had wide-ranging benefits to his product's user experience.

Innovation / Experiment
Product
Scaling Team
Dev Processes
Matias Pizarro

Matias Pizarro

CTO and VP of Residents at ComunidadFeliz