Back to resources

What Failures Can Teach Us: Migrating to Cloud-Native Microservices

Dev Processes
Embracing Failures

16 August, 2021

Richard Li
Richard Li

Senior Software Engineer Manager at ASSURANCE IQ

Richard Li, Senior Software Engineer Manager at ASSURANCE IQ, speaks of his failed attempt to move a monolith product to microservices infrastructure, as he outlines some valuable lessons.

Problem

This was my first time working on a cloud-based project where we had to move a monolith product to a microservices infrastructure. In hindsight, I could tell that we approached it too aggressively. We knew we had to break those monoliths down into pieces, but we didn’t know how small those pieces should be. Lacking the experience but being determined to deliver, we splitted the product based on functionalities into 14 different services. This looked fine until we realized how complicated and laggy it was when we standed up these services and tried to orchestrate them. And with no doubt this project failed due to the complication and unbearable long response time. But that experience was not only a valuable lesson in technology but in patience too.

Actions taken

“You don’t know what you don’t know” is a saying widely popular among engineers. Not surprisingly. We approached the migration with the best of intentions and with the knowledge we had at that moment. As a result, we ended up breaking microservices into each of the function pieces that were too small to be hosted as a service. The problem was that breaking them down into smaller pieces added significant complexity to the whole project. We were perplexed how we could manage and orchestrate those services. Even though we were using a lot of orchestration software like Kubernetes, we still ran into a lot of troubles.

In hindsight, I believe we went with over-engineering of the project. The project suffered from lagging because data was stored at different places, and it would take a lot of roundtrips to get some data, which made the whole optimization a nightmare. Also, the speed significantly decreased. Everything was simply too slow and with too many bottlenecks to maneuver around. It made it hard to manage, not to mention that we spent over six months working on that. We rushed into it, but as it seemed, we took a detour.

Knowing what I know now, the first thing I wished I did was consider the whole picture. I should have planned two years from now to be able to understand where we wanted to be and then assess if our current setup was going to take us there. I would be more patient and take more time to reflect and see if we were moving in the right direction. I felt that we should deliver fast, and thus I solely focused on implementation.

Lessons learned

  • Microservice infrastructure is a great thing, but it is difficult to do it right if you don’t know what exactly you need. I hear many people saying that moving to microservice infrastructure only takes mapping the functions right. I don’t think that it is a feasible approach because it doesn’t match your needs. You have your own needs and understanding the big picture of where you are and where you should be in the future is critical. This is what we should have done at the beginning instead of being impatient and rushing into implementation.
  • The pain point of microservice infrastructure is that it is far more complex to optimize it than what people think. Also, you should always utilize existing tools and infrastructure instead of reinventing the wheel.
  • Don’t over-engineer. I keep telling this to myself as I often, like any other engineer, get carried away. The rule of thumb would be: if you need microservices -- go with it; if you don’t -- don’t do it. Don’t do it for the sake of doing it and because someone labeled it best practice.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

How to Successfully Rebuild Your Product

6 June

Adir Nashawi, Senior Product Manager at Hibob, shares his insight and experience from rebuilding a product to handle many feature requests and offerings.

Customers
Product
Dev Processes
Users
Prioritization
Adir Nashawi

Adir Nashawi

Senior Product Manager at Hibob

Navigating Disagreements When It Comes to Priorities

9 May

Pavel Safarik, Head of Product at ROI Hunter, shares his insights on how to deal with disagreements about prioritization when building a product.

Innovation / Experiment
Product Team
Product
Dev Processes
Conflict Solving
Internal Communication
Collaboration
Convincing
Strategy
Prioritization
Pavel Safarik

Pavel Safarik

Head of Product at ROI Hunter

Why You Should Take Technology Risks in Product Development

25 April

Matias Pizarro, CTO and VP of Residents at ComunidadFeliz, recalls a time in his early career when he took a technology risk that had wide-ranging benefits to his product's user experience.

Innovation / Experiment
Product
Scaling Team
Dev Processes
Matias Pizarro

Matias Pizarro

CTO and VP of Residents at ComunidadFeliz

Fostering Emotional Safety as a Leader

13 April

David Pearson, Sr. Engineering Manager at Square, recalls his experience of reassuring a first-time manager and highlights the importance of emotional support.

Company Culture
Internal Communication
Psychological Safety
Embracing Failures
David Pearson

David Pearson

Sr. Engineering Manager at Square

Taking Healthy Risks in your Career

28 March

Karishma Babu, Founder at Upspeed, shares her experiences taking risks and making jumps in her career.

Goal Setting
Personal Growth
Conflict Solving
Embracing Failures
New Manager
Karishma Babu

Karishma Babu

Founder at Upspeed