Plato Elevate Winter Summit has been announced (Dec 7th-8th)

🔥

Back to resources

What Failures Can Teach Us: Migrating to Cloud-Native Microservices

Dev Processes
Embracing Failures

16 August, 2021

Richard Li
Richard Li

Senior Software Engineer Manager at ASSURANCE IQ

Richard Li, Senior Software Engineer Manager at ASSURANCE IQ, speaks of his failed attempt to move a monolith product to microservices infrastructure, as he outlines some valuable lessons.

Problem

This was my first time working on a cloud-based project where we had to move a monolith product to a microservices infrastructure. In hindsight, I could tell that we approached it too aggressively. We knew we had to break those monoliths down into pieces, but we didn’t know how small those pieces should be. Lacking the experience but being determined to deliver, we splitted the product based on functionalities into 14 different services. This looked fine until we realized how complicated and laggy it was when we standed up these services and tried to orchestrate them. And with no doubt this project failed due to the complication and unbearable long response time. But that experience was not only a valuable lesson in technology but in patience too.

Actions taken

“You don’t know what you don’t know” is a saying widely popular among engineers. Not surprisingly. We approached the migration with the best of intentions and with the knowledge we had at that moment. As a result, we ended up breaking microservices into each of the function pieces that were too small to be hosted as a service. The problem was that breaking them down into smaller pieces added significant complexity to the whole project. We were perplexed how we could manage and orchestrate those services. Even though we were using a lot of orchestration software like Kubernetes, we still ran into a lot of troubles.

In hindsight, I believe we went with over-engineering of the project. The project suffered from lagging because data was stored at different places, and it would take a lot of roundtrips to get some data, which made the whole optimization a nightmare. Also, the speed significantly decreased. Everything was simply too slow and with too many bottlenecks to maneuver around. It made it hard to manage, not to mention that we spent over six months working on that. We rushed into it, but as it seemed, we took a detour.

Knowing what I know now, the first thing I wished I did was consider the whole picture. I should have planned two years from now to be able to understand where we wanted to be and then assess if our current setup was going to take us there. I would be more patient and take more time to reflect and see if we were moving in the right direction. I felt that we should deliver fast, and thus I solely focused on implementation.

Lessons learned

  • Microservice infrastructure is a great thing, but it is difficult to do it right if you don’t know what exactly you need. I hear many people saying that moving to microservice infrastructure only takes mapping the functions right. I don’t think that it is a feasible approach because it doesn’t match your needs. You have your own needs and understanding the big picture of where you are and where you should be in the future is critical. This is what we should have done at the beginning instead of being impatient and rushing into implementation.
  • The pain point of microservice infrastructure is that it is far more complex to optimize it than what people think. Also, you should always utilize existing tools and infrastructure instead of reinventing the wheel.
  • Don’t over-engineer. I keep telling this to myself as I often, like any other engineer, get carried away. The rule of thumb would be: if you need microservices -- go with it; if you don’t -- don’t do it. Don’t do it for the sake of doing it and because someone labeled it best practice.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

How to Pivot a Product Idea at the Right Time

23 November

Adi Purwanto Sujarwadi, VP of Product at Evermos, shares how he diligently managed a product in one of the biggest eCommerce companies by being an individual contributor.

Innovation / Experiment
Product Team
Product
Embracing Failures
Adi Purwanto Sujarwadi

Adi Purwanto Sujarwadi

VP of Product at Evermos

The Right Way to Ship Features in a Startup

11 November

Matt Anger, Senior Staff Engineer at DoorDash, shares how he took the risk and shipped features in a startup.

Alignment
Product
Dev Processes
Matt Anger

Matt Anger

Senior Staff Engineer at DoorDash

The Problem-Solving Process: A Modern, Data-Driven Approach for Engineering leaders

28 October

Sudheer Bandaru, CEO at Insightly Analytics, recalls how he formed a company for carrying out data-driven solutions to daily engineering problems.

Dev Processes
Team Processes
Sudheer Bandaru

Sudheer Bandaru

CEO at Insightly Analytics

Taking The Lead As A Manager In Crisis

14 October

James Tobias, Senior Product Manager at Mapware, unveils a riveting journey to build a product from ground zero successfully.

Product Team
Product
Dev Processes
James Tobias

James Tobias

Senior Product Manager at Mapware

Strategic Ways to Stop Losing Customers

13 October

Mary Fisher, Software Engineering Manager at DrChrono, shares how diligently she worked with teams within her organization to retain customers.

Alignment
Innovation / Experiment
Product
Dev Processes
Convincing
Tech Debt
Prioritization
Mary Fisher

Mary Fisher

Software Engineering Manager at DrChrono

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.