What Failures Can Teach Us: Migrating to Cloud-Native Microservices

Problem

This was my first time working on a cloud-based project where we had to move a monolith product to a microservices infrastructure. In hindsight, I could tell that we approached it too aggressively. We knew we had to break those monoliths down into pieces, but we didn’t know how small those pieces should be. Lacking the experience but being determined to deliver, we splitted the product based on functionalities into 14 different services. This looked fine until we realized how complicated and laggy it was when we standed up these services and tried to orchestrate them. And with no doubt this project failed due to the complication and unbearable long response time. But that experience was not only a valuable lesson in technology but in patience too.

Actions taken

“You don’t know what you don’t know” is a saying widely popular among engineers. Not surprisingly. We approached the migration with the best of intentions and with the knowledge we had at that moment. As a result, we ended up breaking microservices into each of the function pieces that were too small to be hosted as a service. The problem was that breaking them down into smaller pieces added significant complexity to the whole project. We were perplexed how we could manage and orchestrate those services. Even though we were using a lot of orchestration software like Kubernetes, we still ran into a lot of troubles.

In hindsight, I believe we went with over-engineering of the project. The project suffered from lagging because data was stored at different places, and it would take a lot of roundtrips to get some data, which made the whole optimization a nightmare. Also, the speed significantly decreased. Everything was simply too slow and with too many bottlenecks to maneuver around. It made it hard to manage, not to mention that we spent over six months working on that. We rushed into it, but as it seemed, we took a detour.

Knowing what I know now, the first thing I wished I did was consider the whole picture. I should have planned two years from now to be able to understand where we wanted to be and then assess if our current setup was going to take us there. I would be more patient and take more time to reflect and see if we were moving in the right direction. I felt that we should deliver fast, and thus I solely focused on implementation.

Lessons learned

• Microservice infrastructure is a great thing, but it is difficult to do it right if you don’t know what exactly you need. I hear many people saying that moving to microservice infrastructure only takes mapping the functions right. I don’t think that it is a feasible approach because it doesn’t match your needs. You have your own needs and understanding the big picture of where you are and where you should be in the future is critical. This is what we should have done at the beginning instead of being impatient and rushing into implementation.
• The pain point of microservice infrastructure is that it is far more complex to optimize it than what people think. Also, you should always utilize existing tools and infrastructure instead of reinventing the wheel.
• Don’t over-engineer. I keep telling this to myself as I often, like any other engineer, get carried away. The rule of thumb would be: if you need microservices -- go with it; if you don’t -- don’t do it. Don’t do it for the sake of doing it and because someone labeled it best practice.