When Systems Thinking Unlocks a Way Forward
18 November, 2020
My previous company was undergoing re-architecture of our technology stack, making the transition from a monolith to microservices. Within this large effort spanning several years, my team was responsible for the data migration aspect of the problem. We had a proof of concept system in place, but the path to executing at scale and migrating customers en-mass was unclear. Not only was our strategy unclear, but upper management and stakeholders struggled to grasp what we were up to.
I sat down and listed a set of all steps that we would need to take to get a customer from using our old product (on our legacy stack) to successfully adopting our new product through migrating to our microservices architecture using my team’s data migrator. I tried to generate as many steps as I could think of and put all of them into a Google spreadsheet, making space for conversion rates of all of the steps like a growth funnel. Some of these steps involved human action like scheduling downtime with customers, some involved customer behavior like opting-in for early access, and some involved system performance like reliability of our data migration job.
Next, I identified the key variables that informed how customers move through this funnel. I gathered throughput data from our instrumentation in Datadog and customer sizes from our data warehouse to inform the starting values for system variables, and I worked with my PM to make our best guess at the human-centered variables. Putting this all together, I had a spreadsheet model that closely approximated our current situation and capabilities: exposing the key variables made the process much easier to reason about.
Though I started creating this model to better communicate the migration process to stakeholders and explain more clearly what my team was doing, it also helped me uncover a more serious problem: it became immediately clear that the clock time taken to migrate this large volume of data would dominate the project timeline. The team would need to invest in step-function changes in reliability and throughput to achieve the shorter timeline the business needed from us. I started to more heavily invest in projects that would get us more reliability sooner and urged the team to discover creative ways to migrate significantly less data to achieve throughput changes that we weren’t considering before.
Lastly, I worked with my manager to determine which parts of the framework were valuable for communicating with other stakeholders. Right away, our PM and customer support team gained a clear idea of how many customers they needed to recruit for early access for us to be able to ramp up. After validating my takeaways on how this may inform the team’s technical direction, my manager was able to translate this into a deck and story about our strategy for the executive team, bringing clarity to the rest of the organization.
- It is immensely valuable to think about problems like data migrations as a whole system rather than a set of technical components. Our process had human and technical aspects, and both needed to be explored to identify where the bottlenecks were. Revealing those bottlenecks made it clear where the team should invest in and gave us clear data-oriented targets for how we are going to measure our success over time.
- My manager was able to get me more support from other teams because he could more clearly communicate our strategy to his peers and the rest of the organization. I made sure he had a thorough understanding even when it was deeply technical and could have been abstracted.
Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader
Kushal Dalal, Director of Engineering at Suzy, stresses the importance of being well-informed when collaborating with colleagues in a previous period of transition.
Director of Engineering at Suzy
Jean-Benoit Malzac, Lead Product Manager at Ava, trusts his intuition when determining how best to communicate with a peer.
Lead Product Manager at Ava
Xun Tang, Engineering Manager at Twitter, has written the playbook on what it means to be a successful member of her team.
Engineering Manager at Twitter Inc.
Vinodh Sundararajan, Senior Engineering Director at Earnin, shares his experience restructuring a cross-functional team from the ground up, providing context and streamlining the work at hand simultaneously.
Sr. Engineering Director at Earnin
Manzar Kazi, Software Engineering Manager at LinkedIn, speaks of his experience of holding a partner team accountable with whom his team had multiple dependencies.
Software Engineering Manager at LinkedIn
You're a great engineer.
Become a great engineering leader.
Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.