When Systems Thinking Unlocks a Way Forward
18 November, 2020
My previous company was undergoing re-architecture of our technology stack, making the transition from a monolith to microservices. Within this large effort spanning several years, my team was responsible for the data migration aspect of the problem. We had a proof of concept system in place, but the path to executing at scale and migrating customers en-mass was unclear. Not only was our strategy unclear, but upper management and stakeholders struggled to grasp what we were up to.
I sat down and listed a set of all steps that we would need to take to get a customer from using our old product (on our legacy stack) to successfully adopting our new product through migrating to our microservices architecture using my team’s data migrator. I tried to generate as many steps as I could think of and put all of them into a Google spreadsheet, making space for conversion rates of all of the steps like a growth funnel. Some of these steps involved human action like scheduling downtime with customers, some involved customer behavior like opting-in for early access, and some involved system performance like reliability of our data migration job.
Next, I identified the key variables that informed how customers move through this funnel. I gathered throughput data from our instrumentation in Datadog and customer sizes from our data warehouse to inform the starting values for system variables, and I worked with my PM to make our best guess at the human-centered variables. Putting this all together, I had a spreadsheet model that closely approximated our current situation and capabilities: exposing the key variables made the process much easier to reason about.
Though I started creating this model to better communicate the migration process to stakeholders and explain more clearly what my team was doing, it also helped me uncover a more serious problem: it became immediately clear that the clock time taken to migrate this large volume of data would dominate the project timeline. The team would need to invest in step-function changes in reliability and throughput to achieve the shorter timeline the business needed from us. I started to more heavily invest in projects that would get us more reliability sooner and urged the team to discover creative ways to migrate significantly less data to achieve throughput changes that we weren’t considering before.
Lastly, I worked with my manager to determine which parts of the framework were valuable for communicating with other stakeholders. Right away, our PM and customer support team gained a clear idea of how many customers they needed to recruit for early access for us to be able to ramp up. After validating my takeaways on how this may inform the team’s technical direction, my manager was able to translate this into a deck and story about our strategy for the executive team, bringing clarity to the rest of the organization.
- It is immensely valuable to think about problems like data migrations as a whole system rather than a set of technical components. Our process had human and technical aspects, and both needed to be explored to identify where the bottlenecks were. Revealing those bottlenecks made it clear where the team should invest in and gave us clear data-oriented targets for how we are going to measure our success over time.
- My manager was able to get me more support from other teams because he could more clearly communicate our strategy to his peers and the rest of the organization. I made sure he had a thorough understanding even when it was deeply technical and could have been abstracted.
JJ Fliegelman, CTO at WayUp, shares how he successfully set up a self-sustaining structure that allowed him to avoid the trap of being too in the weeds and being a blocker to the team, and instead focus on strategic objectives and opportunities.
CTO at WayUp
Sameer Kalburgi, VP of Engineering at Fieldwire, debunks the hidden meaning behind the recurring requests for transparency and shares how he managed to enhance collaboration with other stakeholders by drawing his team’s boundaries clearly.
VP of Engineering at Fieldwire
Raghavendra Iyer, Head of Engineering at ReachStack, explains how he envisioned a new product and engineering stack that he was trying to roll out for a yet-unknown problem.
Head of Engineering at ReachStack
Matt Pillar, VP of Engineering at OneSignal, shares how he improved the reliability of high scale systems by securing investment in infrastructure and on-call services.
VP Engineering at OneSignal
Matt Pillar, VP of Engineering at OneSignal, shares how he had to abandon a technology investment his team was pursuing that neglected the real customer problems and instead focused on the brilliance of the solution alone.
VP Engineering at OneSignal
You're a great engineer.
Become a great engineering leader.
Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.