When Systems Thinking Unlocks a Way Forward
18 November, 2020
My previous company was undergoing re-architecture of our technology stack, making the transition from a monolith to microservices. Within this large effort spanning several years, my team was responsible for the data migration aspect of the problem. We had a proof of concept system in place, but the path to executing at scale and migrating customers en-mass was unclear. Not only was our strategy unclear, but upper management and stakeholders struggled to grasp what we were up to.
I sat down and listed a set of all steps that we would need to take to get a customer from using our old product (on our legacy stack) to successfully adopting our new product through migrating to our microservices architecture using my team’s data migrator. I tried to generate as many steps as I could think of and put all of them into a Google spreadsheet, making space for conversion rates of all of the steps like a growth funnel. Some of these steps involved human action like scheduling downtime with customers, some involved customer behavior like opting-in for early access, and some involved system performance like reliability of our data migration job.
Next, I identified the key variables that informed how customers move through this funnel. I gathered throughput data from our instrumentation in Datadog and customer sizes from our data warehouse to inform the starting values for system variables, and I worked with my PM to make our best guess at the human-centered variables. Putting this all together, I had a spreadsheet model that closely approximated our current situation and capabilities: exposing the key variables made the process much easier to reason about.
Though I started creating this model to better communicate the migration process to stakeholders and explain more clearly what my team was doing, it also helped me uncover a more serious problem: it became immediately clear that the clock time taken to migrate this large volume of data would dominate the project timeline. The team would need to invest in step-function changes in reliability and throughput to achieve the shorter timeline the business needed from us. I started to more heavily invest in projects that would get us more reliability sooner and urged the team to discover creative ways to migrate significantly less data to achieve throughput changes that we weren’t considering before.
Lastly, I worked with my manager to determine which parts of the framework were valuable for communicating with other stakeholders. Right away, our PM and customer support team gained a clear idea of how many customers they needed to recruit for early access for us to be able to ramp up. After validating my takeaways on how this may inform the team’s technical direction, my manager was able to translate this into a deck and story about our strategy for the executive team, bringing clarity to the rest of the organization.
- It is immensely valuable to think about problems like data migrations as a whole system rather than a set of technical components. Our process had human and technical aspects, and both needed to be explored to identify where the bottlenecks were. Revealing those bottlenecks made it clear where the team should invest in and gave us clear data-oriented targets for how we are going to measure our success over time.
- My manager was able to get me more support from other teams because he could more clearly communicate our strategy to his peers and the rest of the organization. I made sure he had a thorough understanding even when it was deeply technical and could have been abstracted.
Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader
William Bajzek, Director of Engineering at Sapphire Digital, compares and contrasts a team structure that utilized siloed skill sets and one where everybody’s duties overlap at the edges.
Director of Engineering at Sapphire Digital
Adi Purwanto Sujarwadi, VP of Product at Evermos, shares how he identified the symptoms of his overworked product team and worked towards defining conflicting priorities.
Adi Purwanto Sujarwadi
VP of Product at Evermos
Neelima Annam, Sr Director Information Technology at Outmatch, shares how something as minor as collaboration tools can be a BIG issue during mergers and acquisitions.
Snr Director Information Technology at Outmatch HCM
James Engelbert, Head of Product at BT, shares how managing up is all about being an excellent manager to bring the best out of a team.
Head of Product at BT
Piyush Dubey, Senior Software Engineer at Microsoft, shares his journey of climbing up the career ladder through awkward times dealing with an introverted manager.
Senior Software Engineer at Microsoft
You're a great engineer.
Become a great engineering leader.
Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.