Adapting Scrum to Continuous Deployment
17 February, 2021
Several years ago, I joined a development organization that operated on a very inefficient one-month sprint cycle and manual deployment process. This quickly made it the top of my list of things to improve. However, the existing leadership team was very risk-averse, felt the existing processes were good enough and did not want change.
Gathering data to support the changes
Undeterred, I observed the cycle for several months and documented the many pain points: urgent mid-sprint deploys for bug fixes, wasted developer downtime time at end of or in-between sprints, giant monolithic deploys that were hard to debug and rollback, error-prone four-day long manual QA regression, the pressure to hit sprint deadlines leading to painful last-minute shortcuts, a two hour, 25 people, cross-department pre-deploy “go/no-go” meeting and a manual, late-night deploy that took the product out of service. I took this laundry list of issues to the leadership team and got a conceptual agreement that the process needed to change, but no resources or official support.
Guerilla effort to demonstrate early results
Again, I was not deterred. I set a goal of enabling fully automated deploys at least once per day and created a roadmap to get there. I developed the first several milestones of the automated deploy pipeline myself on nights and weekends. At the same time, I worked with members of the QA team during their downtime at the start of each sprint to develop an automated QA regression test. I also developed and rolled out a distributed risk management process to eliminate the need for the 25 people “go/no-go” meeting.
The demonstrated value of the partially complete deploy pipeline and automated regression tests allowed me to elevate the automation project to an officially sponsored initiative. With the ability to commit dev teams to work on the project I was able to complete the deploy pipeline, extend the regression test suite and make the significant application changes required to allow a zero-downtime deploy.
Small experimentation and iteration
After achieving the goal of daily fully automated deployments, the heavy sprint cycle development process was the next bottleneck slowing us down. Over the course of a year, I experimented with process tweaks on a single team, rolling out each successful change to all the other teams. Eventually, the vast majority of the existing process was stripped away and replaced with a lighter weight process. This process, which we still use today, allows teams to deliver a predictable flow of work while retaining the ability to quickly react to production incidents and changes in project scope.
- Use data to get buy-in to change entrenched practices. While the solutions seemed obvious to me, data and demonstrated improvements were what really convinced leadership to back the project.
- Don’t settle when you believe you are right. Despite the lack of management support, I continued pushing the project because I believed in it. If I had given up, the organization would likely still be doing things the same, inefficient way.
- Start Small. When changing processes/tooling, test it out with the smallest unit you can (probably a team.) Iterate at that level and tune it before fanning out to a larger group.
Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader
Alishah Novin, Sr Program Manager at Microsoft, shares his experience creating a viral LinkedIn post that led to a specialized take on mentorship, micro-mentorship.
Sr Program Manager at Microsoft Corporation
Yang Wang, Engineering Manager at Bond, shares how diligently she transitioned from a large company to a series of startups, all balancing between the role of being a working mom.
Head of Product Engineering at Bond
Jason De Oliveira, CTO for more than 10 years, describes his methods of re-platforming an organization with nearly thirty years of existence using specific techniques and technologies.
Jason De Oliveira
CTO at Vialink
Jason De Oliveira, CTO for more than 10 years, shares his experience completing a reorganization, implementing agile, and collaborating with multiple teams.
Jason De Oliveira
CTO at Vialink
Saarthak Vats, VP of Engineering at noon, shares how he began setting realistic estimations, thinking about the complexity, number of unknowns, and scope of each project.
VP Engineering at noon