Transforming a Waterfall Team Into a High-Performing Agile Team

Problem

I joined a startup that approached development in a very waterfall manner - big upfront design and development cycles lasting quarters after extensive periods of product research. As a result, we delivered products that didn’t meet the market’s expectations - our products weren’t wanted and the long cycles meant we weren’t responsible to customer feedback even after release.

To change that approach, I had to tighten things up and introduce new team processes across the entire company and software development lifecycle. Also, the team had to undergo a mindset shift and reflect on how different processes could help them change as an engineering organization, enabling them to move faster.

Actions taken

To start with, I needed to look at our baselines with the intent of communicating the findings company-wide. I collected a number of metrics: lead time, cycle time, change failure rate, deployment frequency, etc. that would provide us visibility into current performance and trends.

I noticed that it would take us, on average, a month or two just to start working on something from the moment we realized that it was something we should be working on. These lead times represented customer requests going unanswered and were massive for a startup. Many of the items were relatively small wins that could have a tremendous impact on customer satisfaction, and we simply weren’t capitalizing on it.

Obtaining data-backed findings allowed us to make the problem known without laying the blame. In engineering, like elsewhere, presenting the problem without providing a solution is the least favored path. It can result in learned helplessness: people would have no understanding of how they could affect a particular metric or contribute to overall improvement. Therefore, the next step was to come up with a solution - a path forward in achieving the results we wanted to achieve.

I focused on improving our tooling to allow our engineers to deploy more frequently and safely. Our deployment frequency hovered around 0.25 / day, which meant that, on average, we deployed once every four to five days. I created a simple bash script to automate deployment and released instructions on how to use it. The team was quick to adopt this new process which made deployment significantly faster. We also improved our build time from 40 to 10 minutes, and we also improved our CI so that it could run in parallel, greatly decreasing the time it took.

Not surprisingly, our deployment frequency went up. However, the change failure rate also went up because we didn’t have safety and protection mechanisms in place. We started to track more meticulously our change failure rate as we introduced blameless incident post-mortems. We wanted to create a culture where moving fast and breaking things was welcome but should be done in a controlled manner that resulted in learnings.

Our efforts paid off. Over time, as we improved our CI, specs, and tooling, our change failure rate went down from 20 to less than 2 percent while our deployment frequency went up notably to 7 / day.

Once we started delivering faster, it was much easier to advocate for the change and execute on it. The results spoke for themselves, and it showed that the real bottleneck was upstream within the product organization. There was a significant amount of upfront planning for major projects and a lack of decision-making that left us not executing on anything.

I encouraged engineers to skim through the backlog and pick up the items they wanted to work on that they could move forward and deliver in a way that provided us with learning opportunities. Engineers started to take the initiative bypassing Product, and we began to deliver much more value.

The company started to take notice as well as our customers, and it was hard to argue with the results. Instead of analyzing the problem for three to six months, we would try something, learn from it, and iterate. We would iterate 10 to 15 times within a two-week period. We eventually reached a point where there was little need to over-analyze anything because we would get data from our continuous iteration that would inform our future decisions. That was the moment when the team became genuinely agile.

Lessons learned

• You can’t hold somebody to expectations if they don’t have the capability to meet these expectations. Sometimes, you’ll need to lower expectations in order to empower and motivate people. Setting expectations or goals people know they cannot reach just isn’t motivating. Creating the right tools, setting the direction, and improving capabilities should come before you start holding people accountable to a set of expectations.
• Having the right processes in place is incredibly important. But at the end of the day, if something you are trying to do goes against an established mindset, you won’t get far. You will need to do some shadow work and work around the blockades that are preventing you from introducing positive changes.
• Creating accountability for yourself is critical because you will go against established processes and will have to prove the new thing is more effective and beneficial. If you just go against the grain -- without solid results -- you will be labeled merely as a trouble-maker. You will need unarguable data to back up that you are doing the right thing.