Understand the History of Agile, or Be Doomed To Repeat It

Problem

Agile processes are much more than Scrum, and all Agile processes were a reaction to the failures of the software industry in the 90s and early 2000s. Given a historical perspective, one can see that Scrum is a good default starting point in today's world, but it's not flawless nor applicable to all situations, and it won't forever remain the industry standard technique. Agile and Scrum should be treated as imperfect tools, even if they are currently the best we have.

Agile, specifically Scrum, is sometimes seen as the Only Way to do software. If you don't ascribe to it 110% and follow all its tenements, then you're a heathen who must be an outcast. BEGONE! If you do not follow all the processes outlined in Scrum or SAFe, and defer to Agile in all things, then you are definitely Doing It Wrong.

Although Agile is the best default answer we have, this view is too restrictive and prescriptive. Processes work only in the context of the business, and one size does not fit all. Agile doesn't even pretend to solve all software issues - if you believe that it does, you're reading it incorrectly.

Applying any Agile with religious fervor is exactly not the point. Anyone who tells you otherwise does not actually understand the history or basic principles of Agile.

Actions taken

I have lived through the birth of Agile, including many of its early iterations such as XP, Pair Programming, the Spiral Method, etc. The Agile manifesto simply captured these under an umbrella philosophy that was a reaction to what the industry was failing at the time. Agile isn't the penultimate process or technique, it's one of the latest iterations of an ever-evolving process; it's not even the most revolutionary step.

I attempt to share this historical perspective whenever I join a new company, to help use Agile and Scrum as the powerful yet imperfect tools that they are.

In this section, I will be making some broad points, but they are best understood with specifics and with rigor. If you would like some more information on this, the two books I highly recommend are

• Agile!: The Good, the Hype and the Ugly, by Bertrand Meyer (2014) and
• a book from about 10 years earlier: Balancing Agility and Discipline: A Guide for the Perplexed by Barry Boehm and Richard Turner (2003).
  

Note that Meyer and Boehm had a huge impact before and after the Agile manifesto in agile software engineering practices. Meyer is one of the creators of OOP and the main author of the Eiffel programming language. Boehm is a founder of the Center for Systems and Software Engineering at the University of Southern California, and is the creator of the spiral method (1986), which heavily influenced agile methodology (the Manifesto was 2001).

Before Agile methods, software often ran grossly over budget and scope. The management style came from aerospace and telecom, where more specification and getting it right the first time was the emphasis. Note that this was when software was difficult to distribute and often came as shrink-wrapped physical media. There were some famously bad overruns such as many Microsoft operating system releases (NT, Vista) and the Denver airport baggage system. The demand for ever more specification was not creating stable or timely software; the more control that was imposed, the worse the outcome.

Reflecting on these failures of a growing number of managers in software engineering realized that less specification and more empowerment resulted in better results. Furthermore, continuous integration and testing solved some of the "big bang" integration pains that so often plagued the worst projects, and delivery of software over the internet or "over the air" meant that velocity of fixes was more important than getting it right the first time. New programming languages made continuous integration easier and testing more scalable.

By the mid-2000s, we had realized that this new, latest model, embodied by a whole host of Agile processes, usually worked much better given the new context.

Early on it was very common for software developers to be familiar with multiple methods of Agile: Scrum (of course), XP, Crystal, Kanban, and Lean were just a few. For some reason, in the 2010s we eschewed most other forms of Agile in favor of almost exclusively Scrum; I believe this was as much about marketing and training as it was practicality. Although Scrum is indeed a good default, it has some flaws and I strongly recommend that all PMs and engineers are familiar with at least three different Agile techniques. The books mentioned above review the most common Agile methods and their tradeoffs.

It's this understanding that both exalts Agile but also makes it more realistic: Agile is not perfect and it is a period on the continuum of history. It has solved many issues but it, in turn, has generated new problems; admittedly, it has solved many more historical problems than it has created, and we are lucky to have discovered it. Perhaps soon but certainly inevitably, yet another software methodology will evolve to become the gold standard as the industry learns more and product and software techniques evolve.

 

Lessons learned

Many companies are applying Agile processes without understanding the historical reasons why Agile was created. Without this perspective, Agile can be seen as a perfect process, when it was instead only a reaction to prior problems and an extrapolation of existing evolution. It is not a culmination, but instead is the most recent iteration in a long line of process improvements. Agile should not be treated as a fait accompli, but as the most recent best tool we have, and it should work for us, not we work for it.