Plato Elevate Winter Summit has been announced (Dec 7th-8th)

🔥

Back to resources

Understand the History of Agile, or Be Doomed To Repeat It

Agile / Scrum

27 January, 2021

Michael Smith
Michael Smith

VP of Innovation at Omnipresent

Michael Smith, VP of Innovation at Hadean Supercomputing, meticulously dissects the history of agile methodology and critically examines its current, uncritical use.

Problem

Agile processes are much more than Scrum, and all Agile processes were a reaction to the failures of the software industry in the 90s and early 2000s. Given a historical perspective, one can see that Scrum is a good default starting point in today's world, but it's not flawless nor applicable to all situations, and it won't forever remain the industry standard technique. Agile and Scrum should be treated as imperfect tools, even if they are currently the best we have.

Agile, specifically Scrum, is sometimes seen as the Only Way to do software. If you don't ascribe to it 110% and follow all its tenements, then you're a heathen who must be an outcast. BEGONE! If you do not follow all the processes outlined in Scrum or SAFe, and defer to Agile in all things, then you are definitely Doing It Wrong.

Although Agile is the best default answer we have, this view is too restrictive and prescriptive. Processes work only in the context of the business, and one size does not fit all. Agile doesn't even pretend to solve all software issues - if you believe that it does, you're reading it incorrectly.

Applying any Agile with religious fervor is exactly not the point. Anyone who tells you otherwise does not actually understand the history or basic principles of Agile.

Actions taken

I have lived through the birth of Agile, including many of its early iterations such as XP, Pair Programming, the Spiral Method, etc. The Agile manifesto simply captured these under an umbrella philosophy that was a reaction to what the industry was failing at the time. Agile isn't the penultimate process or technique, it's one of the latest iterations of an ever-evolving process; it's not even the most revolutionary step.

I attempt to share this historical perspective whenever I join a new company, to help use Agile and Scrum as the powerful yet imperfect tools that they are.

In this section, I will be making some broad points, but they are best understood with specifics and with rigor. If you would like some more information on this, the two books I highly recommend are

  • Agile!: The Good, the Hype and the Ugly, by Bertrand Meyer (2014) and
  • a book from about 10 years earlier: Balancing Agility and Discipline: A Guide for the Perplexed by Barry Boehm and Richard Turner (2003).

Note that Meyer and Boehm had a huge impact before and after the Agile manifesto in agile software engineering practices. Meyer is one of the creators of OOP and the main author of the Eiffel programming language. Boehm is a founder of the Center for Systems and Software Engineering at the University of Southern California, and is the creator of the spiral method (1986), which heavily influenced agile methodology (the Manifesto was 2001).

Before Agile methods, software often ran grossly over budget and scope. The management style came from aerospace and telecom, where more specification and getting it right the first time was the emphasis. Note that this was when software was difficult to distribute and often came as shrink-wrapped physical media. There were some famously bad overruns such as many Microsoft operating system releases (NT, Vista) and the Denver airport baggage system. The demand for ever more specification was not creating stable or timely software; the more control that was imposed, the worse the outcome.

Reflecting on these failures of a growing number of managers in software engineering realized that less specification and more empowerment resulted in better results. Furthermore, continuous integration and testing solved some of the "big bang" integration pains that so often plagued the worst projects, and delivery of software over the internet or "over the air" meant that velocity of fixes was more important than getting it right the first time. New programming languages made continuous integration easier and testing more scalable.

By the mid-2000s, we had realized that this new, latest model, embodied by a whole host of Agile processes, usually worked much better given the new context.

Early on it was very common for software developers to be familiar with multiple methods of Agile: Scrum (of course), XP, Crystal, Kanban, and Lean were just a few. For some reason, in the 2010s we eschewed most other forms of Agile in favor of almost exclusively Scrum; I believe this was as much about marketing and training as it was practicality. Although Scrum is indeed a good default, it has some flaws and I strongly recommend that all PMs and engineers are familiar with at least three different Agile techniques. The books mentioned above review the most common Agile methods and their tradeoffs.

It's this understanding that both exalts Agile but also makes it more realistic: Agile is not perfect and it is a period on the continuum of history. It has solved many issues but it, in turn, has generated new problems; admittedly, it has solved many more historical problems than it has created, and we are lucky to have discovered it. Perhaps soon but certainly inevitably, yet another software methodology will evolve to become the gold standard as the industry learns more and product and software techniques evolve.

 

Lessons learned

Many companies are applying Agile processes without understanding the historical reasons why Agile was created. Without this perspective, Agile can be seen as a perfect process, when it was instead only a reaction to prior problems and an extrapolation of existing evolution. It is not a culmination, but instead is the most recent iteration in a long line of process improvements. Agile should not be treated as a fait accompli, but as the most recent best tool we have, and it should work for us, not we work for it.

Discover Plato

Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader


Related stories

How to Build a Software Team from the Ground Up

12 November

Deepesh Makkar, Sr Director of Engineering at SunPower Corporation, shares his experience transitioning his organization from contractors to a 50/50 split of full-time employees and international vendors.

Hiring
Motivation
Cross-Functional Collaboration
Agile / Scrum
Deepesh Makkar

Deepesh Makkar

Sr Director of Engineering at SunPower Corporation

Choosing the Perfect Implementation of Agile

25 October

Shubhro Roy, Engineering Manager at Box, stresses the importance of the holistic nature of Agile methodology; picking and choosing Ă  la carte may cause more problems than it solves.

Goal Setting
New Manager
Agile / Scrum
Shubhro Roy

Shubhro Roy

Engineering Manager at box

Optimizing With Scrum

13 October

Phillip Derosa, Global Director of QA at OneSpan, takes Scrum seriously; he knows that the methodology is only as effective as its implementation.

Goal Setting
Agile / Scrum
Phillip Derosa

Phillip Derosa

Global Director of QA at OneSpan

How to Wear a Lot of Hats

13 October

Phillip Derosa, Global Director of QA at OneSpan, has never been afraid to fill in the cracks between departmental duties, often coming up with a brand new intermediary position to occupy during times of need.

QA Team
Leadership
Agile / Scrum
Phillip Derosa

Phillip Derosa

Global Director of QA at OneSpan

Powerful Reasons Why Goal Setting Is Important

12 October

Mary Fisher, Software Engineering Manager at DrChrono, shares how goal setting provides the foundation to drive an organization.

Goal Setting
Dev Processes
Deadlines
Productivity
Motivation
Cross-Functional Collaboration
Prioritization
Agile / Scrum
Mary Fisher

Mary Fisher

Software Engineering Manager at DrChrono

You're a great engineer.
Become a great engineering leader.

Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.