The Agile Manifesto turned 20 this year, and there are two facts that seem self-evident:
- Agile, as a label, won; nobody wants to be called non-Agile.
- Agile, as it is practiced, falls woefully short of the revolutionary ideas of its founders.
How did we get to this point? Everybody says they do Agile and yet almost nobody is Agile.
Whence The Manifesto
In February, 2001, a group of seventeen expert software practitioners met at The Lodge at Snowbird ski resort in the Wasatch mountains of Utah. Over the course of a few days of discussion and debate, they collaboratively wrote the “Agile Software Development Manifesto”.
The first point to highlight is that these were practitioners. They weren’t project managers or CTOs or VP Engs. They were developers, programmers, scientists, and engineers. They were still slingin’ code* and working with their stakeholders to solve problems. This is important.
The other point is that the Agile Manifesto wasn’t created in a vacuum. Many of these people already had a methodology they had created and/or were proselytizing. I might have my timing slightly off, but I think all of these methodologies pre-existed “capital A Agile”: Extreme Programming, Scrum, DSDM, Adaptive Software Development, Crystal, Feature-Driven Development, Pragmatic Programming. I know Schwaber and Sutherland spoke publicly about Scrum in 1995, and Beck and Jeffries started talking about Extreme Programming (XP) in 1996, I think.
Everyone in this group had deep experience writing software, and they were all looking for an alternative to documentation-driven, heavyweight software development processes that ruled the day. The heart of the manifesto was four statements of value:
We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.
A New Hope
From our vantage point in 2021, it’s easy to take so much of modern software development practice for granted, but in 2001, these ideas were wildly radical.
What do you mean you’re going to start building the software before you’ve gathered all of the requirements and estimated every piece of functionality? That’s insane!
The important piece that gets forgotten is that Agile was openly, militantly anti-management in the beginning. For example, Ken Schwaber was vocal and explicit about his goal to get rid of all project managers – not just get the people off his projects, eradicate the profession from our industry.
Agility & PMI
We have found that the role of the project manager is counterproductive in complex, creative work. The project manager’s thinking, as represented by the project plan, constrains the creativity and intelligence of everyone else on the project to that of the plan, rather than engaging everyone’s intelligence to best solve the problems.
Ken Schwaber, manifesto signatory and Scrum co-creator
Scrum Masters had almost no authority, no votes on issues. They were servant leaders, helping shield and unblock the team but not managing the team. XP was similar. If I recall correctly, XP originally had trackers and coaches, which had a similar facilitating, supportive vibe.
Alistair Cockburn, manifesto signatory and creator of the Crystal methodology and hexagonal architecture, had a marvelous, insightful thread on this recently, including this idea (paraphrased):
Scrum struck a magnificent bargain in hostile territory:
- Management got 12 times per year to change direction in any way they wanted, after each sprint.
- Teams got an entire month of total quiet time with no interruptions or changes of direction to do heavy thinking and working.
- Teams got to announce what they could and couldn’t do in the month, with no management interference in their bid.
No executives ever got a better deal.
No development team ever got a better deal.
I’m a certified Scrum Master, working on Agile teams for 15+ years, and I’ve read many of the popular books in the space. I’ve never seen anyone frame the idea so explicitly and succinctly (again paraphrasing Cockburn):
Scrum was invented to function in hostile environments. It’s a contract between hard-pushing managers and developers needing time to think and explore.
The Empire Strikes Back
In some ways, Agile was a grassroots labor movement. It certainly started with the practitioners on the ground and got pushed upwards into management. How did this ever succeed?
It’s partially due to developers growing in number and value to their businesses, gaining clout. But the biggest factor, in my view, is that the traditional waterfall approach simply wasn’t working. As software got more complicated and the pace of business accelerated and the sophistication of users rose, trying to plan everything up front became impossible. Embracing iterative development was logical, if a bit scary for managers used to planning everything.
I remember meetings in the mid-2000s where you could tell management wasn’t really buying it, but they were out of ideas.
What the hell, let’s try this crazy idea the engineers keep talking about. We’re not hitting deadlines now. How much worse can it get?
Then to their surprise, it started working, kind of, in fits and starts. Teams would thrash for a while and then slowly gain their legs, discovering what patterns worked for that individual team, picking up momentum. After a few sprints, you’d start to see the real power of prioritizing working software, collaboration, taking time to inspect and adapt, and all the rest.
In the course of about 5 years, Agile went from a methodology that you’d heard of but probably not used to something everybody did. In 2005, I was switching jobs, and I remember the fact that I knew a bit about Agile and TDD was a real differentiator. By 2010, it was assumed modern software teams were doing some flavor of Agile. At least, this was true for my bubble in the consulting world; large enterprises always move slower.
🎉🎉🎉 We did it! We won! Congratulations everybody! 🎉🎉🎉
And that’s the end of the story. You can go ahead and close the browser tab.🥳
Winning was easy, young man. Governing’s harder.
George Washington, as portrayed in Hamilton
Unfortunately, like many revolutions, the history of Agile didn’t unfold how the founders envisioned.
- It turns out that prioritizing individuals and interactions is a hard concept to sell. It’s much easier to sell processes and tools.
- It turns out that working software is harder to produce than unrealistic plans and reams of documentation.
- It turns out that collaborating with customers requires trust and vulnerability, not always present in a business setting.
- It turns out that responding to change often gets outweighed by executives wanting to feel in control and still legitimately needing to make long-term plans for their businesses.
That doesn’t mean the four values are wrong. It just means this whole thing takes some effort to get right, some courage to accept that software is inherently messy and chaotic at times. You have to understand and believe that if you keep learning and adapting and improving and shipping, you’ll eventually get to a much better place, a much more honest and realistic and productive place than you ever could with waterfall.
The Agile movement is not anti-methodology, in fact many of us want to restore credibility to the word methodology. We want to restore a balance. We embrace modeling, but not in order to file some diagram in a dusty corporate repository. We embrace documentation, but not hundreds of pages of never-maintained and rarely-used tomes. We plan, but recognize the limits of planning in a turbulent environment. Those who would brand proponents of XP or SCRUM or any of the other Agile Methodologies as “hackers” are ignorant of both the methodologies and the original definition of the term hacker.
Jim Highsmith, History: The Agile Manifesto
Those are important points. We still need to plan and document and have rigor in Agile. It’s about balance. However if your organization is struggling with an Agile transformation, drowning in chaos, you’re going to leap when someone offers you a lifeboat in the form of certifications and processes and tools. Executives understand processes and tools much more than they grok self-organizing teams.
Return of the Rogue One?
This is where my three-act structure breaks down a bit, because unfortunately, I don’t see the plucky rebels coming back on this one, at least not under the Agile label.
It’s been overrun with tool vendors and process consultants and experts making promises that can never be delivered. This is how we’ve ended up with SAFe and Scaled Scrum and all of the other enterprise Agile flavors. These frameworks weren’t created with malicious intent, and they probably even have some value in the right context. But I wouldn’t call them Agile. Trying to scale a methodology that focuses on individuals and interactions will inevitably lead to problems – and erode the original value of the methodology.
This is how we’ve ended up with this famous 2018 piece by Ron Jeffries, manifesto signatory and XP co-creator.
Developers Should Abandon Agile
When “Agile” ideas are applied poorly, they often lead to more interference with developers, less time to do the work, higher pressure, and demands to “go faster”. This is bad for the developers, and, ultimately, bad for the enterprise as well, because doing “Agile” poorly will result, more often than not, in far more defects and much slower progress than could be attained. Often, good developers leave such organizations, resulting in a less effective enterprise than prior to installing “Agile”.
This is how we’ve ended up with the famous 2014 piece by Dave Thomas, manifesto signatory and Pragmatic Programming co-creator.
Agile is Dead (Long Live Agility)
The word “agile” has been subverted to the point where it is effectively meaningless, and what passes for an agile community seems to be largely an arena for consultants and vendors to hawk services and products… Once the Manifesto became popular, the word agile became a magnet for anyone with points to espouse, hours to bill, or products to sell. It became a marketing term.
So I think it is time to retire the word “Agile.”
Retro
The really sad part of this for me is seeing young developers denigrate Agile and think of it as a means for management to extract unrealistic promises and push dev teams to work crazy hours. I get it. The only Agile they’ve ever known has been a mechanism of control imposed on them, not a tool of self-empowerment they joyously embraced. But I hope kicking off some discussions around the history and original vision can help us remember how things were supposed to go.
The good news in all of this is that the principles of the Agile Manifesto are as wise and relevant today as they were 20 years ago. And even supposed apostates like Jeffries and Thomas still think that.
Jeffries in the article mentioned above, says, “However, the values and principles of the Manifesto for Agile Software Development still offer the best way I know to build software, and based on my long and varied experience, I’d follow those values and principles no matter what method the larger organization used.”
I agree.
It’s not hip or cool to talk about Agile now. Agile is boring. Everybody does Agile, right? Now is the perfect time to reflect on the last 20 years and ask ourselves a few questions:
- What went right?
- What went wrong?
- What do we want to do differently next time?
A few of us at Simple Thread who lived through the revolution plan to reflect on each of the original 12 Agile Principles over the coming months, contextualize their original meaning, and consider their value in the current milieu of software development.
My hope is that by studying the founding principles, we can learn from the past, and in the words of Dave Thomas, we can retain our agility even if we choose to abandon “Agile”.
Loved the article? Hated it? Didn’t even read it?
We’d love to hear from you.
Loved loved loved the article.
The history was spot on and as an SPC I live and breath this. Have asked developers to abandon agile and “let go”… use the force.
The references to the trilogy and then swapping in my favorite (R1) of all the movies was especially entertaining.
Shared it everywhere I know. Because sharing is caring. And will be following along these next 12 months.
Go Team!
Agreed; sharing it on linkedin without permission. Hope that’s OK.
Yes, indeed a great article very much enlightening. Shared it with every scrum master.
THE best article I’ve read on agile for a long time.
Thank YOU
Politics of Software Development
I’ve had students asking “Why agile has such a bad reputation? Is it really that bad?”
It’s a difficult question and most of the time I was presented with examples of how to do agile wrong…
This article is the perfect answer. Thank you!
Sharing immediately.
To paraphrase Mark Twain: “Reports of Agile’s death are greatly exaggerated.” This about the sixth article I’ve seen reporting as much.
I’ve worked at places where Agile was done well and where it was done poorly. Like any stick, it can be used as a powerful lever or a club.
Still, what’s missing from the article is the sheer torturous insanity of what came before: Top-down, Big-Design-Up-Front Waterfall development. You could spend 6 months to a year working on a project only to find out in the last month that none of the components worked together, half were totally unfinished but even when things worked there was no guarantee that you were delivering software that anyone would want or use.
Compared to that, even bad Agile at least has a chance to deliver features people want, adapt to setbacks and allow for course corrections. What’s probably failed is the industry of Agile which has tried very hard to formalize and “harden” Agile into a rigid discipline that works exactly the same in all situations, for all teams at all times. Agile is simple: Iterative, collaborative feature delivery. Mostly this works wonders but If management decides to turn sprints into micro-death-marches, they can and will. On the other hand, I’ve seen teams successfully use Agile as a powerful, virtuous wedge to “manage up” and gradually, diplomatically introduce realistic timelines and collaboration into old, rigid I.T. organizations.
Sure, software development may be due for new paradigm shifts and we need more stories about wildly successful and wickedly toxic Agile development to guide us but until something better comes along, I’ll champion Agile and dispel rumors of its death every chance I get.
I agree. Let’s not throw the baby out with the bath water.
This stuff takes WORK. Personal work. Organizational work. Team work.
Work is difficult. Change is hard. But its all worth doing.
With your comments, you made the argument balanced.
Agreed. Any flavor of Agile is better for modern changing-requirements projects. Too bad there was not any easy way to grade/know how close an “agile” implementation is to the original vision.
I quote you here: “Still, what’s missing from the article is the sheer torturous insanity of what came before: Top-down, Big-Design-Up-Front Waterfall development. You could spend 6 months to a year working on a project only to find out in the last month that none of the components worked together, half were totally unfinished but even when things worked there was no guarantee that you were delivering software that anyone would want or use.”
There are plenty of projects like this being delivered in Agile! But then of course everyone goes, “but that’s not Agile”. Which by the same logic means you can’t mention any failed Waterfall projects as Waterfall projects. You’re going to have to call them something else.
I am not anti-Agile or pro-Waterfall but what does amaze me is how any failed Agile project just isn’t Agile, it is “bad Agile”, “Agile poorly implemented” and many other things but somehow it doesn’t count.
The problem Agile proponents have is that when you count the failed Agile projects as actually failed projects one horrific statistic raises it’s head – the amount of failed projects under Agile is no different to the amount of projects that failed under Waterfall.
You’re terrible wrong and here’s why:
When you say “he amount of failed projects under Agile is no different to the amount of projects that failed under Waterfall” you put yourself on the customer’s side. And we all know that a software company should produce ……….., tadaaaaaa: money (not software).
Agile gets money on a regular base, independent of the quality, success or whatever the produced software has. On waterfall there always was the risk to remain unpaid, if your (more or less) final product is a failure.
I had enough of all the agile hypocrisy a while ago: https://www.linkedin.com/pulse/agony-agile-part-1-horatiu-ripa/
Exactly.
Any method can be applied beneficially or insane. Agile is not different. Waterfall neither.
“Still, what’s missing from the article is the sheer torturous insanity of what came before: Top-down, Big-Design-Up-Front Waterfall development. You could spend 6 months to a year working on a project only to find out in the last month that none of the components worked together, half were totally unfinished but even when things worked there was no guarantee that you were delivering software that anyone would want or use.”
Top-down, Big-Design-Up-Front didn’t fail because it was a bad idea. Compared to the undisciplined mess that software development was in before top-down development came in, it was a breath of fresh air – literally giving people breathing space to get the job done. Like Agile, it failed more because it wasn’t done properly than because of any inherent issues in the approach itself. Done properly, it is entirely consistent with the driving forces behind Agile with the benefit of imposing engineering discipline.
Royce’s famous paper that is supposed to be the source of “waterfall” development is far closer to Agile than most Agile proponents want to admit and did not prescribe waterfall development at all. In fact, the term “waterfall” only appeared as a straw man for agilists to have a go at.
Having been around for a similar amount of time a lot of your article resonated with me. I’ve been through the same or similar methodology and technological revolutions (4GLs, Java, Agile, mobile, etc) and seen the same sorts of people getting involved, making all the same mistakes and causing all the same problems.
From my experience it comes down to several things. Firstly management who are unable to leave people along to get their jobs done. As if they feel lost and alone when not constantly interrupting with meetings and applying control mechanisms to every single little task.
Secondly, there’s the issue that once something becomes popular it will become swamped with “experts” (usually with little or no real development experience) who are out to become rich or famous. They’re easy to spot, usually more than willing to go around telling everyone how they “should” be doing things and selling whatever shiny new methology they’re spooking at that time. They’re good at sounding wise and reasoned, telling everyone “they’re just there to help do things better” or some other BS whilst peddling dodgy, poorly thought out practices, patterns, tools or whatever.
But perhaps the biggest failure of all is a top down failure to recognise that no matter the technology, tools and methodologies, software development is highly skilled artisanal work. Sure we can give educated guesses as to time lines and deliverables, but it’s not something that can be quantified down to the day or lines of code. This is agile’s core message. Something that in my experience people who were around before it are far more likely to recognise and understand. Certainly I remember reading about agile and recognising many things I’d been doing for years because basically they worked.
“software development is highly skilled artisanal work”
Yes! Any practitioner knows that software development runs the gamut between being an artist at times and being a day laborer at others. Sometimes there is a lot of work to do that is easy to understand and do, but it’s monotonous. It’s not always trivial (or possible) to script, so we just have to sit down and do it.
Most of my time as a developer, however, is spent thinking about how to translate a particular requirement into something that works. Similar to artists and musicians, it’s impossible for us to quantify and provide estimates for inspiration. If I don’t know how I am going to do something, I can’t really give an accurate estimate on how long it will take to do. I can use past efforts as a guide, but project managers don’t want general guidance they want accurate dates.
Unfortunately for us, business leaders and scheduling managers have been sold on the lie that by using specific tools or processes, software development becomes a simple and predictable endeavor.
I enjoyed reading your article but I’m not sure I’ve got any answers, just my observations (and biases).
There’s always been something of the Wild West about software development and Agile is an attempt to tame the development process. Agile is ok in the round. However, the so called 12 principles are actually a mixture of platitudes, practices and principles. The fact that platitudes, such as the importance of customers, are raised to the level of principles tells you something about the mindset of people the manifesto addresses.
You say software development is chaotic at times and there’s truth in that. However, that’s not something you’d find in a traditional engineering industry where indiscipline could be costly or even deadly.
When I talk about software development I refer to applications in banking, commerce, government and public web sites, etc. I am not referring to industrial grade software found in aerospace, defence, industrial control, manufacturing, etc. I imagine the latter type have strict Quality Control standards. It is this lack of Quality Control, or even awareness of the term, that is, as I see it, the problem.
I’m sure that by now you’ve got my drift about the software industry’s inability to mature into a fully-fledged engineering discipline. There are professional bodies but these have failed to make serious penetration into the industry. Traditional engineering is full of them. You go to university to learn engineering principles. You go to professional bodies to learn about best practice. What you don’t do is teach software engineering practice at university and call it Software Engineering. Software engineering and software engineering practice are two very different things.
My first degree is in Electronic Engineering and Physics. I was able to see at first hand, through the laboratories, that these two disciplines require two different approaches to problem solving. I hope I won’t be accused of bias. However, it doesn’t help that most of the founders of the Agile movement are actually computer scientists (whatever that is, given that the computer has no science to it). If I wanted to do engineering I would not give the job to a scientist without first training him/her to be an engineer. There is more to engineering than the application of human ingenuity to the solution of problems. There’s something far more important: engineering practice. There’s nothing I’ve seen so far in the industry to suggest an awareness that practice is important. Engineering isn’t all about the product (Quality Assurance). It is also about the process (Quality Control). In that respect, Agile is not the solution because it does not embrace QC.
Finally, one thing that sets my teeth on edge:
The Waterfall Model, as I understand it, is some methodology such as SSADM (Structured Systems Design Analysis and Method). SSADM, as I found to my cost in my postgrad project, is downright dangerous. It has no place in engineering. When I enquire in discussions, however, what passes for waterfall is more of a cascade – a waterfall with small internal iterations. If that is the case then there’s nothing wrong with that ‘waterfall’. That’s the way engineering works. Instead Waterfall is just a term of derision used by some to promote their Agile wares.
Wonder, how comment would hold it’s ground after SpaceX 🤔
Oh. And also after good showcase of quality which is so important in the engineering industry. Example by Boing. That indiscipline, in my opinion, implied by chase of money and profits, was really costly. And even deadly.
And it’s sad to read, that Agile does not explicitly embrace Quality control.
For me this is written in between the lines:
A. “Working software over comprehensive documentation”
B. “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”
C. “Continuous attention to technical excellence and good design enhances agility.”
In the 70’s, my company did something like Agile called “seat of the pants” programming. Didn’t work. Users didn’t provide enough detail for successful programming. Systems were not getting delivered. We change the process. We went to the waterfall method that required design documentation that users had to sign off on. It was painful but systems were delivered, often late and over budget, but delivered. A rebellion occurred and we when back to the “seat of pants” method. The result, no systems were being delivered. The solution that the company decided was to buy vendor systems. However integrating these systems have proven painful. The moral of the story is that software is hard because it contains an irreducible amount of complexity.
i am down w/ peter’s and gerard’s comments.
agile is for the internet age and attention span.
add to gerard’s “industrial grade software” list: medical SW. who wants to run their pace-maker on CD?
agile fails/sucks because it has allowed us to shift responsibility to the user; which we happily have done.
You are correct in your observation that responsibility has shifted to the user.
One of the motivations for Agile is the need to adapt quickly to changing market conditions. Your boss owns an online insurance company. A visitor to your site needs to fill in a form with their details, submit the form and then wait a few hours for a quote to arrive by email.
In the meantime, a competitor has just released an instant online quotation calculator. Fill in a similar form and get a quote instantly. Now your boss is going frantic because he is losing market share. He comes to you and demands a similar feature.
If you were to follow best practice you’d deliver a solution in, say, a fortnight. However, this is unacceptable to your boss who’d like a solution by tomorrow. Otherwise, bang goes his annual sales bonus. So, being the ever obliging developer, you junk your best practice and knock up a solution in a couple of days. Great, you’ve just invented Agile.
One interesting side-effect of this way of building software is that you see a lot of code duplication. You dare not reuse existing code because it is undocumented (design documentation, rather than annotated comments within code) and dangerous to refactor. Besides, integrated regression testing takes time and is viewed as a digression and outside your sprint remit.
There is a place in the software development industry for Agile. It is lightweight and makes a good fit for instances, like the example I gave above, where you need to move fast. However, there is a price to pay. Your code gets bloated and rotten and needs to be junked every few years. But that’s ok anyway because of rapid changes in technology. Start afresh every few years.
Where I fail to see the need for it is with versioned software releases, for example. These are often open source or enterprise applications with a captive client/user base. There, there is no marketing imperative and quality of product trumps.
It is the un-nuanced promotion of Agile that I find perplexing. It is, in the eyes of its evangelists, a universal solution for software development.
No, Agile is not designed for “moving fast” it is for dealing with uncertainty – both in requirements and in time to execute. It does not sound like your version of Agile conforms to the practices of the Agile Alliance. It sounds like what you are complaining about is bad programming practice, which Agile does not address. I agree that is a real issue – poorly skilled and poorly disciplined developers will not produce good outcomes regardless of the work management philosophy employed.
The Agile that can be told is not the true Agile
You mentioned SAFe… Founders of the Agile Manifesto should sue them to get the ban on using the word “Agile” in the name. This framework has been created to please managers who still have their heads in late ’80s and want control over everything. The level of complication and misunderstanding is counterproductive and works against any engineers working on the project. And they earn money on selling this junk to large companies as remedy for all failures they had so far. Scary.
“You’re not doing it right!” : the typical response of zealots who see their cherished ideologies fail.
Exactly. If it is a failure it is not Agile. If it is successful however…
Or…. Take the criticism as constructive and be humble enough to consider, maybe it Is not being done right.
Since agile mindset is a tool, blaming the tools (esp if used improperly) is not constructive. Attacking someone who attempts to help that realization be known…
Great article. You brought together very coherently the jumble of thoughts and ideas of many experienced software developers.
There is a philosophy that project managers, management, and even many developers accept from the story of Toyota’s use of Total Quality Management based on the teachings of Dr. Deming. This philosophy lead to lean manufacturing and Six Sigma which teach us that the way to optimize a production process and maximize production output is by eliminating variability in process and ensuring predictability and consistency. Sounds great. Let’s apply that to software too! But that is a mistake. When we develop software we are not trying to produce the exact same output every time. In fact, it is the opposite; we want different output every time. This means that some of the principles, especially those endorsed by Six Sigma, are not helpful, and are perhaps even counter-productive when we want to produce outputs that are different every time. Defined processes such as Scrum, particularly along with a project management office or auditing team that requires all teams to operate with the same processes and tools, attempt to eliminate variability under the belief that elimination of variability increases productivity.
TQM, yet another indication of how the Agile fraternity misunderstands the different phases of traditional engineering. TQM is a production philosophy.
Weird that you’d apply a production/manufacturing technique to a design process. Just weird. Thanks for reminding me of this bizarre little oddity.
I disagree. A Toyota production line does not output the same thing every time. Each model has different versions, different styling, different trim, different sport packages, different colours, lots of different. But the principles are the same. Same for software development. I just see people shouting “it’s always different”, “don’t make us commit to any schedule”. The truth is if anyone spends a bit of time breaking down what they’re going to do they can estimate how long it will take. They can also build in time for encountering unexpected issues, just like on a Toyota production line.
In addition to the four value statements you discuss, the original site at agilemanifesto.org lists twelve principles. In my experience, it’s #11 that encapsulates the core of both the genius and the core challenge of Agile: “The best architectures, requirements, and designs emerge from self-organizing teams.”
Not surprisingly, managers at all levels in an organization prefer to believe that their roles are important. Agile, if done “purely,” diminishes that role – ideally out of existence, at least with respect to the production of software. As virtually all organizations have a hierarchical power structure, with decision-making authority concentrated at the top, the concept of “self-organizing teams” is, almost invariably, a pretense, occasionally mentioned in lip service. The people who do the actual production rarely have any real voice in the major decisions. Of course, this condition is invariably justified in any number of ways, but the bottom line is that power is distributed in inverse proportion to expertise.
If you think that’s a good idea, perhaps you would be happy if a hospital administrator decides what gets removed during your next surgery.
Very true and in addition for the quoted principle:
“The best architectures, requirements, and designs emerge from self-organizing teams.”
— you’re also assuming that once you tell a team to self-organise that they’re good at self-organising. Definitely not a given. Some teams will be great, others, not so much.
Really good article, thank you. Am sharing with my team.
For all the pain and half-kept promises of the original radical dream, agile did pave the way for lean principles to make their way from manufacturing into software delivery. That’s a win, even if it’s perhaps a lot less than we hoped for.
Re: empowered teams, my observation after spending time as both a dev & a manager:
Dev: “I want more ownership!”
Translation: I decide what to work on and when or if it ships. You’re accountable for the commercial impact of my decisions on the company.”
Manager: “I want you to take more ownership!”
Translation: I tell you what to build and how long it will take. You’re accountable for making my deadline and for the success of my idea with customers.”
Right now we have flocks of developers who don’t like writing documentation, and stakeholders who don’t like paying for the time needed to write documentation, all claiming that “shipping working code is more important than having documentation.” First off, many times they are using the term “working code” as a euphemism for more bright and shiny features and not reliable core functionality. People were writing working, reliable code that shipped with useful and effective documentation just fine before agile. Anyway, these people with a biased dislike of writing documentation have now found a codified excuse not to do it anymore. Meanwhile, never once have I heard a customer say “You know, I really wish there were less documentation.” Never once when I’ve spent days reverse engineering someone else’s Rube Goldberg machine did I think to myself “You know, I really wish they hadn’t documented this code so well.” To the contrary, I see users struggling to understand and effectively use software and having to sift through a plethora of poor quality and incomplete youtube videos instead, if they are lucky enough to find any. I see countless development hours wasted trying to not only figure out what legacy code does, but what the author intended it to do. Sure, maybe users are getting more features more quickly this way, at least at first in newer code, but as software matures and the vast majority of truly necessary functionality has already been coded, they are just getting more and more esoteric features that the vast majority of them will never need or use in exchange for losing the documentation they do need. Developers are being saddled with undocumented code that has become so cumbersome to work with that it’s just cheaper to throw it away and start over, reinventing the wheel and redoing the same work over and over again. What a waste. But the Agile cult churns on blindly like every other cult, refusing to question the foundations of its own ideology which in reality are just the opinions of a handful of founders, opinions that were never really investigated, verified or ratified by any kind of scientifically rigorous process. What would you think if architects who build skyscrapers one day said “Hey, instead of making all of these blueprints lets discard this “blueprint-centric” mentality and adopt agile instead. We’ll build skyscrapers one floor at a time, completing each floor as we go and let people move in immediately. Then well just figure out the rest of the floors as we go along.” And yet many software architects seem fine with the idea.
One of the four values statements captures that mindset:
Working software over comprehensive documentation
Still, it is an advance on one of the Agile founders’ dictum that documentation is unimportant because no one reads documentation. The other variant of this is to claim that good code is self-documenting. In the world of Agile, the expression ‘RTFM’ is taken as an affront and an insult.
Anyone who has ever worked in engineering will know that you don’t get anywhere without documentation. It is the cornerstone of good engineering practice. At a minimum, I’d like to see the following:
1. Conceptual
2. Design
3. As Built
The third, an ‘As Built’, is what really nails your work. And while you are at it, I’d like to see loads of diagrams. I know this is all a pain in the backside to the genius developers and code warriors out there who think documentation is an impediment to the flow of their creative juices.
“Still, it is an advance on one of the Agile founders’ dictum that documentation is unimportant”
But they didn’t say that. In the manifesto, they write:
“Working software over comprehensive documentation”
and then:
“while there is value in the items on the right, we value the items on the left more.”
Many people have overly simplified the core message and assume that it implies “documentation is unimportant”, and I think it’s been used as a strawman to show that Agile is an impractical pipe dream of entitled engineers. And that’s a shame.
This reminds me of a meme I saw yesterday:
Scientists: Our research is worthless if quoted out of context.
Media: Scientists say their research is worthless.
Seems somehow similar to developers “quoting” the Agile Manifesto saying that documentation is unimportant.
And people are not reading the comments with a quoted directly fromsource to realize that there is no contradiction.
It is just trolling.
An interesting article. As a retired software engineer, I have seen many such “fantastic” new concepts come and go. One of my former colleagues (now also retired) put the concept of software engineering very simply:
I can produce a system for you quickly
I can produce a system for you cheaply
I can produce a system for you that is robust.
In fact you can choose two of the above, but you cannot have all three.
Agile and all the other whiz-bang methodologies are attempts to produce all three by emphasising two of the qualities while hoping that the third will fall into place
I was once told the story of an old woodcutter who would tell his visitors what it was like in the “old days”. He also brought out the axe that his father had given him as a young man. He had looked after the axe well and had replaced the shaft five times and the head twice.
Do I hear you saying “That is not the original axe”? From one point of view, you are right, but one thing has remained unchanged – the specification behind the interface between the shaft and the head. If that had not been carefully defined, the woodcutter would have had to replace the whole axe at some stage. Likewise with software.
Its easy to point out problems.
Its easy to blame others – consultants, tools, processes, project managers, managers.
As a side note – you forgot authors, which IIRC, was the business of many of the original signatories of the agile manifesto. If not then, certainly for the last 20 years.
You conclude Agile is hard. Its Messy.
But, if we rebrand it (Again. Add it to the long list that’s gone before – XP, RUP, Waterfall, Top-Down, bottom-up, Pair Programming, DSDM, TDD, Scrum, SAFE)) and keep sticking with it, “learning the lessons” (What lessons?), even after 20 years, Agile , will magically provide the silver bullet.
Just keep ignoring the managers, the broader organisational reality, the regulatory environment, the financial constraints, the consultants, tools and processes.
They are the problem not me.
Good luck with the religious dogma.
No wonder young developers are so cynical.
So are many of the old ones.
I work in a large, monolithic central government department, doing continuous delivery and doing it well (we believe), and the people who really need to read this excellent article are the same people who already believe they’re doing agile. They do stand ups, they even use Jira, yet they ‘transform’ on premise apps by lifting the same monolithic app into public cloud and they certainly won’t deploy anything to Prod without an approved Change Request, model practitioners of Discontinuous Delivery.
I wrote ‘Is Agile the new Waterfall?’ (https://www.linkedin.com/pulse/agile-new-waterfall-glenn-bishop/) in 2016 and it’s interesting to see how things have moved on. Your excellent post echos some of the real world challenges of using it ‘out the box’: https://www.simplethread.com/agile-at-20-the-failed-rebellion/ and I have added it as a reference. Hope this is ok!!
There are many ideas being expressed in these comments. The one notable thing missing, I think, is that no-one takes into account the size of the projects. When I read up on SSADM, it suggested the project needed at least 2 full time people to do some task, not a big requirement as it was less than 1% of an overhead. This implied that the methodology is for large projects. No wonder the documentation is extensive.
Another point missing is the experience of the project team. If the team has been maintaining a system for a long time, they should know it backwards. There is much less need for all of the documentation than if everyone is starting out.
Requirements seem to be downplayed by some comments. If you don’t know where you are going you may as well go anywhere. You must have a target, however vague. The requirements can always change, and usually do. The end user usually does not know what you are capable of, nor quite what they really wanted. Close communication with the end user fine tunes the requirements and gets there faster. It does not matter what the methodology is.
guys, you’re here all sw developers, but can you imagine that this agile madness is being applied through the company no matter what activity or process. i’m sitting in finance department and we are forced to standups, scrums, epics and stories and sprints with our daily operational work … monthly reporting, quarter end, year end, budgeting, ad-hoc requests, business partner communication, financial analyses … it’s causing unbelievable overhead and imposes total control over you, killing your creativity, growth and curiosity … you just make sure you have ticket, content does not matter. you feel not like human, but like an unknown unlabeled capacity … that’s my observation sofar
“… you have ticket” is one of the things why they wrote “Working software over comprehensive documentation” or in your case maybe “accounting work over overburdened tracking”. It is without question that software needs documentation. But to spend a year for the requirements document, another 1,5 for the detailed specification and then start developing the software and covering all the changes (correctly!!!) is a lot of wasted effort. Especially when each and every change is accompanied with 3 months of additional negotiation (change! more money vs. change! you didn’t deliver as promised — less money).
The idea was to lessen this waste if you are working with the customer together and creating a software that works because “Oh, I imagined that differently” does not occur after 3 years of project time.
Peace
A quote of a quote:
“Teams got to announce what they could and couldn’t do in the month..”
More like guesstimate what they think they will do, and then regret that for a month because it was horribly naive.
Agile was the worse rebranding of common sense, ever. With a relatively vague vocabulary. Yet, almost all of it was common sense. The reason for this vague vocabulary was the most important thing being ignored: the language. Humans are a species that live inside the language. If you remove the language from a human, you get a bag of biochemical material that is driven by instinct.
The need for having a structured conversation between human beings was totally ignored. And one reason for that was, there was no domain language for the software development itself, to begin with. Other attempts at creating a way for having a structured conversation were pretty late, like event storming.
Interestingly there are many different systems with their own mechanics for establishing a structured conversation, from industrial engineering to tabletop RPG games. Why was this part ignored? “Everyone understands this much and has a thought process similar to this” was a big, untrue assumption. I presume no non-technical person was there when Agile was being created.
This lack of proper communication between people who were creating a product – technical and non-technical alike – led to some very bad practices (to fulfill bad assumptions and expectations like controlling speed and scope at the same time).
First was the notion of reusable code which was the most counterproductive way of thinking ever. The goal should be to write disposable code. If at each step we ask ourselves “Is this code disposable?”, many essential practices and patterns come to us in a natural way (like DIP and SRP).
The second side effect of inefficient communication was the shift from object discovery to object invention. When you have no problem to solve (mostly because of not having a proper understanding/communication structure) and you feel the urge to write something, object invention happens, piles of liability.
And pretty much like after 9/11 when all marketing schemes contained the word security (security cameras, security apples, security underpants), the same thing happened to Agile and the meaning shifted from Agility (react) to Velocity (act blindly and move “something” forward).
Agile at 20: The Failed Rebellion – really?
I would challenge the statement that Agile was a rebellion at all, as described in the article. Maybe the Manifest was. So if you use Agile as a term that represents what is stated in the Manifest for Agile Software Development, then you might be right. This is more or less a software development view.
The term Agile in management was used 10 years earlier when a bunch of people came together at the Iacocca Institute of Lehigh University. They were not dissatisfied with software development practices, they were horrified by the success of companies from the Asiatic rim employing a methodologie called “Lean”. So they put forward a document called “21st Century Manufacturing Enterprise Strategy” with the aim to counter the “Lean” methodology with another one they called “Agile”. This was no rebellion at all 😉
In a complex world, we have to be very conscious about our mental models.
Rule #0 of starting a rebellion:
Don’t give it a name everybody wants to attribute to themselves.
I mean…if I was to start a radical political movement, and I called my radical new ideas “fair politics”, I can imagine all kinds of established politicians saying “yeah, this guy is right, we need fair politics. And hey, look at us, we already were fair before this guy even was born”. They would not even have to read a single line of my ideas – it would be up to me to explain that “that’s not the fair I mean”.
Not even capital letters could save Fair Politics now.
Visiting the Agile Manifesto’s website I say it’s high time we changed its background picture. In 2021 Agile shouldn’t be about balding white men gathering in front of a whiteboard anymore.
Great article. I’ve been in software engineering for thirty years. I started working with Agile over the past eight years. I have been very disappointed with it. I believe the primary reason I don’t like it is that it (as it is being used in my company) gives little real consideration to the big picture. This always results in constant bug-fixing and changes being done to previously completed sprints due to lack of foresight.
I’ve worked on several Agile projects. The first one went well. We adhered to the manifesto principals. The remaining projects were a disaster. The customer wouldn’t show up to scrums, we used Jira as a tool (ugh), sprints went on forever because defects were not treated as user stories in a backlog, the customer after not showing up to scrums didn’t like what we delivered blah, blah, blah … After several months of this we started calling it Fragile instead of Agile.
I’m writing these comments while listening to a lecture on Agile.
Cybersecurity is failing, all because of agile. It turned infosec upside down. The NIST Framework puts the responsibility of security squarely on the business stakeholders and rightly so makes IT the custodian of the business plan. Agile has no plan and none of the 17 signatories had a background in cybersecurity. Breaches are a way of life and until we understand exactly what the IBM Cost of a Data Breach says that DevSecOps is the path to cybersecurity and the lack of zero trust is costing the economy and consumer in a way that mean that pretty soon we will all be living off the grid and money won’t matter, and code won’t either. When IBM states that cybersecurity is a Marathon, not a sprint, they pulled no punches at agile.
So get ready to “Leave the World Behind.” The next war will be fought with rocks and stones and maybe that is the way to learn.