The Pitfalls of Agile and How We Got Here

Agile is big right now, and it has been for a while. The State of Agile report received a record number of respondents to its most recent survey in 2018, and Google Trends shows a steady increase in interest around Agile software development over the past 15 years. But you don’t really need statistics to tell you that Agile is popular — just count occurrences of the word “Agile” in any industry blog or conference schedule.

And there’s good reason for this. When done right, Agile processes can help teams get working software in front of stakeholders early and often, react rapidly to stakeholder feedback, and identify issues quickly so they can be fixed early on — all of which lead to a better final product.

The problem is that in addition to its virtues, Agile also has a number of blind spots baked in to its core practices. Its focus on short, narrowly scoped design and development cycles can make it difficult to define a finish line for projects, which hampers everything from resource planning to application architecture decisions. Similarly, although the Agile Manifesto’s prescription that “business people and developers must work together daily” is not inherently flawed, a dogmatic implementation of this principle in the form of daily stand-up meetings or sitting together while coding can end up wasting time and frustrating all parties involved.

Solutions to these blind spots are as varied as Agile methodologies themselves, but the most versatile and widely applicable is prototyping. Building and iterating on medium and high fidelity prototypes as part of your design process helps your team understand what the final product will look like, giving you a picture of the finish line to use for project estimates and architecture decisions. The prototype also acts as a visual and interactive functional specification for developers, which can preempt questions they might otherwise have to spend time getting answered by a product owner or designer.

We’ll go into more detail on how to avoid the pitfalls of Agile with high fidelity prototyping in the next article, but first let’s take a look at how Agile came to be what it is today so we can better understand the roots of its flaws.

The Time Before Agile: Physical Engineering and Waterfall

When the software industry was first stumbling into being in the 1960s, development was pioneering, experimental …and largely inefficient. Industry professionals had no established production methodologies to follow because they were still figuring out what was needed to build successful software in the first place. Without past experience to draw on, teams were ill equipped to estimate project costs and time to completion, leading to software that was often delivered over budget, over schedule, and lacking key functionality — if it was delivered at all.

This “software crisis,” as it was later termed, spurred the industry to look to other engineering disciplines for guidance. In the 1970s and ’80s, the software industry was seized by the idea that building software could be made as predictable as building something like a bridge or a car, and software teams adopted production methodologies from traditional physical engineering and manufacturing to achieve this.

With hindsight we know how naive this hope was, but you can see the appeal at the time. Software projects had been consistently failing for over a decade, and physical engineering and manufacturing processes promised the reliability of experience, having been refined over the millennia of human history. No longer would software professionals be stumbling blindly forward.

Of course, building a piece of software is dramatically different from building a bridge or a car, as we now know, and these borrowed methodologies didn’t fully deliver the results the industry had been hoping for. They did help, though. Following a process of carefully gathering requirements and designing a cohesive solution before moving on to actual production (coding) enabled far more software teams to successfully deliver products to their customers. This “Waterfall” process became the new standard.

Where Waterfall Failed Software Teams

The problem with treating software development like traditional engineering is that it ignored the unique challenges facing software teams. Waterfall worked so well for physical manufacturing projects because requirements were comparatively easy to determine and were unlikely to change during the time it took to complete a project. Likewise, technological innovations for physical manufacturing rarely outpaced production time, so products weren’t at risk of becoming obsolete before going to market. Software development, on the other hand, offered neither of these assurances.

Before analysts and researchers had access to tools that let them create rich, interactive prototypes without code, requirements were notoriously difficult to gather. Users can’t tell you what they need, only how they work, which tools they use, and what their pain points are — it’s up to you to figure out a solution. Conducting user testing with a prototype can help you catch missed or incorrect requirements before you send specifications to development, but early teams often couldn’t afford to allocate programmers to create prototypes. Under Waterfall, this meant they couldn’t be sure they’d landed on the correct solution until the software had been built and delivered to end users.

Compounding this fundamental drawback of traditional Waterfall, businesses and technology evolved rapidly in the 1970s and ’80s, and software development often couldn’t keep up with changing customer needs. Even if a software team was lucky enough to have built the correct solution for a customer per the original project requirements, the customer’s requirements were almost guaranteed to have changed during the time — frequently years — it took to build and ship the software. This made it exceedingly difficult to hit the mark with software products, and customer dissatisfaction was rampant.

Traditional Waterfall, then, failed software development teams in two major ways:

1. It didn’t provide space for teams to gather ongoing feedback from stakeholders to ensure they were building the correct solution for the customer.

2. Its unidirectional flow of work prevented teams from going back to earlier stages of the development process to account for changing requirements or new technologies.

How Agile Attempted to Address the Drawbacks of Waterfall

In the 1990s, a slew of “lightweight” methodologies, such as Scrum and Extreme Programming (XP), began to emerge in response to the failings of the “heavyweight” Waterfall model. The common theme among them was a rejection of rigidly structured and strictly managed processes in favor of providing software teams greater autonomy and freedom. The idea was that this would give teams the flexibility to revise requirements and designs when needed in order to produce the best solution for the customer.

This philosophy was later enshrined in the Agile Manifesto‘s famous declaration of values in 2001:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

It’s also echoed in three of the Agile Manifesto’s 12 principles:

Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

The best architectures, requirements, and designs emerge from self-organizing teams.

This, along with the push toward early and continuous delivery of working software to stakeholders, revolutionized the software industry. Interacting with working software on a regular basis during the development process allowed stakeholders to give valuable feedback on the direction of the product. Software teams could then respond to that feedback in whichever way they deemed most appropriate since they could now return to any stage of the development process when needed. The result was speedier production and higher customer satisfaction.

The Pitfalls of Agile

There’s no doubt that Agile continues to offer these benefits and others well into the present. Respondents to the 2018 State of Agile survey report flexibility and decreased time to market as some of the top benefits of adopting an Agile methodology, and The Standish Group’s 2015 CHAOS Report lists “Agile Process” among the ten Factors of Success for software projects. Clearly something is working.

Why, then, are thought leaders foretelling the end of Agile? Why are there myriad Subreddits and Stack Exchange conversations dedicated to what developers hate about Agile? Why is Ron Jeffries, one of the original authors of the Agile Manifesto, urging developers to abandon Agile methodologies altogether?

The issue is that many popular Agile methodologies exacerbate Agile’s inherent blind spots by abandoning the good parts of Waterfall entirely and deviating from the core values of the Agile Manifesto.

For all of its weight, Waterfall pushes teams to define a concrete goal for everyone to work toward. This enables better technical decisions and resource planning, and it can inspire teams by showing them what their efforts are contributing to. Agile rejects the notion that this can be done well and instead favors experimentation and iteration in the code. This can make a project feel more flexible, but too often that flexibility comes at the cost of technical debt, unclear timelines and funding needs, and developer confusion.

The authors of the Agile Manifesto sought to free teams from inflexible, heavyweight Waterfall processes, but in practice Agile methodologies often formalize aspects of the manifesto in process-heavy ways that go against this initial intent. Daily stand-ups, collaborative coding, and other such activities are meant to promote open communication between teammates — ostensibly this is why “business people and developers must work together daily throughout the project” — but it doesn’t take long to find complaints that these often become frustrating accountability exercises that demoralize the team.

Agile is working, yes, but not as well as it could be.

The Solution: Prototyping

So where do we go from here? How do we salvage the good bits of Waterfall, address the drawbacks of Agile, and do it all with the least amount of friction? The answer is prototyping. In the next article, we’ll get into the specifics of how to bridge the top five pitfalls of agile with medium and high fidelity prototypes.