Agile Is Not A Silver Bullet, Part 1
Hopefully this headline got your attention. I considered trying to come up with a hashtag like #NoTDD or #NoEstimates that seem to be popular these days. I opted for a (hopefully) catchy title instead. The purpose of this post is not to bash agile methods or to somehow suggest that we should not be following agile principles. Far from it. My purpose is to get you thinking about how agile, by itself, has limitations in its ability to transform development teams and to challenge some of the dogma around “just-in-time” software design in agile environments.
In this part of my two-part series, I am covering responsibilities, goals, and struggles that development teams are facing today. In part 2, I will cover our experiences and how we must take a “big picture” look at how we’re working in a world of constantly changing requirements.
What Is Our Responsibility?
When you think about where software development fits within your organization, what do you see as the ultimate responsibility you have within the company? What is it that you provide? What is your ultimate role in contributing to the success of the business?
I have given some thought to this over the years, especially when we formed Nebraska Global and Don’t Panic Labs. Our goal in building the software development team at NG/DPL was to reduce the risk of the development side of startups and enable accelerated MVP (minimum viable product) timelines. We would do this by creating a “flexible” product development resource that we could move in and out of the various startups and projects we were building. Over time, this internal flexible resource has become an external one as we have built a thriving contract product development business that enables other companies to flex us in as they need help with various innovation, product development, and legacy application migration initiatives.
The insight I had on all of these efforts is that, regardless whether you are developing a product for sale or developing a support system for a company, we all have similar responsibilities to our partners – to enable business agility.
So what do I mean by this? We must be able to rapidly react and respond to new information and new insights that our partners receive. This means we should be providing the necessary features and support systems that allow them to effectively leverage the new information and insights. To the extent we are able to do this defines how agile we really are.
Agile Methods Are Essential, But…
… they are not a silver bullet. We see a lot of teams and organizations transitioning to agile processes with the hope/expectation that this will give them enough agility to be responsive to business needs, create better alignment between stakeholders and developers, and increase the team’s productivity.
I would be surprised if most organizations adopting agile methods don’t get at least some lift in terms of alignment between stakeholders and the ability to respond to changing priorities. However, unless the development team is quite mature and already developing loosely coupled, highly cohesive, testable software designs, I doubt whether they see much, if any, improvement in their productivity. In fact, I can see the potential for a significant decrease in quality as well as the ability for these teams to complete the work they are committing to in each sprint. Why? I see two main reasons…
- Ever-increasing complexity – the problems we are trying to solve with software today are far more complex than they were 5-10 years ago, and this is not going to slow down. The increasing sophistication of the solutions and the need to react quickly to changing market demands is putting increasing pressure on software platforms that often have a significant amount of technical debt and have become fragile and rigid.
- Team maturity – it’s easy to go to a conference and watch a thought leader talk about the benefits of agile processes and how they have dramatically changed their culture and success. These are genuine stories and I have no doubt there is little hyperbole in their claims. The problem is, most organizations do not typically have the consistent level of maturity in processes and ability that these thought leaders’ teams have. If I had a team led by Robert Martin, Martin Fowler, Kent Beck or folks of that caliber (i.e., folks with extremely strong software design/development capability), I could see where layering in agile processes would be sufficient to achieve the sustained business agility that is promised. The reality is that most of our teams are composed of “mere mortals” who lack the background and experience necessary to design and develop software in a way that takes full advantage of agile processes.
Addressing The Software Development Side Of The Equation
What’s missing from this conversation, and many of the implementations of agile, is the recognition of the necessary changes required on the software development side of the equation. In other words, what are the changes and processes we need to adopt in the areas of software design, maintainability, quality assurance, etc to fully realize the benefits of an agile culture?
The irony here is that the folks who developed the Agile Manifesto saw this. Many of us have seen the home page of agilemanifesto.org and are familiar with the list of what is valued. I suspect most people did not see the “Twelve Principles of Agile Software” link at the bottom of the page. There are three of these principles that I think we, as a community of agile developers, fall short on:
- Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
- Continuous attention to technical excellence and good design enhances agility.
- Simplicity–the art of maximizing the amount of work not done–is essential.
The first bullet states that we should be able to achieve sustainable development velocity indefinitely. Ask yourself this… “How many projects have I worked on where it was just as easy to make changes to the system in year seven as it was on day one?” or “Have I ever worked on a code base that was more than five years old where I was quite comfortable making changes to it?” I suspect most of you have never worked on a project where either of these has been the case.
I suspect some people might look at the second bullet and assume it refers to how proficient our developers are in a programming language or how comfortable they are using the latest framework. I see it quite differently. What this says to me is that we need more REAL software engineering. These are the software architecture, software design skills, and the processes to nurture and maintain these architectures and designs that are necessary to achieve the sustainable development discussed in the first bullet. Unfortunately, we are falling short on how these skills are taught and, as a result, many development teams are lacking.
The final bullet is included because this is a hot-button issue for me. I too often see folks pursue esoteric, overly complex solutions to problems as opposed to looking for the simplest (i.e., most understandable, testable, maintainable) solutions. When we are creating overly complex solutions to simple problems, we are creating technical debt. I see this most commonly in the unnecessary asynchronous processing and messaging that exists in a lot of software applications.
Managing Complexity
Steve McConnell stated in “Code Complete” that unmanaged complexity is the most common technical reason projects fail. In other words, we don’t do a good job managing complexity. One needs only look at the healthcare.gov debacle a few years back. Why is this? What is going on here? I already mentioned above that the problems we are solving today are more complex than in years past and yet I do not see much being done on the software design side to mitigate this complexity risk. The main challenges facing us are human limitations:
- Humans are not capable of managing all details of a complex program at once
- Some systems can grow so complex that no one really understands what it does, making further development very difficult
Unless we change our approach to software design and development to meet these challenges head-on, we will never be able to “maintain a constant pace indefinitely”.
