As the software world marks the 20th anniversary of the agile manifesto, it occurred to me that there is an entire generation of developers who know nothing but that world. They know little of the decades of work done prior, nor, it seems, can they imagine how any of it actually got done.
Tales emerge of a horrifying “waterfall” world where projects were drawn up in excruciating detail only to flail and collapse under the weight of their many change orders and process challenges. They read Fredrick Brook’s “The Mythical Man Month” as a cautionary tale of what inevitably befalls such a process. They vow never to repeat this, and find comfort in a process where agility vanquishes the many variables that jeopardize their projects.
The difficult truth is that complex project management has always been hard. One can easily imagine the builders of the pyramids wrestling the very same variables and bemoaning “there must be a better way”.
Projects have three fundamental constraints: what is being achieved, the resources available, and the time to completion. These are, and always have been, the only mutable variables.
One can add to or remove from any of these three variables to achieve completion. You can alter what is being done: adding or removing components or varying the resultant quality. You can adjust the time to completion, sooner or later. And you can add or remove resources (money, material, tools, people). These truly are the only real variables.
Alas, these variables are interdependent. Each change deeply impacts the others, and each comes with peril. Brooks shows there is much danger in blithely throwing resources at a project. Every startup struggles with their MVP as they wrestle with the “what”.
And of course every project manager knows all too well the time variable. Everyone from the customer, the stakeholders, the dependent teams, and their boss wants to know when it will finally be done. Of the three variables, this one feels the most visible, and painful.
For decades software developers have wrestled with these variables. Early on, “it was what it was, and it was done when it was done”. As computers moved into large businesses, the first purchasers of these exorbitant machines, this was simply unacceptable.
Then followed a period of detailed control over the “what”, with months of specification and contractual formality. This is the world Fredrick Brooks lived in, and is the “waterfall” world the agile advocates decry. These extensive specs created the need for the resources and resulted in the predicted completion date.
This high rigor evoked pain when the any of the variables needed to change. Complex expectations all around were disrupted. And yet the changes were everywhere and seemed inevitable.
The ‘what’ could vary because of requirement changes or unexpected hurdles in the project. The resources could vary from any number of external forces, from economic to political. And the time constraint usually bore the brunt of it all. Projects would be delayed, seemingly indefinitely.
Many solutions can be used to ease this struggle. To handle inevitable variance in the what, change management tools are used. To handle the resource changes, teams are reorganized and streamlined. To manage time, milestones defend against uncaught catastrophic failure. And throughout it all metrics and monitoring are checked and rechecked. Many of those were also codified and structured in things like ISO 9000 – more formality and structure.
But huge complex projects are always hard. Change is difficult. Creating things anew never proceeds as expected. The dynamics of technology and markets and even corporate politics are frustrating. The agony of this struggle is felt all around, from the customer to the project manager to of course those getting it done.
The software developers at the core of it all feel much of the pain yet have little control. That is especially hard as they are the ones doing the actual creation, often working technical miracles.
From this frustration the agile manifesto was born in 2001. It says, essentially, if change is inevitable, let’s just embrace it. And as the developers are the ones creating, let’s trust them to create. They want to build great products for customers, let them do that.
The result was the agile methodology that puts the developers in the driver seat. They work from customer stories to create solutions. Further tweaked by scrum, they work on short cycles, adjusting the target with each cycle, embracing the changes that flow. And with completed work at each cycle, progress is laid bare. What’s not to like?
Alas many managers fret about the power shift from their seat to the developers. They are responsible for the project, yet have vastly less control of two of the three variables, the what and the when. Agile proponents highlight this as the core of the objections to agile. The “old guard”, they feel, resents the loss of their power over developers.
The true issue is less a power struggle than it is a fundamental need of business to have control over their core variables. Businesses loathe uncertainty. Their markets, their customers, their investors, their accountants, even their employees want predictability. And that means control.
The dynamism alone in an agile methodology makes business leaders blanche. As a result, many organizations don’t actually do “true agile”. They do short cycles, and create stories, and let developers drive much of the to-do list. But they apply more traditional tactics of monitoring and control on top of it. This frustrates agile proponents deeply.
The conflict is further aggravated by the recent movement in some agile circles to embrace “no estimates”. The sentiment is that since estimates are hard and usually wrong let’s not do them at all. Let agile teams do their thing, don’t ask them when it will be done, they can’t tell you. Don’t worry, it will all be fine. We seem to be returning back to “it will get done when it’s done”.
Business leaders are aghast at this total lack of control. They deal with countless areas of uncertainty, from financial markets to personnel issues. They do enormous building projects loaded with uncertain variables and environmental unpredictability. But they do them all with some idea of where they are headed. They can predict with some reasonable confidence the financial markets, the people concerns, the building projects, based largely on estimates from experts and history. They can budget, evaluate, and plan based on some, perhaps variable, level of predictability and control.
For most business leaders, the problem with these dynamic methodologies is not their process, it’s not the details of how the project gets done. It’s the lack of control. And that goes back to the three variables.
Every business leader since the pharaohs has known those same three variables: what gets done, the resources used, and the time it takes. As a crusty veteran of software for over forty years, a project manager and business executive for most of that, I know these truths in my soul. Perhaps agile presents a different way of looking at it, but an agile process doesn’t magically change these fundamental principles. Every change to the what begets a change to the resources or time. Even if done every two weeks.
The disconnect between the proponents of agile and the business leaders who commission their work is, in many ways, one of culture and communication. I have lived this as well. When I moved from the leadership of a $100m software business into Human Resources, the culture change was shocking and difficult.
I firmly believed in much of what HR was doing and advocating for. But the way in which they talked about it was often lost on their business leader customers. Further, much of what they advocated — significant changes in culture or specific changes in behavior — lacked a business context. So not only did the HR people speak in terms that were difficult to understand, their programs and changes were dismissed or ignored. I spent the next few years both translating and tweaking projects to have relevance to their customers. I like to believe to some success.
The same needs to be true in project management. If the agile proponents who bemoan that businesses are not “agile from top to bottom” hope to have some success, they would do well to examine their own advocacy. Providing vivid business contexts, demonstrating firm control of the process, and — heaven forbid — even embracing the need for some modicum of predictability will vastly increase the acceptance in the leadership ranks. And probably improve the development process as well.
If the proponents of agile want to remain relevant for the next 20 years, they would do well to find ways to embrace a few realities: projects are always hard, the three variables are inviolate, and the patrons of your projects need control.