I have often wondered why big programs tend to end up in big failures and I think I have finally understood the dynamics behind it. Having taken part in several big programs myself, that all failed one way or the other, I have felt it on my own body and it is not a nice feeling. Recently I was involved with a program that had tried and failed 3 times using the same approach.
By the way, by big programs I do not mean, say a Java program with tons of lines of code, but a collection of projects intending to make major changes to an organization.
When you are in a situation where you have to change the direction of your organization, especially when it means rewriting the core business software, you tend to go overboard. You finally have the opportunity to really provide the best you can for the organization and build something that can be the best possible solution for all stakeholders. Why not? Since you have to start from scratch you might as well get as many requirements as possible for as many stakeholders as possible. And since you have so many requirements, you need a large organization, many developers, lots of documentation and project managers to ensure that everything is in sync.
This is where it goes wrong. You start out in all possible places at the same time and try to manage your way out of a situation that really cannot be managed. Doing everything at the same time and trying to coordinate it is doomed to fail. This is why you often see that everything is great the first 12 months or so, but as soon as the different projects and sub-projects have to start delivering to each other, the fun starts.
Delays, misunderstandings, internal politics and what not, will drive the program to delays, re-planning, reduction of scope and more funding to make sure you can deliver at least something. If you are “lucky” you end up delivering something that is so reduced in functionality that it is not usable. More often the program shuts down after spending lots of money on producing nothing.
What to do about it?
I have come to the conclusion that the more important a new system is to your organization and the business as a whole, and the more people it impacts, the less you must start out with. Both in terms of people and functionality. Iterative and incremental development at its core.
Start with a small core team of really skilled developers and business people and let them drive the project. Build the core functionality and a healthy technical framework and build from that, involving more and more people as the functionality you implement matures. The way to it should be done no matter the size of the project, actually.
What is your opinion?
Am I completely off here or am I on the right track? Judging from all the large and failed projects I read about daily, at least it is obvious that something needs to be done.