I came across this article about the failure of the Virtual Case File project about a week ago. I read things like this in the hope of learning from the mistakes of others (instead of having to make them myself). What follows are some of the conclusions I drew from reading the article, and how they might apply to other projects.
Have the Right People in the Right Roles
The author of the article (Harry Goldstein) calls the appointment of Larry Depew to manage the VCF project “an auspicious start”. Since Depew had no IT project management experience, putting him in charge of a project so large with such high stakes struck me as a mistake. This error was compounded by having him play the role of customer advocate as well. In order to play of the role of project manager effectively, you can’t be on a particular side. Building consensus that serves the needs of all stakeholders as well as possible simply couldn’t happen with one person playing both roles.
Balance Ambition and Resources
The FBI wanted the VCF to be a one-stop shop for all things investigative. But they lacked both the necessary infrastructure and the people to make this a realistic goal. A better approach would have prioritized the most challenging of the individual existing systems to replace (or the one with the greatest potential to boost productivity of FBI agents), and focused the efforts there. The terrorist attacks of 9/11/2001 exposed how far behind the FBI was from a technology perspective, added a ton of political pressure to hit a home run with the new system, and probably created unrealistically high expectations as well.
Enterprise Architecture is Vital
This part of Goldstein’s article provided an excellent definition of enterprise architecture, which I’ve included in full below:
This blueprint describes at a high level an organization’s mission and operations, how it organizes and uses technology to accomplish its tasks, and how the IT system is structured and designed to achieve those objectives. Besides describing how an organization operates currently, the enterprise architecture also states how it wants to operate in the future, and includes a road map–a transition plan–for getting there.
Unfortunately, the FBI didn’t have an enterprise architecture. This meant there was nothing guiding the decisions on what hardware and software to buy.
Delivering Earlier Means Dropping Features
When you combine ambition beyond available resources with shorter deadlines, disaster is virtual certainty. When SAIC agreed to deliver six months earlier than initially agreed, that should have been contingent on dropping certain features. Instead, they tried to deliver everything by having eight teams work in parallel. This meant integration of the individual components would have to be nearly flawless–a dubious proposition at best.
Projects Fail in the Requirements Phase
When a project fails, execution is usually blamed. The truth is that failed projects fail much earlier than that–in requirements. Requirements failures can take many forms, including:
- No written requirements
- Constantly changing requirements
- Requirements that specify “how” instead of “what”
The last two items describes the VCF’s requirements failure. The 800+ page document described web pages, form button captions, and logos instead of what the system needed to do.
In addition, it appears that there wasn’t a requirements traceability matrix as part of the planning documents. The VCF as delivered in December 2003 (and rejected by the FBI), did things that there weren’t requirements for. Building what wasn’t specified certainly wasted money and man-hours that could have been better spent. I also inferred from the article that comprehensive test scenarios weren’t created until after the completed system had been delivered. That could have (and should have) happened earlier than it did.
Buy or Borrow Before You Build
Particularly in the face of deadline pressure, it is vital that development buy existing components (or use open source) and integrate them wherever practical instead of building everything from scratch. While we may believe that the problem we’re trying to solve is so unique that no software exists to address it, the truth is that viable software solutions exist to subsets of many of the problems we face. SAIC building an “an e-mail-like system” when the FBI was already using GroupWise for e-mail was a failure in two respects. From an opportunity cost perspective, the time this team spent re-inventing the wheel couldn’t be spent working on other functionality that actually needed to be custom built. They missed an opportunity to leverage existing functionality.
Prototype for Usability Before You Build
Teams that build successful web applications come up with usability prototypes before code gets written. At previous employers (marchFIRST and Lockheed-Martin in particular), after “comps” of key pages in the site were done, usability testing would take place to make sure that using the system would be intuitive for the user. Particularly in e-commerce, if a user can’t understand your site, they’ll go somewhere else to buy what they want. I attribute much of Amazon’s success to just how easy they make it to buy things.
In the case of the VCF, the system was 25% complete before the FBI decided they wanted “bread crumbs”. A usability prototype would have caught this. What really surprises me is that this functionality was left out of the design in the first place. I can’t think of any website, whether it’s one I’ve built or one I’ve used, that didn’t have bread crumbs. That seemed like a gigantic oversight to me.