Planning To Fail

Like many of you, I'm staring in utter horror and disbelief at the ecological catastrophe unfolding in Gulf of Mexico.

Indeed, the Deepwater Horizon will likely go down in history alongside the Exxon Valdez, Bhopal, Chernobyl and other examples of massive infrastructure failure destroying the environment on a mind-numbing scale.

And, as I go through all the articles and analysis to date, I am struck by one simple fact: it is likely that they didn't have a plan for the inevitable. 

And if the story comes out as I believe it will, I will consider that unforgivable -- simply because it was avoidable.

Bad Stuff Happens

We live in an imperfect world where -- despite our best efforts -- things routinely fail.  So we do our best to imagine what can go wrong, and put in place either preventative or remediation mechanisms for when the inevitable happens.

Just about every infrastructure industry does this reasonably well most of the time.

For example, if you run an airline, sooner or later you will have a bad day.  As a result, airlines have both an extensive set of preventative practices as well as having a good plan in place when the inevitable happens.  I could say the same about cruise ships, power generation, telecommunications -- even those of us in IT!

If you go back and look at the first few days -- or weeks -- after the BP accident, it appears that they may not have had a viable plan in place.  Strong anecdotal evidence that preventative measures weren't taken seriously.  And, once the inevitable happened, it looks to many like valuable time was lost trying to shift blame, spin public perception, etc. 

This type of response was perhaps more sickening than the accident itself.

Ultimately, this sort of tragedy may come to be seen as a cultural failure, rather than a specific failure around technology or process.  The story may come out on how management culture may have directly caused key safeguards to be overlooked or obviated, leading to the catastrophe we all are witnessing. 

We've seen this sort of cultural failure before, we will likely see it again.

Lessons For All Of Us

I am not here to judge any company or organization, simply to learn lessons that may apply.

Many of us are involved in the IT industry and associated infrastructure.  We're going through a period where (a) we are much more dependent on IT in our daily lives, and (b) that IT is progressively being delivered from denser and denser data center infrastructures.

The stage is being set for IT professionals to have their own version of a really bad day -- on a scale and perhaps with an impact unimaginable before.  Thousands of servers and petabytes of information in a single location is becoming the norm, and not the exception.

We as technologists have to do our part to create more reliable and recoverable architectures that are not unduly expensive or complex.

We as architects and operational managers also have to do our part to constantly visualize new kinds of threats, devise procedures that both prevent and help remediate the inevitable, and continually test our processes and plans to make sure they are effective when needed.

But -- more than anything -- we as IT professionals need to guard against the same sort of cultural complacency that may have created the context for the disaster in the Gulf of Mexico. 

Yes, perhaps the consequences of a massive IT failure pale in comparison with the current ecological catastrophe, but that matters little.

Because failing to plan is planning to fail.

Comments

[John W]

Plan for the worst, hope for the best. It's an old story. Learning from history and experience seems to be a consistent challenge.

[Mike Foley]

A massive IT failure may pale against the ecological disaster that is going on, but I can envision a time when an IT failure on a massive scale could have tremendous economic impact, equal in economic scope to what's happening today. I hope I'm not around for that!

mike