Outsourcing Disasters or Disastrous Outsourcing!

I read with interest the recent posting by Claus Mikkelsen, Outsourcing Disasters about disaster recovery (DR) planning and how little overlooked details can bring you down.

His posting reminded me of the blackout day when I was stepping out of the building of a power utility company right when everybody in the region lost power. Just few minutes before blackout, I was discussing with the company about DR data center.

The utility company was in the process of setting up a DR data center not very far from the main data center. In light of how data centers in NY/NJ were impacted during 9/11, one of my concern was the impact to DR data center operations in case something happens near main data center. Power failure was also one such point raised during the discussion. And I got an earful on grid resiliency from the utility company guy. When I found out the grid being the reason for blackout, I just thought to myself "How Ironic!"

As for the story of Simmons changing DR vendor because of blackout, cynic in me says that there is more to this story than just the reason given in the article (See Disaster recovery supplier dumped).

I can only hope that Simmons learnt from their experiences and performed extensive due diligence. Hopefully, this time around, instead of just looking at written contract terms, Simmons really performed an assessment of vendor DR plan and any cascading impacts to its own operations and DR plan. And also has in place regular testing of different DR scenarios.

My favorite story: The primary network of an hospital went down due to a worm infection. So the IT staff switched to backup network and worm took that down too! Finally hospital resorted to SneakerNet to transfer medical images to ER and ORs. The moral of the story is to have an alternate process in place that is architecturally and physically different than your primary process.

Many times, I have come across this. The companies are quick to outsource a function. They worry about contractual terms instead of performing real due diligence on vendor's ability to deliver on those contractual terms.