Power - The Theme of Data Center Outages

As soon as I finished my last rant (See, Infrastructure Failure - Be Paranoid), a reader pointed out a similar incident close to home, at Fisher Plaza in Seattle.

Fisher Plaza is considered to be a premium data center housing numerous high profile clients and ten different telecom carriers bringing fiber to the buildings. I visit the facility time-to-time as several customers are co-located there.

It seems Fisher Plaza experienced an outage due to electrical power equipment failure few weeks ago knocking KOMO TV and KOMO 1000 News Radio stations offline (See, Unsinkable Data Center Crashes in Seattle). There were also several similar incidents reported in other cities (See, Data Center Outages Bring Headaches, Headlines and InterNAPPing?)

The moral of these incidents for customers are very simple:

• There is no 100% uptime SLA. Anyone who promises you this is smoking something they shouldn't be. In a AFCOM survey, 20% had been hit with at least five failures in past five years (See, Five Predictions: 9 of 10 Companies Face Failures)
  

• When planning mission critical operations, look beyond just redundant power supplies, UPS and WAN connections. Facility is as important as your equipment in keeping your operations up. "Many data centers just can't handle new technologies coming out," Comment by a presenter at AFCOM Data Center World Conference (See, Five Predictions: Relocations and Outsourcing)
  

• Perform due diligence not only on tenant happiness and satisfaction but also crisis handling and management.
  

• Murphy's Law is alive and kicking even for well planned, thought out, mission critical activities.

Useful Free Tools & Information

WinMerge - An Open Source visual text file differencing and merging tool for Windows. A great tool for comparing log files.

RealVNC - A remote access tool for system administration and shared workspace.

Virtual CDROM for Windows XP - Store all your CD images on hard drive and use them with Virtual CD-ROM.

ISO Recorder - Create ISO images of data stored on your hard drive.

MSDN Library - Subscription free download of MSDN Library to your computer, a valuable resource for IT people developing and supporting Microsoft tools, products and technologies.

Disaster Recovery for Microsoft Exchange - Guidelines on developing and implementing a backup strategy for Microsoft Exchange Servers.

Books Online for Microsoft SQL Server - The primary documentation for SQL Server.

Microsoft Baseline Security Analyzer - Perform scans of Windows systems to identify common security holes.

Microsoft iSCSI Software Initiator - Connect your server to iSCSI targets using regular Network Adapters.

Storage meets Web 2.0

Jeremiah is organizing a storage event for Web 2.0 companies at HDS HQ in Santa Clara, Sept 12 (See Lunch 2.0 Web Expo Blowout at Hitachi Data Systems and Lunch 2.0 Web Expo). With the popularity of Web 2.0 companies like Six Apart, YouTube and MySpace, their data storage requirements must be growing very fast.

I am going to miss this great event just by a day! as I will be in Bay Area from Sept. 7 through 11. I wish I knew about the event little earlier to extend my trip by a day.

Was the event organized at short notice? Most probably, Jeremiah would have mentioned it in our email exchanges while planning to meet during his visit to Seattle or my visit to Bay Area. BTW, anybody want to get together in Bay Area during my visit to chat about storage, blogs, startups and anything else, contact me.

Congratulations to HDS (specifically, Jeremiah and Hu) in taking the lead among SAN storage companies to target Web 2.0. SAN storage has been not that prevalent among the Web 2.0 infrastructure in my observation. I seem to find more NAS and Linux based commodity storage within Web 2.0 infrastructure.

I am sure HDS event will be a great success. Hopefully it will encourage them and other storage vendors to take such events on the road to meet "hot" Web 2.0 companies outside Bay Area. Here are two Web 2.0 lists for vendors who want to help them grow their infrastructure:

Seattle's Web 2.0 List

National Web 2.0 List

Infrastructure Failure – Be Paranoid

Just after writing my previous blog post Data Center Power Consumption and Heat Generation, I came across an interesting blog post with 400+ comments. It details the failure of hosting infrastructure at DreamHost (See, Anatomy of a(n ongoing) Disaster..) In my opinion, it is a recommended reading for everyone who manages or designs IT infrastructure for living.

Here are excerpts from the post with my commentary and takeaways.

Ironically, all the recent disasters stem somewhat from us attempting to take some proactive steps to head off any sort of future power outages like the kind we experienced last year.

Instead of narrowly focus on preventing something from happening again, use the event as wake up call. Assess your environment for other potential risks and develop comprehensive plan to address them. Also, be aware of new problems that may arise while solving another problem. I like to use the Chess analogy - Further you anticipate moves, better your chances of prevailing.

We're now basically 95% of their data center.

Consider how important you are to your vendor and leverage your position to negotiate better deal.

The Garland Building is supposed to be an excellent place for data centers. There are more than a dozen in the building. Companies like iPowerWeb, Media Temple, BroadSpire, and even MySpace (now the most popular website in the whole US!) are in there.

Do your own due diligence even if you think vendor passed the due diligence by a larger well known company. Their failure may be critical to your business and a drop in the bucket for these "other"” companies. It is not uncommon for vendors to offer sweet deals to attract high profile companies.

Around last June though, the building informed all its data center tenants that they had essentially run out of power!

Don'’t wait for other shoe to fall before taking actions. Be paranoid.

After months of searching and negotiating with Alchemy, we still had to get Switch and Data to allow us to put a cross-connect in from their data center over to their competitors down the hall.

Finding an alternate data center down the hall may seem quick and easy fix to existing power problem. But such short-sighted and point solutions fail to address other lingering issues such as Disaster Recovery / Business Continuity. I guess the company plan is to wait until problem occurs before addressing them.

No incumbent makes competitor entry easy and painless. Get specific deliverables when you have the leverage. Be prepared to work alone and have a Plan B considering total non-cooperation from vendor.

Data Center Energy Consumption and Heat Generation

This morning, I read interesting news about LBNL, Sun and others demonstrating DC power distribution in data center as a way to reduce power consumption and heat generation (See, Engineers: DC Power Saves Data Center Dough). Six months ago, I did come across few data center people interested in ways to reduce power and space requirements and decrease heat generation (See, Happy New Year & Food for Your Brain).

It looks like technology industry has been making some efforts in this area. DC power sounds an interesting alternative. I wonder what are the drawbacks of using 380-volt DC power instead of AC. The eWEEK article focuses on the good side only. I am sure there is bad side to it too. Aren't there reasons for preference of AC over DC in power distribution? Can't recall exact details but something to do with distribution losses.

On facility wide basis, 15% reduction in energy consumption does sound significant. I am looking forward to reading the complete report when and if available. I expect that efforts focusing on reducing energy consumption at rack level may produce more significant results.

This area is expected to be pretty Hot "literally" in near future considering projections indicating cost of powering up and cooling down exceeding cost of equipment.

Last year, Luiz Andre Barraso of Google also published an interesting article on the same topic in ACM Queue (See The Price of Performance) where he described cost trends of large IT infrastructure such as Google's with couple of interesting graphs. Some of the key points mentioned are:

• Every gain in performance has been accompanied by a proportional inflation in overall platform power consumption. The result of these trends is that power related costs are an increasing fraction of the TCO.
  
• The energy costs of that system today would already be more than 40 percent of the hardware costs. (The system is a x86 server worth $3,000 consuming 200 watts on average).
  
• If performance per watt is to remain constant over the next few years, power costs could easily overtake hardware costs, possibly by a large margin.

Storage Strategy for Digitized Medical Films

Last week, I received a phone call from Thomas looking for some help in figuring out the optimum storage strategy for his environment. Thomas is IT Director for a Texas based medical imaging group. The phone discussion took place 5:30AM so details are little bit hazy.

Recently, he installed a turnkey PACS solution encompassing Sun servers and EMC SAN and NAS storage for storing medical imaging studies (CT, X-ray, MRI, ...). He also has lot of archived medical films that he intend to digitize and store online ... 50TB+ if I remember correctly. He is concerned about storing this data on his expensive storage purchased as part of turnkey PACS solution!

He is looking for someone to advise him on the best strategy for storing this data. Unfortunately due to my added responsibilities and increased workload at ADIC, I am not able to help him actively. So if you have previous strategy experience in the medical imaging vertical and storage of digitized data, I will be happy to hook you up with Thomas.

Despite my hectic work schedule, I hope to continue being a sounding board for Thomas and people like him in healthcare segment.