Data Deluge - Storage software need to step up!

From EMC sponsored IDC study, The Expanding Digital Universe: A Forecast of Worldwide Information Growth Through 2010 [PDF].

In 2006, the amount of digital information created, captured, and replicated was 161 Exabyte (EB) = 161,000 Petabyte (PB) = 161,000,000 Terabyte (TB). Three million times the information ever written in books.

From storage perspective, 403 million hard drives of average 400GB capacity each without any RAID protection would have been required to store all the digital data, 93% of all hard drives produced in 2006.

In 2010, the information added annually to the digital universe will increase six folds to 988 EB.

To use same number (403 million) of hard drives in 2010 as in 2006, disk capacity will have to increase to average 2.4 TB.

The most telling statement in IDC study, I found, is:

In 2007 the amount of information created will surpass, for the first time, the storage capacity available.

This only goes to show that we just cannot rely on capacity of storage media to store all information created. Software has to step up to make sure that we can store all information created on the storage available to us. And that is why I believe technologies like data de-duplication and compression will finally become appealing in primary storage arena. See, my previous post Data De-duplication for Primary Storage.

Also, check out my new favorite storage startup Storewiz focusing on compression for primary storage.