OpenStack: Virtual Image Instances using Horizon Dashboard

Install Addendum

Enable VT in BIOS

An addendum to install steps defined in my previous post OpenStack: Quick Install using DevStack is required to avoid a surprise that I encountered after the install. Please check to make sure BIOS is at latest version available from the system manufacturer and Intel's Virtualization Technology (VT) is enabled in BIOS.

anil@OSCloud:~$ sudo apt-get install cpu-checker
anil@OSCloud:~$ sudo kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

If CPU doesn't support VT, the output will show CPU does not support KVM extensions.
The OpenStack Horizon Dashboard is implemented as a Python/Django web application that provides admin and user interface to OpenStack services.

Horizon Dashboard

Log in

In web browser, type the IP address for the dashboard. On Log In page enter User Name and Password and click Sign In. When signing in as Admin, the home page shows the Admin panel - System Panel - Overview.

Existing Virtual Machine Images

By clicking Images category in Admin - System panel on the left, a list of available images are viewed. In default installation, CirrOS x86_64 image is made available in AMI/ARI/AKI format.

CirrOS images are tiny cloud guest images with minimal Linux distribution that can also be downloaded from LaunchPad. The AMI/ARI/AKI is the image format supported by Amazon EC2. AMI (Amazon Machine Image) is a virtual machine raw image. ARI (Amazon Kernel Image) is a kernel file (vmlinuz) that will load initially to boot image. ARI (Amazon Ramdisk Image) is ramdisk file (initrd) mounted at boot time.

Launch Instances

Clicking on the Project tab in left panel shows the overview of current project.



To launch an instance from an image, click Images and Snapshot category in Project - Manage Compute panel on the left.



Select an image and click Launch. A Launch Instance modal pop-up appear. Enter a name in Instance Name field in Details tab.



In Access & Security tab, enter a passphrase in Admin Pass and Confirm Admin Pass fields.



Upon clicking Launch, Horizon dashboard switches to Project - Manage Compute - Instances page and shows the Instances running.



Clicking on Instance Name hyperlink shows the Instance Details for that specific instance with three tabs for Overview, Log and Console.



Though the Project - Manage Compute - Instances page shows instance to be Active and Running, the console for the instance is displaying an error message.

This kernel requires an x86-64 CPU, but only detected an i686 CPU.
Unable to boot - please use a kernel appropriate for your CPU.

Error Troubleshooting

A little bit of googling suggested to check whether the 64-bit PC (amd64, x86_64) or 32-bit PC (x86) version of host operating system is installed. Sure enough, the Ubuntu version installed on OSCloud host is x86 and not x86-64 version. I can't use x86-64 instance images on OSCloud host.

anil@OSCloud:~$ uname -a
Linux OSCloud 3.2.0-58-generic-pae #88-Ubuntu SMP Tue Dec 3 18:00:02 UTC 2013 i686 i686 i386 GNU/Linux

After terminating the newly created instance test1 and deleting all x86_64 Images, the next step was to either find or build x86 images and start a new x86 instance.

Prebuilt Virtual Machine Images

As OSCloud host is using QEMU Hypervisor, it made sense to look for qcow2 (QEMU copy-on-write) format x86 images. At CirrOS download page, I found a bootable qcow disk image for i386 and decided to try it out.

Create Images

To create images, on Admin tab, select Images and then click Create Image button in right pane. On Create An Image page, enter Name for the image, select Image Source, Image Location, and Format. Select the Public checkbox to make available this image to everyone. Then click Create Image. The image will be queued for creation.



Once images are created, they will be available to launch instances in projects following the steps listed above in Launch Instances section.





In next blog post, I will start to dig deeper into high level solution design using OpenStack. Your feedback and comments are welcome.

OpenStack: Quick Install using DevStack

Though not a recommended method for installing OpenStack for production, DevStack offers an easy method to install and run an OpenStack cloud either on hardware or even within virtual machine.

In this post, I walk through installing OpenStack using DevStack on an old Sony Vaio laptop on local network. The DevStack site provides instructions for installing OpenStack on Virtual Machines and on Hardware. The detailed instructions for installing OpenStack on a single hardware machine, that I followed, are available at DevStack.

Install Ubuntu Server OS

I repurposed a SONY VAIO laptop for DevStack install that was originally wiped clean with Darik's Boot and Nuke in preparation for disposal.

As DevStack downloads and installs all dependencies, I downloaded 64-bit PC (amd64, x86_64)Ubuntu 12.04 "Precise Pangolin" using Minimal CD mini.iso and burnt a CD on my MacBook Pro.

Booted Sony laptop using the Ubuntu minimal CD and selected Install from Installer boot menu, followed the prompts, and accepted default options for most prompts. It takes over an hour for the installation to download and install the base system.

Installed OpenSSH Server to enable access to Ubuntu server over SSH from my MBP.

$sudo apt-get install openssh-server

Checked whether SSH process is running by using one of the two commands listed below. The output shows process running.

anil@OSCloud:~$ ps aux | grep ssh
root      1394  0.0  0.0   6684  2416 ?        Ss   21:43   0:00 /usr/sbin/sshd -D   
anil      1534  0.0  0.0   4384   836 pts/0    S+   22:06   0:00 grep --color=auto ssh

anil@OSCloud:~$ service ssh status
ssh start/running, process 1394

Checked the IP address of Ubuntu server so that I can remotely access the server.

anil@OSCloud:~$ ifconfig
eth0    Link encap:Ethernet  HWaddr 00:1d:ba:23:9a:c5  
        inet addr:10.0.1.25  Bcast:10.0.1.255  Mask:255.255.255.0
        inet6 addr: fe80::21d:baff:fe23:9ac5/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
        RX packets:875 errors:0 dropped:0 overruns:0 frame:0
        TX packets:202 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:1000 
        RX bytes:522168 (522.1 KB)  TX bytes:19796 (19.7 KB)
        Interrupt:16 

lo      Link encap:Local Loopback  
        inet addr:127.0.0.1  Mask:255.0.0.0
        inet6 addr: ::1/128 Scope:Host
        UP LOOPBACK RUNNING  MTU:16436  Metric:1
        RX packets:0 errors:0 dropped:0 overruns:0 frame:0
        TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0 
        RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

While trying to access Ubuntu server from MPB over SSH, I received following error.

ANILs-MacBook-Pro:~ anilgupta$ ssh anil@10.0.1.25
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
26:54:a3:4e:cd:a3:6c:80:f3:36:2c:b3:c9:17:f0:db.
Please contact your system administrator.
Add correct host key in /Users/anilgupta/.ssh/known_hosts to get rid of this message.
Offending RSA key in /Users/anilgupta/.ssh/known_hosts:6
RSA host key for 10.0.1.25 has changed and you have requested strict checking.
Host key verification failed.

This is well-known error when there is fingerprint mismatch between the host (Ubuntu Server) and client (MBP). In this case, as there is no malicious attempt, I removed the offending key from MBP using ssh-keygen -R command.

ANILs-MacBook-Pro:~ anilgupta$ ssh-keygen -R 10.0.1.25
# Host 10.0.1.25 found: line 6 type RSA
/Users/anilgupta/.ssh/known_hosts updated.
Original contents retained as /Users/anilgupta/.ssh/known_hosts.old

After removing the offending key, I was able to successfully SSH into Ubuntu Server.

ANILs-MacBook-Pro:~ anilgupta$ ssh anil@10.0.1.25
The authenticity of host '10.0.1.25 (10.0.1.25)' can't be established.
RSA key fingerprint is 26:54:a3:4e:cd:a3:6c:80:f3:36:2c:b3:c9:17:f0:db.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.1.25' (RSA) to the list of known hosts.
anil@10.0.1.25's password: 
Welcome to Ubuntu 12.04.4 LTS (GNU/Linux 3.2.0-58-generic-pae i686)

* Documentation:  https://help.ubuntu.com/
Last login: Tue Jan 28 21:40:01 2014

Gave sudo privileges to anil user account

anil@OSCloud:~$ sudo visudo

A file /etc/sudoers.tmp was opened in nano editor. Added the following line to the opened file and then saved using ^X.

anil    ALL=(ALL) NOPASSWD: ALL

At this point, the Ubuntu server is ready for DevStack install.

Install DevStack

Before installing DevStack, it is good idea to review DevStack GitHub ReadMe file. A few points that stood out in this file for me were:

• Be sure to carefully read stack.sh and any other scripts your execute before you run them, as they install software and will alter your networking configuration.
• The DevStack master branch generally points to trunk versions of OpenStack components. For older, stable versions, look for branches names stable/[release] in DevStack repo.
• You can also pick specific OpenStack project releases by setting the appropriate *_BRANCH variables in the localrc section of local.conf.
• You can override environment variables used in stack.sh by creating file name local.conf with a localrc section.
• Swift is disabled by default. When, it is configured with only one replica to avoid being IO/memory intensive.

The last point was particularly important for my install as I really would like to understand storage components in OpenStack.
Install Git distributed version control system.

anil@OSCloud:~$ sudo apt-get install git -y
[sudo] password for anil: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
    git-man liberror-perl patch
Suggested packages:
    git-daemon-run git-daemon-sysvinit git-doc git-el git-arch git-cvs git-svn
    git-email git-gui gitk gitweb diffutils-doc
The following NEW packages will be installed:
    git git-man liberror-perl patch
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 6,703 kB of archives.
After this operation, 15.5 MB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/main liberror-perl all 0.17-1 [23.8 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ precise/main git-man all 1:1.7.9.5-1 [630 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ precise/main git i386 1:1.7.9.5-1 [5,963 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu/ precise/main patch i386 2.6.1-3 [86.0 kB]
Fetched 6,703 kB in 16s (409 kB/s)                                             
Selecting previously unselected package liberror-perl.
Reading database ... 46735 files and directories currently installed.)
Unpacking liberror-perl (from .../liberror-perl_0.17-1_all.deb) ...
Selecting previously unselected package git-man.
Unpacking git-man (from .../git-man_1%3a1.7.9.5-1_all.deb) ...
Selecting previously unselected package git.
Unpacking git (from .../git_1%3a1.7.9.5-1_i386.deb) ...
Selecting previously unselected package patch.
Unpacking patch (from .../patch_2.6.1-3_i386.deb) ...
Processing triggers for man-db ...
Setting up liberror-perl (0.17-1) ...
Setting up git-man (1:1.7.9.5-1) ...
Setting up git (1:1.7.9.5-1) ...
Setting up patch (2.6.1-3) ...

Clone the DevStack repository from GitHub.

anil@OSCloud:~$ git clone https://github.com/openstack-dev/devstack.git
Cloning into 'devstack'...
remote: Reusing existing pack: 14896, done.
remote: Total 14896 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (14896/14896), 3.55 MiB | 1.59 MiB/s, done.
Resolving deltas: 100% (10138/10138), done.

Change to devstack directory.

anil@OSCloud:~$ cd devstack
anil@OSCloud:~/devstack$ ls -la
total 260
drwxr-xr-x 11 anil anil  4096 Jan 29 12:15 .
drwxr-xr-x  4 anil anil  4096 Jan 29 12:15 ..
-rw-rw-r--  1 anil anil  1857 Jan 29 12:15 AUTHORS
-rwxrwxr-x  1 anil anil  2871 Jan 29 12:15 clean.sh
drwxrwxr-x  2 anil anil  4096 Jan 29 12:15 driver_certs
-rw-rw-r--  1 anil anil  1552 Jan 29 12:15 eucarc
-rw-rw-r--  1 anil anil  1145 Jan 29 12:15 exerciserc
drwxrwxr-x  2 anil anil  4096 Jan 29 12:15 exercises
-rwxrwxr-x  1 anil anil  1962 Jan 29 12:15 exercise.sh
drwxrwxr-x  2 anil anil  4096 Jan 29 12:15 extras.d
drwxrwxr-x  7 anil anil  4096 Jan 29 12:15 files
-rw-rw-r--  1 anil anil 66652 Jan 29 12:15 functions
drwxrwxr-x  8 anil anil  4096 Jan 29 12:15 .git
-rw-rw-r--  1 anil anil   226 Jan 29 12:15 .gitignore
-rw-rw-r--  1 anil anil    81 Jan 29 12:15 .gitreview
-rw-rw-r--  1 anil anil 11243 Jan 29 12:15 HACKING.rst
drwxrwxr-x  6 anil anil  4096 Jan 29 12:15 lib
-rw-rw-r--  1 anil anil 10143 Jan 29 12:15 LICENSE
-rw-rw-r--  1 anil anil   259 Jan 29 12:15 .mailmap
-rw-rw-r--  1 anil anil  3343 Jan 29 12:15 openrc
-rw-rw-r--  1 anil anil 14473 Jan 29 12:15 README.md
-rwxrwxr-x  1 anil anil   638 Jan 29 12:15 rejoin-stack.sh
-rwxrwxr-x  1 anil anil   870 Jan 29 12:15 run_tests.sh
drwxrwxr-x  2 anil anil  4096 Jan 29 12:15 samples
-rw-rw-r--  1 anil anil 12425 Jan 29 12:15 stackrc
-rwxrwxr-x  1 anil anil 43835 Jan 29 12:15 stack.sh
drwxrwxr-x  2 anil anil  4096 Jan 29 12:15 tests
drwxrwxr-x  6 anil anil  4096 Jan 29 12:15 tools
-rwxrwxr-x  1 anil anil  3769 Jan 29 12:15 unstack.sh

Make a copy of local.conf file from devstack/samples folder to devstack folder

anil@OSCloud:~/devstack$ cp samples/local.conf local.conf

Open local.conf file in editor and add the listed configuration changes in [[local|localrc]] section. I chose to have same password as ADMIN for other accounts. I also chose to enable Swift. Save the file.

anil@OSCloud:~/devstack$ nano local.conf

ADMIN_PASSWORD = select_password
MYSQL_PASSWORD = select_password
RABBIT_PASSWORD = select_password
SERVICE_PASSWORD = $ADMIN_PASSWORD

enable_service s-proxy s-object s-container s-account

Run DevStack using stack.sh.

anil@OSCloud:~/devstack$ ./stack.sh
[sudo] password for anil: 
Using mysql database backend

################################################################################
ENTER A SERVICE_TOKEN TO USE FOR THE SERVICE ADMIN TOKEN.
################################################################################
This value will be written to your localrc file so you don't have to enter it 
again.  Use only alphanumeric characters.
If you leave this blank, a random default value will be used.
Enter a password now:
select_password
2014-01-29 12:48:57 + echo_summary 'Installing package prerequisites'
...

Once install finishes, access information for the OpenStack cloud is displayed.

Horizon is now available at http://10.0.1.25/
Keystone is serving at http://10.0.1.25:5000/v2.0/
Examples on using novaclient command line is in exercise.sh
The default users are: admin and demo
The password: select_password
This is your host ip: 10.0.1.25
stack.sh completed in 339 seconds.

The OpenStack logs are stored in /opt/stack/logs directory. The latest detailed log can be accessed using symlink /opt/stack/logs/stack.sh.log. Unless there are errors, the latest summary log symlink /opt/stack/logs/stack.sh.log.summary is sufficient to check for installed, configured, and status of services.

anil@OSCloud:~/devstack$ cat /opt/stack/logs/stack.sh.log.summary
stack.sh log /opt/stack/logs/stack.sh.log.2014-01-29-134716
Installing package prerequisites
Installing OpenStack project source
Installing Tempest
Starting RabbitMQ
Configuring and starting MySQL
Starting Keystone
Configuring and starting Horizon
Configuring Glance
Configuring Swift
Configuring Cinder
Configuring Nova
Starting Swift
Starting Glance
Starting Nova API
Starting Nova
Starting Cinder
Uploading images
Initializing Tempest
stack.sh completed in 346 seconds.

At this point, I was able to log in to Horizon Dashboard using http://10.0.1.25/ from MBP.

Miscellaneous Operations

Stop DevStack

unstack.sh stops the processes that were started by stack.sh. All processes can be stopped by setting UNSTACK_ALL or specifying --all on the command line.

anil@OSCloud:~/devstack$ ./unstack.sh --all
    * Stopping web server apache2                                                                                                                       apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
    ... waiting                                                                          [ OK ]
tgt stop/waiting
tgtadm: can't send the request to the tgt daemon, Transport endpoint is not connected
tgtd seems to be in a bad state, restarting...
stop: Unknown instance: 
tgt start/running, process 13058
tgt stop/waiting
mysql stop/waiting
Stopping rabbitmq-server: rabbitmq-server.

Restart DevStack

Bring volume group online so that cinder-volume will start without errors

sudo losetup -f /opt/stack/data/stack-volumes-backing-file

Use rejoin-stack.sh to restart the DevStack

./rejoin-stack.sh &

Reboot Host

After rebooting the Host, when I tried to login to dashboard, I received the error An error occurred authenticating. Please try again later. The /var/log/apache2/horizon_error.log only shows Login failed for user "admin" as entry. Checked the SELINUX status.

anil@OSCloud:~/devstack$ sestatus
SELinux status:                 disabled

The reason turned out that I needed to restart Devstack by using the command listed above in Restart DevStack section.

In next post, I will take the Dashboard for a spin.

OpenStack: Overview of Service Components

OpenStack is made up of several projects that together build up the OpenStack cloud. Three core components are Compute, Storage, and Network.

OpenStack Training Guides provide a nice conceptual architecture of OpenStack service components, created by Solinea.

Compute (Nova)

Nova services provide computing resources through virtual machine (VM) instances and works with several virtualization technologies, such as KVM, QEMU, VMWare ESX, Xen, Hyper-V, and LXC. A complete list of supported Hypervisors is listed at OpenStack Wiki.

Nova is also called Cloud Controller as it provides framework for provisioning and managing VMs. Nova also provides ephemeral storage.

With Nova services, IT departments can offer private cloud services to internal departments that can scale computing resources on-demand as workload varies. It is analogous to Amazon EC2.
Nova is made up of several service components:

• nova-compute: Runs the VM instances.
• nova-scheduler: Decides which host will run the requested instance.
• nova-api: Service interface to Nova, such as a call to start up a Nova instance.
• nova-network: Network services
• nova-objectstore: File storage services
• nova-common: The underlying common libraries
• nova-cert: Certificate management service used to authenticate to Nova

Object Storage (Swift)

By default, all storage disappears when VM instance is terminated. Swift services provide cost-effective scale-out redundant persistent storage to VM instances and responsible for ensuring data replication and integrity. The object storage is suitable for static data and stored as Objects. The objects are stored and replicated on disks spread across storage cluster nodes. The stored data persists until deleted by users.

Swift is analogous to Amazon S3. Swift provides RESTful API for integration with other application, storage and services.

Swift is made of several service components:

• swift-proxy: Accepts, authorizes, authenticates incoming requests.
• swift-account: Manages database of accounts.
• swift-container: Contains mapping of containers.
• swift-object: Contains mapping of objects.

Block Storage (Cinder)

Cinder provides high performance persistent block storage for use with VM instances. Cinder volumes are appropriate for database, file system and raw block storage. Cinder is supported by several storage platforms including Ceph, Nexenta, SolidFire, Zadara, CloudByte, Coraid and Scality to name a few startups in this area. A complete lists of storage devices supported by Cinder drivers is available at Cinder Support Matrix.

Cinder is analogous to Amazon EBS. It also provides snapshot management with capability to either restore snapshot or use snapshot to create a new block storage volume.
Following services are available with Cinder:

• cinder-api: Authenticates and routes requests to block storage.
• cinder-scheduler: Scheduling and routing requests to volume service.
• cinder-volume: Managing back-end block storage devices.
• cinder-backup: Backup Cinder volume to Swift.

Network (Neutron)

Neutron (formerly Quantum) provides virtual network service for connectivity and addressing used by other services. OpenStack networking can have multiple private networks with overlapping IP addressing schemes and relies on Keystone for authentication and authorization for API access. A standard implementation includes management network, data network, external network, and API network. Though started with basic Linux VLANs and IP tables, it now includes plugins which are pluggable back-end implementation of OpenStack Networking API: Open vSwitch, Linux Bridge, NEC OpenFlow, etc.

Following services are available with Networking:

• quantum-server: Passing user requests to the configured Plugin for processing.
• plugin agent (quantum-*-plugin-agent): Perform local vSwitch configuration on hypervisor.
• dhcp agent (quantum-dhcp-agent): Provides DHCP services to internal networks.
• l3 agent (quantum-l3-agent): Provides L3/NAT forwarding for access to external network.

Image (Glance)

Glance provides catalog and repository for disk, server and snapshot images. Glance API service is used for discovery, registration and delivery of images. Users can choose from available images, create their own from existing servers and start new compute instances from base templates. Glance supports multi-format image registry including raw, AMI (machine), QCOW2 (QEMU/KVM), VMDK (VMWare), VHD (Hyper-V), VDI (VirtualBox), and OVF.

Glance is made of several service components:

• glance-api: Accept incoming API requests and communicate with other components.
• glance-registry: Stores and retrieves metadata about images.
• image store: Stores actual image blobs, can be local filesystem or Swift.

Identity (Keystone)

Keystone provides authentication authorization for managing users, accounts and role information for all OpenStack services. It also provides catalog of available services. Keystone provides an authentication token that is passed between services after validation for users to be able to use OpenStack resources. Keystone can also integrate with existing backend director services like LDAP.

Dashboard (Horizon)

Horizon provides a web-based graphical interface to access, provision and automate resources for administrators and users. It is implemented as Python/WSGI/Django web application running on Apache Web Server. As an alternate, custom command line scripts can be developed using OpenStack API.

In the next blog post, I plan to perform a quick install of OpenStack.

OpenStack: The Building Block for Private Cloud

I am back after an extended break from blogging about data storage topics. As I am no longer working at Quantum, I am free to blog about the recent developments in data storage without any concerns.

Last year, I started using cloud servers on Digital Ocean for PeerCube and worked with Amazon Web Services during Coursera's Introduction to Data Science course. Since then, I became very interested in private clouds and methods to establish and manage them.

With the rise of server virtualization, users and applications ability to spin up and spin down pre-built images as needed, and the success of Amazon Web Services (AWS) public cloud, there are fewer reasons for IT administrators to actively manage pooling and allocation of IT infrastructure resources. While there are several commercial (VMware vCloud) and open source platforms (Apache CloudStack, Eucalyptus, OpenStack) that let you build Infrastructure as a Service (IaaS) private cloud, I am particularly excited with the developments happening in OpenStack community.

I was sold on OpenStack as soon as I was able to install a DevStack environment on an old laptop and spin up images within couple of hours. Since then, I have been exploring OpenStack documentation and source code to understand this platform better with private cloud as a specific application in mind.

OpenStack Components

The modularized architecture of OpenStack includes following service components:

1. Compute (Nova)
2. Object Store (Swift)
3. Block Storage (Cinder)
4. Neutron, formerly Quantum (Network)
5. Image (Glance)
6. Identity (Keystone)
7. Dashboard (Horizon)

In the next blog post, I will elaborate on each services components. Being from data storage industry, I am very interested in learning about the storage service components of OpenStack in-depth.

As I realized that I learn best by writing and sharing, future blog posts will be my journey toward understanding OpenStack and how to use it to establish and manage private clouds.

Book Resources

• OpenStack Operations Guide
• Deploying OpenStack
• OpenStack Operations Guide

Why does CORE fail? Part 2

... Continuation of my previous post on CORE deficiencies and how it could be improved upon.

What is CORE?

Let's look at originally defined CORE equation.

CORE = (S x R x V)/(C x tc)

where,

S = The capacity being reduced in TB. Dave in his post fixes the S value at 100TB to compare all solutions.

R = The percent reduction achieved. Dave shows the R value in decimal for different solutions, we can assume though R is described as percent reduction, decimal R is used in calculating CORE.

V = The value of capacity being saved. Though, Dave doesn't list the V values used for different solutions, it is not difficult to reverse-calculate this value using other parameters listed in his table.

C = The cost of solution doing the reducing.

tc = The elapsed time to compress the capacity. As covered in my last post, I consider this parameter to be stated incorrectly, incorporated inappropriately and irrelevant to the CORE. In place, a better parameter would have been the elapsed time to write.

Three things stand out in this CORE equation:

1. CORE equation assumes first-order relationship with its variables. It may seem that for a specified value of S, the high CORE score can be achieved by achieving high data reduction (R) and the value of capacity being saved (V) and reducing the cost of solution (C) and the time to compress (tc).


2. CORE equation has variables (S, R, V) in numerator that are normalized for solutions without data reduction but no such adjustment is made for variables (C, tc) in denominator.


3. CORE equation is composed of dependent variables instead of independent variables.

Isn't V dependent on S and R?

V is defined as cost per TB (Ct) times amount of data reduced (Sr), according to the description of the math for CORE. Amount of data reduced (Sr) is the capacity being reduced (S) times percent reduction achieved (R).

V = Sr x Ct = S x R x Ct

Substituting V in original CORE equation,

CORE = (S^2) x (R^2) x Ct / (C x tc)

To a large extent, this modified CORE equation is composed of more independent variables than original one. Obviously, it is no longer a first order relationship with S and R.

What is interesting with CORE equation is that amount of data reduced has been included twice, once as amount of data reduced and then again as part of cost of amount of data reduced.

What is the CORE value for a solution with no data reduction technology?

For,

S = 100 TB,
R = 0% as there is no data reduction,
V = $0 as there is no capacity being saved,
C = 0 as there is no data reduction technology in play so there is no cost of data reduction solution, and
tc = 0 ms as there is no compression of data taking place,

CORE = (S x R x V) / (C x tc) = (100 x 0 x 0) / (0 x 0) = 0/0

CORE = 0/0 (indeterminate) this expression has no meaning.

You may agree that a relevant CORE value for a solution with no data reduction technology should be 0 or 1. It also makes sense in calculating value of a data reduction solution to have a solution with no data reduction as baseline.

How can we avoid division by zero?

1. Replace tc with tw or (tw + tc)

An equation that takes in to account time to write (tw) instead of or in addition to time to compress (tc) could help avoid division by zero when there is no compression/deduplication being used as even baseline solution with no data reduction will have a non-zero time to write. Either tw or (tw + tc) will be a better choice in place of tc in original CORE equation.

2. Redefine tc and tw

Of course, as originally defined in Dave's post, tc is time to compress the smallest unit compressed in the solution (e.g. file or multiple files or blocks) which ignores the variation in tc due to variation in the size of smallest unit across various solution. I recommend changing the definition of tw and tc, respectively, to time to write and to compress S amount or certain % of S, the value of S should remain same across all solutions. This will remove the parameter dependency on smallest unit compressed and normalize parameter across same amount of S.

3. Redefine C

As originally defined, C is the cost of data reduction solution. As Dave' post indicate NetApp doesn’t charge for ASIS – we took a percentage of the array’s cost, we can safely assume that C is only the cost of data reduction part of the solution, and not the whole solution. In this scenario C = 0 for a solution with no data reduction, thus making CORE value indeterminate again.

An equation that takes in to account the total cost of solution, i.e. cost of solution with no data reduction plus the cost of data reduction solution will help avoid division by zero. Of course, for a data reduction solution that uses existing storage, the total cost of solution will be net present value (NPV) of existing storage plus the cost of data reduction solution. Even better, subtract cost of capacity saved (V) from this cost instead of using V in numerator will result in Net cost of solution.

A better CORE equation, may be?

CORE = (S x R) / (C x tw x tr)

where,

S, R and V are same as originally defined.

C = Net Cost of Solution = Cost of data reduction solution + Cost of capacity used after reduction

Cost of capacity used after reduction = S (1 - R) x Ct = (S x Ct) - (S x R x Ct) = (S x Ct) - V

tw = time to write a pre-defined storage capacity or fraction of S

tr = time to read a pre-defined storage capacity or fraction of S

Of course, some may object to not including read/write ratio, there is no reason why read/write ratio shouldn't be included.

In the end, a CORE equation that is function of Storage Capacity (S), Percent data reduction (R), Net Cost of Solution (C), Read/Write ratio, Time to write (tw), and Time to read (tr) will be more valuable than the originally defined CORE equation. Of course, a lot more work is required to determine the interdependency of these variables.

Why does CORE fail? Part 1 - Response

Steve Kenniston of Storwize made detailed comment in response to my last post Why does CORE fail? Part 1. I thought my response to his comment deserved a separate blog post. Frankly, I haven't kept up with developments at Storwize since May 2007 when I last wrote a series of blog posts on Storewiz so I don't claim any knowledge of current Storwize solution.

First, I am not so sure that time to 'uncompress' ... is a valid parameter IF all solutions are being compared identically,....

The time to decompress/reconstitution is as much important, if not more, than time to compress/dedupe. The compression/deduplication can be managed 'internally' to keep up with write expectations of applications and users whether through delaying writes just enough to allow data reduction in-band or through data reduction after writes complete or some hybrid approach. But, the read expectations must be met in-band so any decompression/reconstitution need to take place correctly and completely in the expected time. A solution that requires lower time to decompress should be rewarded in same fashion as a solution with lower time to compress being rewarded in CORE.

... First I think we can all agree that decompression or rehydration is faster than optimization (compression, deduplication). ... the performance of time to 'compress' (I prefer optimize) and then cut the time in half and call this time to rehydrate. Now apply the formula. I would assume that the new CORE value would come out very close as they are now.

I am not so sure of time to decompress/reconstitute being faster than time to compress/dedupe or being 50% of time to compress/dedupe as I haven't heard of a solution or seen data yet that supports such claim. Actually, the relationship may be reverse specially for solutions with large amount of compressed/deduped data and high data reduction ratio. Only related published data, I am aware of, is that of read speed being direct function of the smallest unit used for decompression/reconstitution - larger the unit size, higher the read speed.

As I questioned in my last post, are time to decompress and compress proxy for time to read and write from data reduction solution? If it is the case, CORE could be improved upon by including actual time to read and write (instead of time to decompress or compress) or including time to decompress/compress as penalty over normal read/write with a solution that has no data reduction technology - in essence, additional cost in the form of lower read/write performance in exchange for higher storage efficiency.

Also, without understanding how the solution works it is very difficult to debate the merits of the value of performance on that solution. ...

If CORE stays with the parameters that can be judged externally for a solution, it will be more relevant and valuable than trying to incorporate parameters internal to a solution like time to compress (tc). A CORE based on externally measured parameters like reduction ratio, read and write performance, and cost of solution over a range of storage capacity and time may produce a better value indicator. Any attempt to include internal mechanisms weakens the CORE due to lack of complete information and understanding of every solution and rapid changes in technology and techniques incorporated in such solutions.

How can you possibly say that a post process solution that has users: 1) Buy full storage capacity (vs. less capacity with an inline solution) ...... is a good solution? ...

Please read my post again. I never claim any one solution is better than other. CORE includes cost of solution as a parameter which supposedly should penalize the solution that includes more storage than required by other solutions.

Step out of the vendor shoes for a moment and put yourself in the shoes of the customer. Which would you want?

As a customer, I want a solution that will provide additional storage efficiency at reasonable cost while meeting my expectations for read and write performance, safeguards my data and doesn't require additional management overhead. Anything beyond that is vendor coloring the customer expectations to fit it's solution.

Why does CORE fail? Part 1

Recently, David Vellante at Wikibon wrote in his blog post Dedupe Rates Matter ... Just Not as Much as You Think about his Capacity Optimization Ratio Effectiveness (CORE) value for ranking dedupe/compression/capacity optimization solutions. He also applied CORE to few dedupe solutions for primary storage.

As I commented on his blog, right away I noticed that CORE formula had an important parameter missing - time to uncompress/reconstitute (hereafter referred as time to uncompress) deduped data. It is an important parameter that impacts the rate of reading data from dedupe solution by applications/users. As time to uncompress need to be happen inline for both inline and post-processing solutions, logically there will be no major discrepancy in using time to uncompress and reading data from a dedupe solution interchangeably.

Is time to compress/dedupe also proxy to rate of data written to dedupe solution?

Another important parameter is rate of writing data to a dedupe solution as applications/users have certain expectations on how quickly data must be written to a storage system. David includes time to compress (tc) in his CORE calculation, what I assume, as a proxy to rate of data written to dedupe solution. I may be wrong as I didn't see an explicit statement about why time to compress/dedupe is important.

In my opinion, he incorrectly assumes the impact of time to compress/dedupe (hereafter referred as time to compress) to be same across various dedupe solutions whether inline or post processing solutions. The time to compress impacts the rate of writing data, more so, for a dedupe solution that uses inline processing. There is no impact on rate of writing data for post-processing solutions. So, to have apple-to-apple comparison, David need to either use the rate of writing data across all solutions or include time to compress data as penalty for inline solution due to slowing down the rate of writing data.

The low time to write data is a requirement of applications/users which inline solutions meet by reducing the time to compress as much as possible (possibly at the expense of lower dedupe ratio). Post processing solutions meet the same requirement by delaying the compression/deduplication for later (possibly at the expense of additional capacity required for storing pre-deduped data).

Including time to compress data in CORE calculations without discrimination inaccurately biases the CORE toward inline solutions. Just because a solution have sub-ms time to compress in-band doesn't mean it should be rewarded over a solution with few ms time to compress out-of-band.

Assuming that time to compress in inline mode and post processing mode are equivalent, in CORE calculation, is flat out incorrect.

Why is Time to Compress being used as Time to compress the smallest unit compressed in the solution (e.g. file or multiple files or block)?

Is a dedupe solution that compresses 16KB block size in 0.001ms better than a solution that compresses 64KB block size in 0.003ms? The CORE fails right here.

For all other factors being equal, a solution that claims 0.001ms for compressing 16KB (smallest unit for the first solution) will produce higher CORE value than a solution that claims 0.003ms for compressing 64KB (smallest unit for the second solution). As specified currently, the time to compress, in turn CORE, doesn't take into consideration the variation in different unit size used by different solution. Is the CORE formula assuming that compressing/deduping in smaller units better than in larger units?

The smallest unit compressed varies across solutions by a wide range, even >1000x factor. The time to compress should be the amount of time it takes to compress a specified storage capacity and should be normalized across all solutions for CORE to be of any value. Comparing time to compress 16KB units versus 64KB units is like comparing oranges-to-apples. For 1MB data, in first case 64 units will need to be compressed (0.064ms) versus 16 units in later case (0.048ms). CORE using time to compress/dedupe without taking into consideration the unit size will penalize the second solution incorrectly.

In next post, I will further look in to CORE and take apart CORE formula ...

Peril of Working in Cloud

Sorry! We are experiencing technical difficulties and cannot show all of your documents.


Do you have local backup copies of everything important you store in the Cloud?

Any Vendor Strategy, why not?

Initially, I was going to post a comment on Chris Evan's recent post 2V or Not 2V (vendors this is). With the increasing length of the comment, I decided to turn it in to a blog post of my own. Chris succinctly covered the operational aspects and challenges of multi-vendor strategy.

The challenge is how deep do you go in your environment to have multiple vendors. Do you want to have multiple vendors for,

• only large items like storage subsystems?
  
• smaller stuff like HBAs and switches too?
  
• commodity type stuff that has little differentiation among vendors?
  
• specialized products?

Just because you have multiple vendors, doesn't necessarily gives you $ bargaining power. Bargaining power comes with the transaction volume, transaction size, transaction frequency and your value to the vendor.

At the smaller end, though you can achieve better operational efficiency by standardizing on single vendor, you don't have the volume and size for a single vendor to take you seriously. Unless by consolidating all your purchases you get the volume and size to be valuable to a vendor, why not just buy the best-of-breed solutions?

How much operational efficiency are you going to gain by buying three Clariion versus one Clariion, one 3Par and one Compellant?

At the high end, single vendor strategy hinders your ability to adopt innovation and new technologies with minimal gains in operational efficiency (remember large teams can be split among multiple vendors if needed) though you may be valuable to the vendor and get better pricing. How much operational efficiency are you going to lose by adding three 3Pars to couple of dozen AMS, you already have?

I have seen, heard and experienced enough horror stories to believe either single or multiple vendor strategy for any one organization is a right strategy. I favor Any Vendor strategy where your decisions are driven by the best solution that meets your need and not a solution from a pre-selected vendors that somewhat meets the needs.

Adaptec Advisors are Back!

Adaptec PR firm sent a note mentioning that Adaptec Storage Advisor's blog is back! Check it out.

I am also trying to get back to updating my blog after a long hiatus. Hopefully with some small and quick blog posts on regular basis, my writing habit will establish. In the mean time, enjoy the sights from my various trips.

How do you overcome writing drought?

Online Backup Services - Six Questions

During my visit to Denver few weeks ago, I had the opportunity to talk with folks working with online backup and archive cloud services. Some of my impressions from these discussions are interesting and worth sharing. These are based on what I heard from professionals working for or providing services to online backup service providers. These are not result of a full-blown survey, and at best anecdotal. You are welcome to respond to these questions if you like via comments, emails or your own blog post.

Q1: Who are the primary adopters of Online Backup Services?

Individuals and small businesses.
Entities with fewer than a dozen workstations .
Few with a centralized server.

Q2: What was the primary backup method before adopting online backup?

None.
A USB key or USB attached disk drive.
Few with a share on another workstation.

Q3: What was the offsite backup strategy before adopting online backup?

None.
A Floppy, USB key or CD with important files.
Few with a mobile HD.

Q4: What is the subscription and retention rates for online backup service?

High subscription rate.
Very low retention rate.
Most abandoned service within few weeks.

Q5: What are the primary reasons provided for discontinuing use of online backup service?

Excessive use of Internet connection.
Backup takes too long.
Poor experience during primary use of workstation.

Q6: What was the backup method after discontinuing online backup?

A USB attached disk drive.
A NAS device on network.
Few with no backup method.

Summary

Overall, online backup services seems to be a great way to introduce backups to people with no prior backup methods as only few reverted back to no backups after discontinuing use of online backup service. Tape is non-existent in environments that are finding online backups attractive. Despite heightened awareness of online backup service, the low bandwidth connection to Internet continues to be main hurdle in retaining subscribers, a focus on spending limited resources on sales improving or cost reducing services over a fear-based buying decision. A comment I heard was,

I prefer to allocate 50% of Internet bandwidth to VoIP services that reduce my telecommunication cost instead of to offsite backup.

Denver Visit, New Piñata & Scalability Videos

Week in Denver

I will be in Denver this week till Friday June 27th. Unfortunately, I will miss nPost Golf 2.0 event in Seattle.

Despite a busy work day schedule in Denver, I am looking forward to seeing some friends and colleagues also. If you are a fellow storage blogger or reader or working on a cool storage technology and located in Denver area, ping me and we can meet one evening during my visit.

New Piñata for EMC & IBM

Recently, a reader alerted me to new Data Domain blog Dedupe Matters written by Brian Biles. Welcome Brian to the world of Bloggers. Lets see how quickly EMC and IBM bloggers make you the new piñata like they did to HDS bloggers. ;-)

In any case, it’s a nice change from their rumored no blogging policy. Hopefully, blogging at Data Domain will go beyond people in Ivory Towers.

Video of Presentations from Google Scalability Conference

Google already uploaded the videos of presentations from last week's Google Scalability Conference. I also plan to discuss some of the presentation topics in further details as time permits.

Welcome Remarks by Brian Bershad



GIGA+ by Swapnil Patil



HPC with NetworkSpaces for R by David Hendersen



Chapel by Brad Chamberlain



SMP via Transactional Memory by Vijay Menon



Communicating Like Nemo by Jennifer Wong



Maidsafe by David Irvine



CARMEN by Paul Watson



Scalable Wikipedia by Thorsten Schuett

Google Conference on Scalability - First Impression

As expected from conference schedule, Google conference turned out to be a technical event primarily focused on parallel programming and infrastructure scalability. At last minute, Google decided to merge two tracks in to one. Though, I got to attend all the sessions, they felt time-compressed and rushed. I was surprised to see lot of attendees who came from outside Seattle. I met quite a few people from Bay area, Canada and Europe. I enjoyed the sessions though some audience members commented about very technical nature of the conference compared to previous year. As Brian Bershad, Google commented in his welcome speech, the challenge is to find technologies and solutions to scale handling search queries from 600 million to 6 billion. And, I came away better informed on different challenges and potential solutions we may see down the road.

I also sat down and chatted with Robin Harris. We decided to forego making a video of our conversation. I am not a big fan of talking head videos or podcasts unless they leverage the unique values of these methods not available through written words or pictures. And who wants to listen to two storage bloggers chatting about nothing. I find them miserable myself so why put others through the same misery.

In my opinion, three sessions: CARMEN: a Scalable Science Cloud [PDF], GIGA+: Scalable Directories for Shared File Systems [PDF] and maidsafe stood out at the conference from infrastructure scalability perspective. Communicating Like Nemo was very entertaining. The common theme in audience questions on most infrastructure presentations was reliability, availability, scalability, and security of the offered solution. It is a good indication of what is on the mind of people when evaluating new infrastructure offerings. With the popularity of hashing in storage of data, speeding up hash lookup is becoming an interesting problem for scalability.

David Irvine's session on maidsafe was the only session where a speaker white-boarded most of the presentation. His confidence and knowledge was commendable. Not many speakers can pull off white-boarding 80% of presentation with 100s in audience. Comparing maidsafe with ant colony was an interesting way to show scalability and simplicity of solution. Maidsafe solution seems to be in same category as RevStor, Seanodes, Cleversafe, Oceanstor, Farsite and several others that are trying to leverage storage across 100s and 1,000s of distributed nodes in a peer-to-peer or quid pro quo network, a solution most likely attractive to players in cloud and web distribution market.

Online Backup: 100% Install

My last post Online Backup any different from Traditional Backup for Laptop/Desktop? was quickly turned in to us vs. them argument by Beth Pariseau in her blog post Blog dialogue: Online vs. traditional backup. I guess my curiosity and conversation starter about slow adoption of online backup didn't come across clearly.

… Gupta probably has “too much” experience with backup clients to necessarily see things from the SMB customer’s point of view. For him, installing a backup client isn’t a big deal–for some, it might be enough of a reason to let somebody else deal with it.

Initially, I thought about pulling Tony on her. On a side note, I wonder why Tony spills coffee every time Hu sneezes.

More I analyzed her statements, more I realized her opinions most likely resulted from what she heard as a storage news writer and from whom instead of her own experiences. Keywords like SMB are a good giveaway whom she is listening to. Not many practitioners try to segment customers with mile-wide brush. ;-)

Lets start with addressing her installation related concerns. Do online backup services magically appear and start working on your laptop/desktop by themselves? No, someone has to download and install them. Only backup clients that come pre-installed on your system are the ones that don't require install. As I understand, there are two main backup clients available that don't require installation and readily available to users, one provided by Microsoft with Windows XP (Windows Backup) and other one provided by Apple with Leopard (Time Machine).

Lets add configuration of the backup client to the part of "difficult to install" equation. Configuration of Mozy Pro [PDF 46 pages] and Windows Backup [Web page - 6 pages if you decide to print], are available online for your review and comparison. Of course, Time Machine is so simple to configure that even someone like me, who misunderstands backup needs of SMB according to a marketer, implemented on MacBook without instructions. BTW, AppleInsider article Road to Mac OS X Leopard: Time Machine is a good overview of Time Machine.

You be the judge how difficult each one is to install and configure.

As I wrote in my comment on Beth's blog, my intention is not to promote one method over another, just to show similarities and question the current implementations. Hopefully, these posts are setting the stage for future opinions and conversations that will help improve current BaaS offerings and develop new ones.

More to come.

Online Backup any different from Traditional Backup for Laptop/Desktop?

Recently, Beth Pariseau wrote in her blog post HP unveils unlimited online storage for SOHO market that bandwidth is one of the hurdles in adoption of online backup services.

Like most online storage offerings to date, this offering is small in scale and limited in its features when compared with on-premise products. Most analysts and vendors say online storage will be limited by bandwidth constraints and security concerns to the low end of the market, with most services on the market looking a lot like HP Upline.

Though, it is a validation of my thoughts expressed in blog post Bandwidth, one hurdle in adopting Cloud Storage, I am not totally convinced of bandwidth being the root cause of limited adoption. There may be something else hindering adoption of online backup services.

Recently, Scott Waterhouse, an EMC blogger also has been discussing the virtues of Mozy, an online backup service (acquired) by EMC. I agree with his argument about the challenges of traditional backup clients in post Mozy as the Future of Backup.

Big business has a lot of data on laptops and desktops. Traditionally, installing backup clients on these systems has been costly, full of headaches, and generally causes more problems than it solves. The consequence of this is that most folks just don't protect them.

Is Mozy client any different? Is there any difference in installing, configuring, using and maintaining traditional backup client versus Mozy client on laptop/desktop? Nothing, I noticed after reading his posts.

My intention is not to pick on Mozy or Scott but there is nothing unique in most Online Backup Services that couldn't be in traditional backup for laptop/desktop. At least traditional backup also come with peace of mind that all backups are stored on company's own infrastructure. In last few years, I tried over a dozen online backup services in addition to putting up with traditional backup clients for laptop/desktop and I don't see much difference among the two.

IMO, most online backup services are just taking existing on-premise backup strategy for laptops/desktops and repackaging it to run backups to somebody else's infrastructure instead of your own. What do you think?

Storage Jobs @ Startups

Recently, Nathan Kaiser at nPost contacted me regarding his new widget displaying Startup Jobs on blogs. As sidebar on my blog is already too long, I decided to include his widget in a blog post. Try it out and let me know your feedback (positive and negative).

P.S. If you are using a RSS reader like Google Reader and don't see the widget, please visit my blog. While I am writing this post, I am not sure if widget will show up in the blog post either. In case it doesn't, please visit nPost Startups Jobs site to check out the startup jobs. Use keyword "storage" to find storage jobs at startups.

Is number of objects true indicator of Amazon S3 growth?

In my last blog post, I estimated the data stored on Amazon S3 in exabyte range using 18 billion objects stored reported by Amazon CTO, Werner Vogels in his blog post.

In retrospect, it was an over-estimation by several order of magnitude (my bad) that was promptly corrected by MikeDoug using another data point AWS revenue. MikeDoug estimated (comment excerpts below) the data stored to be in 20PB (petabyte) range, way short of my estimates and may be more closer to reality.

No, doubt, it is still a significantly large number for a service that is only few years old. But, S3 growing up fast may not be as obvious from growth in stored objects as Vogels would like us to believe.

A recent report puts ALL of AWS at the 50 to 70 million in revenue for the year.

Let us pretend that, of the 70 million, 40 million in revenue was attributed to S3 alone for last year. That would be $3,333,333 a month for S3. This converts to 22,222,222 gigabytes, or 0.02 exabytes.

Other interesting tidbits if S3 has 20PB of stored data, 18 billion objects and 330,000 registered developers:

On average, each object is only storing about a megabyte of data. This number seems quite low so either deleted objects are being included in the published number of objects or developers are keeping object size low to prevent transfer timeouts.

On average, each developer is only storing 54GB of data. Considering some services like SmugMug are storing terabytes of data on S3, most probably there are lot of registered developers either not using S3 actively for storing data or have services under development.

How much data is in Amazon S3?

Today, Werner Vogels mentioned in his blog post Happy Birthday, Amazon S3! about the second birthday of Amazon S3 and also shared that by Jan 2008, S3 is storing 14 billion objects. I am not sure why Werner and others at Amazon are so cagey about sharing actual storage capacity used in AWS. In the past, I also have met with either silence or "trade secret" or "competitive advantage" response to my inquiries.

In my opinion, it only creates room for speculation as I am going to do with this post. So, how much data is stored on S3?

My initial guesstimate for stored data volume is between 14 and 70EB (Yes, EB is Exabyte) based on the published information about the size of individual object being one to five GB. Doesn't it seem very high? At first, it did to me. I have been trying to come up with alternate methods to estimate stored data volume like the typical size and type of data being stored by various services that are using S3. Even with an average value of 100MB per object, the stored data volume comes out to be 1.4 Exabyte, still a huge number for such a young service.

What is your estimate? Any suggestions on estimation method to arrive at more accurate number for data volume stored on S3.

Considering that S3 may be hosting Exabyte or more of data with in two years of existence, no wonder all established vendors EMC, IBM, HP and Dell are salivating on getting a piece of the "Cloud Storage" pie.

Bandwidth, one hurdle in adopting Cloud Storage

This weekend, I read NY Times article Video Road Hogs Stir Fear of Internet Traffic Jam.

Last year, by one estimate, the video site YouTube, owned by Google, consumed as much bandwidth as the entire Internet did in 2000. …

In a widely cited report published last November, a research firm projected that user demand for the Internet could outpace network capacity by 2011. …

Moving images, far more than words or sounds, are hefty rivers of digital bits as they traverse the Internet’s pipes and gateways, requiring, in industry parlance, more bandwidth.

While reading the article, it occurred to me that isn't bandwidth going to be the main hurdle in adoption of storage in the cloud. When clients are not happy with 10/100/1000Mbps connection with application/server/data center, how can they be happy with DSL/Cable/T1/T3 connection to the cloud? I am sure everyone has felt the pain of trying to transfer large datasets over the Internet.

If you review the introduction and growth of various Amazon Web Services (AWS), a comparatively established cloud player, you will notice very limited use cases of Simple Storage Service (S3) on its own with clients outside the cloud. Most S3 usage is fronted by another AWS in the cloud such as Elastic Compute Cloud (EC2). Such combinations overcome the challenge of transferring large amount of data between storage cloud and an application/server outside the cloud over Internet. For cloud storage to be successful, it need to be in the same cloud with application/server or connected to application/server cloud with high speed link.

Any technology that can reduce the data transfer between the cloud services and clients outside the cloud will be the big beneficiary in this trend. Caching, Compression, and Data De-duplication will most likely benefit in the near term. And, the future seems to be very much like the past aka mainframe - Desktop Virtualization, Streaming, and On-the-Fly Visualization.

So, how will new cloud players like Nirvanix, EMC Mozy and Rackspace differentiate?

Are you using online storage services and how?

Last week, Ethan Oberman alerted me to his online storage service SpiderOak after coming across my post Online Backup Services - What's Next?. Since my post last year, I was contacted by several online backup and storage service providers.

Ethan highlighted differentiation of his service primarily in the area of file versioning, delta transfer, secure sharing across machines and users, and zero knowledge security.

Our approach to online backup and storage varies greatly from our competitors - creating a personalized network concept as opposed to simply online backup. …

Similarly, last year, Marcus Hartwell introduced me to Diino service that also focuses in the area of online backup, storage and sharing.

Most online storage services, since late 90's, are mainly focused on serving one or more activities in data management:

1. Backup,


2. Sharing, and


3. Access.

These services are primarily targeting consumers and small businesses, a bottom up approach with hopes that over time mass adoption will result in acceptance by enterprise IT departments. Strangely, none have been able to make significant impact and gain wide-spread momentum. As previously mentioned, dozens of them have come and go, and I am sure you noticed this trend too.

Though, I did try out several services for a short time, I just couldn't see any becoming part of my daily online routine. And, the main adoption challenges seems to be that either I need something that operates "invisibly" or integrates with my current tools and online activities.

Are you using online storage services and how?