Storage Is Software

The title of this post is a variant on the intriguing "infrastructure is code" meme from the #devops community. I think it's a useful idea to remind ourselves of -- especially as technology transitions.

Even though some of you reading this might thing the statement is blindingly obvious, it's clear that the vast majority of people think of boxes with blinking lights when you say the word "storage".

And I think this is going to be up for change sooner than later.

Things Change

Having now been directly involved in storage for over 15 years, I feel I can safely make a reasonable judgment when things are changing.

So let's go look at the current landscape ...

For starters, most storage hardware today is built out of the same industry-standard parts bin used by the server guys.  Yes, there are a few storage stalwarts trying to claim differentiation through this bit or that bit of unique silicon, but the secular trend is pretty obvious -- parts is parts.

Now, I think there's still room for useful hardware differentiation in areas like innovative architecture, or clever packaging, or using the latest merchant silicon chips, or perhaps more reliable manufacturing processes. 

All that being said, I think the opportunities for sustained differentiation through hardware prowess alone are becoming more rare over time.

And we all play in a very competitive market indeed  Much like customers won't accept dated or over-priced server hardware designs, they won't accept dated or over-priced storage hardware designs.

Thinking About Storage Software

At its most basic level, you expect to write information to a storage platform, and get it back again. 

You'd like to do so in a convenient format -- more traditional block and file formats, perhaps something newer like objects, even maybe something like tables.  That's a function of software, not hardware.

You'd like the integrity of the data protected from all sorts of bad things that can happen -- hardware failures, software failures, human error, the list goes on.  That's a function of software, not hardware.

You'd like to wring the maximum in performance and efficiency from the hardware you own: move the popular data to the high-perfrmance media, the less-popular data to cost-effective stuff, and wring the excess capacity out with things like compression and deduplication.

More software.

If you tend to think geographically, you'd like the right information in the right location at the right time if possible.  Whether that's to better protect, or improve user experience, or something else -- that's all software as well.

I could go on, but -- when you think about it -- just about everything we talk about that's new, interesting and useful tends to boil down to a software discussion. 

Sure, there are new hardware bits like faster processors, and enterprise flash drives, and newer 10GbE interconnects -- but it takes software to make all that stuff really useful.

The Impact Of Open Source

Much like industry-standard components and architectures set the floor for cost-effective hardware, I think open source software sets the floor for cost-effective software functionality.

There's still room to innovate in software, but you have to do it in areas that haven't been well-covered by the open source community.  And -- make no mistake -- it's a safe bet that open source software will be an ever-increasing part of our enterprise environments.

Resistance to either trend appears futile :-)

Separating Software From Hardware

We've just come to assume that storage software is inevitably woven to storage hardware.  But as the industry moves to more standard components and architectures, that's becoming more of a business model discussion, and less of a technology discussion.

Examples are starting abound, especially within EMC's portfolio. 

Our Atmos cloud storage platform is now available as a VMware virtual machine.  Run it on just about any VMware-supported hardware platform, and you've got a fully featured, next-generation distributed object-oriented metadata-rich policy-driven cloud storage environment

One could separately debate the merits of running Atmos storage software on a generic hardware platform vs. one that is specifically built for purpose, but that's more of a discussion around implementation choices -- and choice is good.

Many of you are aware that the Avamar client-side dedupe backup platform works basically the same way -- your backup target can either be a dedicated hardware device running Avamar, or the same functionality running in a VM on generic hardware -- it's your choice.

Going further, there's a much larger universe of EMC storage products just waiting to escape the confines of physical hardware: RecoverPoint, VPLEX, Celerra, Centera -- the list goes on. 

Even some interesting open-source choices if you go looking: for example, the EMC LifeLine stack which powers the increasingly more powerful Iomega unified storage devices.

So why aren't all these great things being done today?  Lots of issues, but the big one is -- it's hard!

Making storage software work predictably and reliably in a virtual machine takes substantial engineering effort.  And that incremental effort  has to be balanced against other investment opportunities: things like adding new features, or supporting new hardware, or perhaps deep integrations with other environments.

It's happening -- it's just not an overnight process.  Sorry to say, the future isn't quite here yet ...

Fast Forward Several Years

Imagine you're in charge of storage decisions at your company, and you're trying to put together a solution for part of your operation.

You might start by assembling a set of services you'll need to provide for applications and users.  You evaluate different software stack options. for functionality, price, reliability, support, ease-of-use, integration, APIs, etc.

You do so by composing various storage software VMs, and putting the resulting stacks through their paces.  Basic presentation services (file, block, object, etc.).  Some replication stuff, maybe some auto-tiering and or intelligent archival stuff.

You test features and functions, integration points and management interfaces.  All using virtual machines in whatever test bed you've got handy.

No need to consider storage as hardware just yet.

When you're ready to implement, you've got more choices: you can stick with the storage-software-in-a-VM approach, or perhaps consider purpose-built hardware if your needs so dictate. 

Functionality first, implementation second.

Farther Down The Line

The migration of storage functionality from hardware to software will likely change how storage hardware itself is built.  At the low end of the market, all-in-one storage can learn new tricks simply by invoking new elements of a (presumably virtualized) software stack.

And at the high end of the market, it's not hard to imagine larger, dynamic pools of virtualized storage capabilities that flex both resources and functionality much the way virtualized servers do today.  To be fair, though, that's a reasonable description of what a VMAX and VPLEX does today.

Indeed, we can easily see storage software functionality running flexibly where it makes the most sense -- on a general purpose all-in-one storage hardware platform, or perhaps as a set of virtualized tasks in a server farm, or perhaps on an appliance dedicated to a task -- or any combination as needs shift.

And that's going to force some changes in thinking all around.

Final Thoughts

The runaway success of VMware has caused many of us to think of "servers" in terms of software images that are invoked as needed.  The hardware is still there, and it needs to do its job, but we think about it differently.

Will we learn to think of storage the same way?

Comments

[Ray Lucchesi]

Chuck,
Call me old school but I can't help but think there is still some place in storage architectures to differentiate with some form of silicon. AND remember with FPGAs and their ilk, generating new hardware is as simple as compiling some code with design checkers and other software oriented tools (although the cost is considerably higher than just running code).

The question, can FPGA hardware be as cost effective as doing everything with general purpose "server parts". I tend to think the answer to that is yes and the trends in FPGAs like hardware are similar to the trends in standard processor/interface off the shelf chips - Moore's law applies to all of these. So as each new generation of FPGAs come a long, you can do more with them.

There's no denying that FPGAs will always cost more than just running code on standard server parts but if you can get enough functionality enhancement from it I got to believe it's worth it.

Ray

[Chuck Hollis]

Hi Ray

I think you've only exposed half the debate. The other half is "will people pay a premium for what it does?".

There are plenty of examples where this is the case, and plenty of example where it's not. Indeed, folks like Apple can make many of us re-think the entire model.

My discussion, though, was specific to storage, and in many senses, it's following the broader server market. How much opportunity is their for FPGA-style differentiation with servers these days? Not a lot, I'd offer.

Thanks for the comment --

Chuck

[Behrens]

Good viewpoints. I also know that customers want interoperable storage devices. Standards are needed to avoid having an inventory of bricks in the future. This is counter to making a better widget (and more sales), however at some point, the distribution of data should simple, secure, and ubiquitous. It is great that EMC and other companies support standards organizations (like SNIA, OGF, etc).

[Ray Lucchesi]

Chuck,

Decided my response was to long to do here, so moved it to my blog. But my answer is that people will pay a premium if it's worth it.

Ray