This theme plays out in two primary ways -- one of which is storage's continuing alignment with fully virtualized servers and networks. And the other is how -- once we fully abstract logical from physical -- many more things are possible in the storage domain than we might previously have assumed.
This second post digs in deep on the second idea -- what new things are now possible once we fully embrace "virtualized storage"?
And one of those "new things" is an entirely new take on storage tiering.
Information continues to grow at a breakneck pace. Good economy or bad economy, many of the industry estimates are now pegging 50% enterprise information growth going forward. Some organizations are experiencing much, much more.
Yes, the costs of storage devices are coming down -- but nowhere near 30%. Storage operational and management models are getting more efficient, but not nearly fast enough on their own.
We, as vendors, tried to address this problem by arguing for better information management and governance models. Categorize your data, we pleaded, and manage it efficiently. We made the situation better, but not nearly enough to stem the information tsunami.
Although there will be exceptions, the default behavior we're forecasting is that most enterprise information will be largely un-categorized and largely un-managed. And we, as technology vendors, had better get busy on *dramatically* reducing the costs of storing enterprise information -- and do it in a way where very little effort on the part of IT is required.
Optimizing The "Long Tail" Of Information Storage
If you remember ILM (information lifecycle management) concepts from several years ago, the idea was simple -- progressively and automatically age your information to lower tiers of storage over time.
Even a few years back, this approach generally had three major problems:
First, many of the schemes required metadata-driven policies to make them work effectively. The categorization never happened, or was too difficult.
Second, the cost-savings payback wasn't as compelling as it could have been -- after all, we were pretty much working with ordinary FC disk, PATA disk and tape at the time.
Third, because of the lack of generic metadata and supporting automation, there usually was an enormous amount of heavy lifting to place the right information at the right place at the right time. Get it right, save a lot of money. Get it wrong, users unhappy with storage service levels.
So people tended to default towards putting information at higher service levels than were actually needed, as kind of an insurance policy -- which defeated the whole purpose in the first place.FAST In A Different Light
If we start to take all the storage optimization technologies, and put them end to end, a very interesting picture emerges.
Let's start with FAST -- fully automated storage tiering -- which basically breaks application storage into chunks, and dynamically puts the most popular information on flash, and the stuff no one ever accesses on SATA.
Big performance improvement (through the selected use of enterprise flash), and simultaneously big cost savings (through the use of massive SATA drives).
Now, let's take another bit at storage inefficiency. Let's add on virtual provisioning (a subset of which is called thin provisioning elsewhere), and only hand over physical storage to applications when it's actually being used, rather than when it's provisioned.
Yet another big bite at storage costs -- one that many people have seen in their own environment.
Let's go at it again -- let's use compression, single-instancing and data deduplication technologies to squeeze out all the different redundancies. Can't be used everywhere, but -- where it fits -- it's one more big bite out of storage costs.
We're not done yet -- not even close. A significant amount of enterprise information is used *very* infrequently. So infrequently, in fact, that the disk drives can be spun down, or -- at the least -- be made semi-idle. Another big chunk out of storage costs, this time it's power and cooling.
And, finally, at the outer reaches long tail of enterprise information, there's probably information you don't even want in your data center anymore. Much in the way many enterprises subscribe to "records management" services (basically, someone who trucks boxes of paper from your facility to theirs), there's no doubt we'll want to do the same in the digital world.
We'll want the ability to federate with external storage service providers who can provide trusted bulk information storage that's cheaper (and presumably safer) than internally providing this service.
And, of course, we'll want a convenient abstraction and associated tools to manage all of this in a transparent and efficient manner.
The New Storage Efficiency Mantra -- FAST, Thin, Small, Green, Gone!
Put it all together, and a rather simple yet compelling picture emerges.
FAST does the heavy lifting of putting the right information on the right media at the right time.
Thin (virtual) provisioning eliminates provisioning waste.
Compression, single-instancing and deduplication technologies eliminate information redundancy.
Spin-down saves power if the disk drives aren't being accessed very frequently.
And -- eventually -- the information gets shipped out to a specialized service provider as an option.
Now, is that some distant vision of the future? No, not really.
For example, consider EMC's unified storage platform -- Celerra.
FAST for Celerra became GA as of this announcement. Virtual provisioning (filesystems and iSCSI) has been there for a while.
The first wave of space-saving technologies are available today, more coming. Spin-down of SATA drives is supported, I believe. And there's now an Atmos adapter to ship bulk information off to either your own internal storage cloud, or perhaps someone else's.
Policy orchestration, in this case, is provided by Rainfinity functionality -- it establishes and enforces the policies as to when files move from one category to another, or back again.
Granted, we're doing this today by assembling a few different portfolio technologies. We have more work to do to make all of this more seamless and transparent, not to mention bringing it fully to the world of real FC LUNs and their demanding applications.
But, without a lot of squinting, you can clearly see where it's going, and far more of the pieces are now in place -- and usable today -- than most people realize.
Is ILM Finally Real?
Yes and no. In one sense, the original vision of ILM may have overpromised and underdelivered.
Too much depended on the ability of IT to partner with the business to classify information, assign metadata and implement information management policies. Sure, it worked well in some domains (e.g. email, e-discovery, document repositories, etc.) but not well enough in the general case.
This time around, it's far more promising. We have a much wider palette of storage technologies to use. We're smarter about what's important, and what's not. And we all now realize getting 80% of the way there with very little work is more attractive than getting 95% of the way there with a ton of work.
One thing hasn't changed, though.
The information beast continues to grow.
And How Does This Change The Role of The Storage Architect / Administrator?