Storage is sexy again. In the information economy, it's a fundamental enabler of new wealth and value, much like networks have led previous revolutions.
At a high level, there are two fundamental storage technology transitions going on that bear close scrutiny.
One is the inevitable transition from tape to disk for secondary data. Say what you will, but it's happening -- and happening fast. For example, EMC's BRS division is a key beneficiary of that particular trend, amongst others.
And the second is the more-fascinating transition from disk to flash: from rotating rust to semiconductor sand.
Today's announcements at EMC World -- once fully appreciated -- will hopefully make people realize that this second transition is happening far faster -- and is far more impactful -- than perhaps once thought.
Three Years Later
The flash-as-disk-in-the-enterprise party got seriously started back in January 2008 when EMC became the first vendor to support the newest enterprise flash drives alongside traditional disk in the DMX.
Oh, the howls we heard from our competitors. It's too expensive. It's not reliable. No one will ever want it. And -- if you listened carefully -- vague vendor statements that they would inevitably be forced to do the same.
Well, as we stand here today, almost all storage array vendors have gotten on board to a certain degree (with some notable exceptions!) realizing that -- yes -- flash is becoming the new disk.
Multiple flash-based announcements at EMC World today; everything from storage media pronouncements to the code-named Project Lightning. So let's get started ...
From SLC to MLC
Up to now, only single-level cell flash technology was considered suitable for demanding enterprise use. And three years is a *long* time when it comes to semiconductor evolution.
The current crop of third-generation SLC drives are a far cry from the first versions we shipped back in 2008. They're much faster across a far wider range of I/O profiles. They've far exceeded anyone's reliability expectations. They've become much denser (and hence less expensive) at a faster clip than anyone expected.
But the real technology breakthrough we've all been waiting for is MLC-based drives that perform as well in demanding environments as the current SLC-based ones. Multi-level cell technology means the potential for storing more bits for less money. Much lower cost means much faster adoption.
One of our announcements at EMC World is pretty impactful, especially in the longer-term view: a new crop of enterprise-class MLC flash drives are now starting to come to market, and we think they're now becoming good enough for the demanding workloads previously only supported by SLC-based drives.
The eventual implication is clear: the accelerated pace of price erosion should increase dramatically. Put differently, we're now on a new price/performance curve.
If you're thinking out a year or two, you should be thinking "MLC drives" for all of your high-performance and medium-duty workloads. Go out a few years, and disks quickly become the new tape, but in a different way: rotating disks are now only used for your really big data.
SLC will likely still have a role, more around high-speed write-intensive caching than persistent storage (think FAST Cache in the VNX, for example). It isn't going away anytime soon, but its role is likely to change in the storage heirarchy as enterprise-grade MLC drives push their way in.
So, in this new world of ever-cheaper semiconductor-based storage, what will we see?
The All-Flash Array
You may be skeptical, but the first customer request came in 2009:
Them: can you build us an all-flash version of one of your arrays?
Us: uh, yeah, of course -- what do you want to use it for?
Them: we'd really rather not say ...
Since then, interest has steadily picked up to the point where we're announcing our formal intention to build storage arrays that don't use traditional rotating disks at all. These are turning out to be very important applications where almost *all* the relevant data is hot. Automatic tiering and dynamically blending disk and flash just isn't interesting to this select crowd.
Today's all-flash arrays from EMC are simply specific configurations of existing architectures and software -- and intriguing in their own right -- but there's also an interesting opportunity to build a entirely new classes of storage arrays when you don't have to consider spinning disks anymore and traditional I/O interfaces any more :)
Sure, the majority of storage users will want a blend of both flash and disk, and have come to appreciate EMC's FAST secret sauce in getting to the benefits without a lot of heavy lifting.
But the directional statement here should be clear: we fully expect the demand for all-flash storage arrays to increase as (a) the costs associated with MLC-based enterprise flash drives drops dramatically, and (b) more and more workloads emerge in the "all data is hot" category.
Of course, if you're in a *real* hurry, we can get you one today ...
Redrawing The Boundaries Of Storage?
Perhaps of all the flash-based storage announcements at EMC World, the announcement of server-side storage flash integrated with array-side storage (code named Project Lightning) might get people thinking about things differently.
Lots of industry interest in some of the newer vendors who are seeing some traction with the idea of server-side flash-based storage. In one sense, it's the internal DAS vs. storage network discussion all over again, just this time with different storage media.
I've seen this movie before, and I know how it turns out :)
Server-side storage caching has been a fixture of the computer industry ever since I was a wee lad. For example, read caching is most effective when it's as close to the processor as humanly possible. But -- up to now -- it hasn't been really thought of as an integral part of the overall storage heirarchy.
In a nutshell, that's the new opportunity -- putting an intelligent and coordinated flash-based storage component *inside the server*, along with the intelligent software (largely based on FAST algoritms) that can shuffle data back and forth as needed.
The algorithms have to be good at a couple of things. First, attempting to predict what the server-side cache will want next, and prefetching it if possible. Same sort of things we've been doing on our arrays for over a decade. But there's no sense read-caching the same data in two places (server and storage) so there's an opportunity for additional optimization here.
Second, being smart enough to figure out which writes should be persistent, and which ones are transient. A database record update is persistent, and should be confirmed by the array. A write to a paging file or temporary index may be different. Again, lots of room for addtional intelligence here.
As I draw this for customers, I draw a picture of servers with flash-based storage, and then a picture of the arrays, and then a big circle showing the new boundaries of storage: some in the server, most in the array, and the secret sauce that makes it all work as one.
I'm sure there will be a lot of discussion around this going forward. I can't wait -- it's a brand-new storage architecture debate to consider, and those are *always* fun ...
Stepping Back A Bit
I was in the thick of things when EMC made our first "flash changes everything" announcement way back in January in 2008. We made some very bold pronouncements about the future back then, and we were met with the predictable skepticsm.
Well, if you didn't believe us back then, just maybe you'll believe us now ... flash is quickly becoming the future of storage. And the future has this way of getting here sooner than anyone really expected ...