Big news today for IT infrastructure architects -- and the businesses that depend on them.
Today's "Lightning Strikes" announcement from EMC represents a new milestone in giving our customers a new-found ability to deploy advanced infrastructure that's more efficient, more optimized -- and a helluva lot faster :)
It's a big deal. There's a lot to cover, so I'm breaking this into three posts.
In this post, I'm going to set context -- what the problem is, what server-side flash brings to the party, what challenges exist with existing approaches, and so on.
In the next post, I'm going to dive into the details of VFCache (formerly known as Project Lightning): what it does, how it's different, what's there now, what's down the road, etc.
And, in my final post, I'm going to introduce you to Project Thunder -- which *hasn't* been as widely talked about -- but, as you'll see, is a natural extension of what Project Lightning (now VFCache) is targeting -- only at scale :)
If you're into IT infrastructure at any level, go get a cup of coffee, and let's dig in together. Yes, there's a lot to digest, but you'll probably end up just as impressed as we all have been internally for a while now.
To Start With
CPUs (as best exemplified by Intel) follow Moore's Law. Every 18 months or so, we all get a big, healthy boost of CPU performance for roughly what the previous technology cost.
It's one of the best deals in tech :)
Not so when it comes to traditional storage performance. IOPS/spindle (the primary measure of rotating disk performance) has been largely limited to discouragingly modest performance increases over that same time period.
To put it mildly, this technology gap has always frustrated our customers. Buying faster servers won't necessarily deliver faster performance if reading or writing data becomes the bottleneck.
There are two historical approaches to delivering more storage performance. The first involves using lots of spindles and spreading the workload. But this can be expensive from a wasted capacity and increased footprint perspective. The other has been to use large non-volatile DRAM caches and smart algorithms, such as you'll find in the VMAX. Better, but still not ideal.
The technology answer to this -- as most people know -- lies in adapting flash memory to enterprise storage requirements.
While not entirely a new idea, its application is evolving fast -- as we'll see in a moment.
EMC Gets Onboard Early
While it was a game-changer in its own right, we ended up having to do an inordinate amount of missionary work -- made even more difficult by the extreme FUD on behalf of all the storage vendors who were caught flat-footed by this industry shift. At the time, we described enterprise flash as a new performance tier, which it was.
If I played back for you the silly, dangerous and ignorant things that were being said about the technology at the time, you'd probably laugh -- until you cried. While that's past history, it does illustrate what usually happens when a major vendor attempts to advance a significant technology considerably ahead of its peer group :)
The 2008 technology was quickly followed in 2009 with Fully Automated Storage Tiering (or FAST) which provided the important capability to automatically tier information between high-performance enterprise flash storage and more traditional rotating rust. First on the VMAX, and then on the CLARiiON, the game had advanced significantly.
In 2010 VNX went even further by introducing FAST Cache, which used the technology as a big hunk of non-volatile storage cache that (unlike competitive alternatives) could be used for both reads and writes, as -- obviously -- most everyone does both :)
The Magnifying Effect Of FAST
Since EMC's implementations basically allow a modest amount of enterprise flash storage to positively impact a far larger pool, we tend to keep track of two numbers. The first, ostensibly, is the total amount of enterprise flash we've shipped.
This slide shows a current figure of ~24 petabytes (24,000 terabytes) which is no small number.
The more interesting number is the leverage effect: that amount of enterprise flash is dramatically improving 1.3 exabytes, or 1.3 million terabytes of customer capacity. A little quick division shows an average "multiplier effect" of 54x -- on average, for every terabyte of flash storage you purchase, you'll be cranking the performance of ~54 terabytes.
Or, as we've long said, a little bit of flash can make a huge difference :)
And Considerable Contextual Intellectual Property
For example, we've modeled tens of billions of I/Os from actual customer application traces. We've taken the technology into many, many thousands of customer environments and documented the results.
The net result is that we've become *extremely precise* in quantifying specific customer benefits around their actual use cases. Not to point out the obvious, but that sort of intellectual capital just isn't available to smaller vendors, those that are bound to an antiquated architecture, or larger vendors that treat storage as an interesting side business.
But There's More To Do ...
Despite all the success we've had, we're quite aware that all of our efforts -- up to this point -- have been "in-the-box". Yes, we've made classic storage arrays faster, more efficient, more easy-to-use, etc.
And to take that next step, we have to redraw the "storage domain boundary" to include the server. Some outside observers have (mistakenly) thought that would be a culturally hard thing for EMC to do, but -- then again -- they don't work here, do they? :)
The motivation for server-side flash storage is clear and compelling -- it's a heck of a lot faster than even the very best storage-side flash.
Although the storage media technologies are somewhat similar, the real reason for the performance bump is that you're eliminating the latency (and occasionally the bandwidth constraints) of the "round trip" from server to storage and back again. While there's still a protocol handshake, it now happens at internal PCIe bus speeds vs. I/O channel speeds.
So, why doesn't everyone go out and buy server-side flash cards from their favorite PCIe vendor? Well, more than a few people have gone down that road, but have encountered not-unexpected problems.
First, we're usually talking valuable data here. That means it has to be protected, backed-up, replicated either locally or remotely, etc. The data might be moving faster, but requirements around data protection haven't changed.
Second, as we've learned from our past, the important secret sauce is to get the right data in the right place at the right time -- without it being a manual process. Smart software is needed to do this. We did that with FAST within the array; we now need to do the same between server and storage.
Third, like any storage, flash storage is expensive, and you'd like to use as little of it as possible by deduping, pooling, etc. as you would any other scarce resource.
Finally, when we "cross the chasm" into meaningful enterprise applications, it's not enough to put raw technology out there and encourage people to give it a go.
Enterprise customers expect their vendors to assess their environments, propose complete solutions, quantify the expected benefits, be able to certify a wide range of hardware and software environments, provide deep integration with the other parts of their environment, and -- above all -- be there in full force when there's a problem.
None of this should come as a surprise to anyone :)
The Stage Is Now Set
Thanks for your patience -- it's now time to dig into the specific details around VFCache -- what is it, what does it do, and how it's significantly different that what's already out there.
I'd invite you to join me on the next chapter of the story ...