Yesterday was Atmos and COS (cloud optimized storage) -- essentially storage at internet scale.
Today I'd like to delve back into the flash discussion -- storage at near-memory speed.
I think it's interesting because many of the industry people looking at this issue are travelling down the exact same discussion path that EMC did way back when, long before Jan 2008 when we announced EFDs (enterprise flash drives) for the DMX.
Like any journey, we asked ourselves many of the same questions that are just now coming up in the broader discussion. At that time, we arrived at a consensus view, and continued our journey.
It's now been almost a year since we've had these products available for customers. Not to point out the obvious, but (as of today), the only mainstream arrays that support enterprise flash drives are the DMX and CX series from EMC.
I'm guessing we'll be seeing the very first flash-based offerings from other vendors next year -- which will lead to the inevitable shifting of the discussion from "good idea or bad idea" to "which vendor is doing it better?"
That's how markets evolve.
Flash memory can easily play two roles in a server/storage architecture. You can think of it as "cheap DRAM" that augments the ability of a server or storage array to cache data, or "fast disk" where it directly replaces spinning media.
This one's got some healthy discussion going, because -- in all actuality -- flash can easily play both roles. Strangely enough, many people are looking for the "right" answer. I tend to think of it in terms of "pros and cons".
Flash As Cache
Turning to the "flash as cache" discussion for just a moment, the thinking is that modest DRAM amounts used for storage caching can be beefed up with ostensibly cost effective (albeit slower) NAND.
While this is true, it brings up a few observations:
- Flash is not as fast as DRAM, so the implication is that there will be a level of logic that decides which data lives in DRAM, and which lives in NAND. Those algorithms are important to get the most out of any caching scheme. Whether this function lives on the server, or lives on the array -- these algorithms are not off-the-shelf technology.
- Our experience with storage caching (e.g. a decade-plus of Symmetrix with ostensibly a very large cache) shows diminishing returns as you add cache for many workloads. Sure, things run a bit faster, but at some point spinning disks get involved, and you generally see declining benefits from paying for additional caching.
- If "cache" is to be written to, extra design costs can be involved -- remember discussions around nonvolatile cache? Read cache is simple, write cache takes a bit of careful thought that you don't lose changed data in any scenario.
Will we see NAND (flash) being used to build server-based storage caches, as well as find their way into storage array cache designs? Most certainly -- it's an interesting ingredient.
But -- from a storage geek's point of view -- cache is cache -- we all know what it can and can't do.
Flash As Storage
This one is far more interesting, because we see more than a few discussions where server vendors are trying to say that flash in a server is equivalent to flash as storage.
Yes and no.
Yes, in much the same way that a bare disk drive sitting in a server enclosure is storage in its simplest form.
However, implemented that way, it can't easily pooled or shared, needs some form of data protection (e.g. RAID), it's more difficult to make local and remote copies, it's really hard to tier and make part of an ILM scheme, it's difficult to manage as a separate entity, and so on.
That's why most enterprise storage sold today is external to the server, and not internal to the server.
I've written before that there's a bit more involved in doing this than just shoving an existing EFD into an existing array, especially for those vendors who have a "spindle randomizing" approach to their design.
Flash as cache will eventually become less interesting as part of the overall discussion -- there are no dramatic differences for most use cases in implementing storage cache with NAND, DRAM or some sort of mix.
Interesting, but not so much.
Flash as storage? Well, that's going to be really interesting, simply because the differences in what applications experience are so spectacularly dramatic. As we've seen time and time again, just moving a few hot spots to an EFD device can make an eye-popping difference in what users experience.
And that's *after* caching has done its job ...
Looking a bit farther out, as EFD prices tumble to points that are less breathtaking than today's (a process already well underway), there will be more and more of a case to put more and more data on these devices.
And, like the disk storage that came before it, you'll see most of going into external storage.
The Limits Of Big Disks?
There's also been some interesting parallel discussions recently about how disk capacities are ballooning, but I/O rates per device are holding relatively steady.
Throwing more SATA disks at the problem boosts I/O rates somewhat, but at considerable expense -- as many smaller vendors who built offerings on this particular premise are finding out.
Use 2, 3 or more SATA drives to handle a demanding workload, and costs mount. Unused storage. Cabinets to put them in. Power to keep them all spinning. Data center space to rack them all.
Not to mention the uncomfortable optics of people asking difficult questions about all that unused storage.
We've heard stories of customers asking particular vendors about this, and the glib answer comes back -- "disks are cheap". Well, they're not -- not if you really look at what you're doing.
And, even after all of that, there's a hard limit to how fast data can be moved to a SATA drive -- no matter how many of them you use -- a limit that EFDs comfortably blows past.
Now, to be fair, there are many information access patterns that are well-suited to big drives -- and there will always be a ready market for the next-largest drive.
I'm just expressing my view that -- before too long -- we'll see far fewer vendors who try to get adequate performance by throwing lots of cheap drives at the problem, and far more solutions built on the intelligent use of all available technologies -- including EFDs, if I'm not mistaken.
This Is Fun
Why? Because I happen to be lucky enough to work for a company that saw this coming years ago, and had the intestinal fortitude to make the big investments required to bring EFDs to market at least a good year ahead of anyone else.
This isn't about bragging rights (ok, so maybe it is, just a little bit), the real story here is that EMC now has a substantial lead on understanding how these devices work in real-world customer environments, and are using our learning to busily work on the next round of technologies and integration.
This isn't about SPC benchmarketing.
We'll see if EMC called this one right or wrong in a few years.
That's the fun part -- customers decide, not vendors.
Courteous comments welcome as always!