Three years ago, I wrote a popular post entitled "Storage Caching 101". For some reason, it's held up well as one of the top ten posts I've ever written.
But a lot can change in 3+ years, so I thought it time to update and enhance that post, especially as we now have a variety of newer server-side caching options to consider.
Things are moving fast here, so it takes some effort to keep up-to-date. I've done the best I can with what I know.
And, as before, please feel free to add to the comments if you think I've missed something ...
Memory Is Faster Than Disk
In a nutshell, it's all about speed.
The vast majority of the world's data sits on spinning disks. Access that same data from a memory technology (flash, DRAM, etc.) instead of rotating rust and you win from a performance perspective.
CPUs are getting ever-faster; spinning disk not so much -- so it's an increasingly important topic.
But any memory technology is significantly more expensive than disk technology, so the devil is in the details in how to use the smallest amount of memory to achieve the greatest results.
Using a memory technology for storage caching is different than using it for persistent storage, although that's becoming popular as well. All-flash arrays minimize the need for caching by storing *all* the relevant data in memory devices vs. just a working subset. But going all-flash will remain a comparatively expensive option for many use cases.
In this discussion, we also have to consider hybrid arrays that use a mix of persistent SSDs and traditional disks for storage. The idea here is simple: by intelligently placing the right data set on the right device, performance can be significantly boosted at more modest cost as compared to the all-flash alternative. However, once again, the devil is in the details.
Here we'll be examining different storage caching approaches, and won't be talking much about either all-flash arrays, or hybrid arrays.
Making matters more interesting is that the approaches are often combined (especially storage caching with hybrid arrays) to achieve the right mix of performance and price.
Let's Cover The Basics?
When it comes to IO profiles, there are four primary dimensions that are relevant: reads (both random and sequential) and writes (both random and sequential). Most use cases are a varying combination of these four, with very few pure profiles existing in the real world. For example, a database might exhibit a random profile during normal use, except when a full backup is done (sequential read) or a restore is done in a hurry (sequential write).
Technically speaking, storage caching can occur in four different places: on the disk drive itself, in the storage array, in the storage network or inside the server.
Disk drives have small embedded caches that optimize performance; we won't be talking about them here as their operation is largely transparent. Read caching in the storage network (especially in larger distributed file environments) is available, but hasn't exactly proved to be widely popular.
Caching in storage arrays has been around for quite awhile: it's well understood. Server-side caching is undergoing a renaissance recently thanks to PCIe flash cards and SSDs, but there's nothing architecturally new here either -- large servers have done storage caching for decades.
Finally, we have to divide the world into nonvolatile and volatile caches.
Nonvolatile caches can be used safely for persistent writes with data you care about. They incorporate protection against a component failure (usually mirroring) and power failures if DRAM or similar is being used. Volatile caches can be safely used for reads, or writes of transient data that's easily recreatable (e.g. swap devices or similar).
Specifically: you don't use volatile storage caches to write data you hope to read back in the future.
Cache Effectiveness With Different Profiles
Read caches can be surprisingly effective on random reads, especially when the IO patterns exhibit locality of reference, or LoR for short. Simply put, LoR means that there's a decent chance you'll re-read a block of data before it gets evicted to store something else.
If this happens, you win from a performance perspective. Historically speaking, a vast number of real-world applications exhibit very good locality of reference: databases, email systems, VDI, software development, etc. so some form of storage read caching makes sense.
The larger (and hence more expensive) your read cache, the better chance you stand of seeing locality of reference benefits. Pure sequential read streams (e.g. HDFS) don't see much benefit from read caches unless they're large enough to hold the entire data set that's of interest to multiple readers -- which can be very expensive proposition.
Writes to a nonvolatile cache are a different matter entirely. Most writes tend to be random in nature, and even a small nonvolatile write cache can soak up a lot of IO and transform it into a relatively efficient sequential de-stage to disk. Of course, if writes arrive in cache faster than the disks can destage, the cache fills up, and things degrade pretty quickly from that point -- until the writes subside.
Array Caches Vs. Server Caches
Any storage cache sitting on a storage array is inherently a shared, pooled resource: it serves all attached hosts. Almost all storage caches on arrays are designed to be non-volatile, which means they can be used for both reads and writes and can safely recover from all sorts of perils.
There is an awful lot of arcane secret sauce that goes into the algorithms that manage these caches. The efficacy of these algorithms can be relevant: two arrays from different vendors with ostensibly similar hardware resources may perform very differently, largely due to the algorithms being used.
The disadvantage of array caches is read performance -- at least, compared to server-based storage caches. To access an array-based cache requires calling a host bus adaptor, traversing a storage network, communicating with the controller at the other end, and waiting for the return reply.
All of this happens at server bus speeds when using server-side caches. And this can frequently result in some eye-popping benchmark results when compared to array-based read caches.
So does that mean that server-side caches are always better? No, it doesn't.
For starters, server-side cache can be difficult to manage as a shared resource: either for multiple applications running on the same host as in a virtualized environment, or across clusters of servers. There's a lot of hand-configuring going on today, with a few exceptions I'll discuss later.
More importantly, using server-side caching to write to an external storage array can lead to a particularly ugly scenario of what's called pinned writes. It's worth examining this in closer detail.
Let's say you had a pair of servers, each with a flash card used as a storage cache, and to protect yourself against component failure, you mirrored the writes between the servers, incurring a small performance penalty in the process.
If one of the servers should fail, you'd be theoretically covered. A copy of your write would be safely stored in the mirrored server component. So far, so good.
But let's say the storage network should fail, or perhaps the array -- and that this happened before the flash card could flush out its write to the array.
You'd then have a pinned write situation: the writes have been committed as safely stored back to the application and/or operating system, but they haven't made it out to your storage array.
In this situation, you'd have to first restore service to either the network or array, and then manually destage each block of written data to the exact correct location using your byte tweezers. The potential for data corruption in this scenario is close to 100%. Sure, it might not happen all that often, but when it did, you'd never want to have it happen again.
Put differently, when server-side flash is used for caching persistent writes to external storage, it becomes part of the array itself: the entire subsystem responsible for safely storing written data. Or if you prefer: server-side cache used with an external array and network can be considered volatile in the face of network and/or array failures.
And that's before we get into the issues in using array data services like snaps and remote replication ...
Combining Server-Side Caching with Server-Side Persistent Storage
You knew there was a VMware angle coming here, didn't you?
I don't want to disappoint you, so here it is: VMware's VSAN does efficient server-side caching of both reads and writes using a pooled resource model as well as avoiding the potential for pinned writes as is the case with external storage.
I think it's quite unique in this regard.
The basic VSAN architecture uses multiple commodity servers, each with at least one flash device used for read/write caching, with multiple disks behind each flash device. All cache and disk resources are managed as a pool by VSAN, with nothing more than a low-latency network hop away.
Writes are written to a minimum of two nodes before being confirmed back to the application. Should a server or component fail, its cluster partner(s) preserves the data. Unlike using server caches with external storage, there's no potential storage network or external array failures to consider, so no pinned data scenarios.
All the benefits of familiar array-style caching, with the performance and economics of server-side caching. Slick.
The Bottom Line
Flash is all about performance, and nothing performs faster than flash or other memory technology sitting inside a server.
Flash sitting inside a server is great for read, but it's difficult to share effectively and shouldn't really be used for persistent writes with an external array unless you know exactly what you're doing. Historically, arrays have done better at resource sharing and handling persistent writes.
But perhaps this has started to change with products like VSAN.
Who knows what else we'll see in the future?
Like this post? Why not subscribe via email?