I get asked all sorts of interesting questions, some of which sound easy to answer, but turn out to be a bit more difficult to do so once you dig into them.
As an example, I was asked to provide a quick, not-too-technical answer to the key differences between grid, scale-out, clustered and scale-up approaches to storage.
Sure, that's easy to do when it's one storage geek talking to another, but it's a bit harder to do if you're talking to someone with more of a general interest.
So, let me try my best, and please remember I'm dramatically oversimplifying here.
If you think that I've missed something, please help out in the comments section below :-)
Why Is This An Interesting Question?
Well, a lot of buzzwords get tossed around in our industry.
Some people use the terms precisely; others are looking for a bit of marketing buzz. The result is that it's very hard to carry on a useful conversation unless you invest the time to understand a few key differences to various approaches.
Second, it's all about use case, isn't it? Very often, there's no "best" or "superior" approach (much to the consternation of everyone), just pros and cons to keep in mind as you're having the discussion.
So the answers to these sorts of questions generally deserve a reply question: what exactly are you trying to do?
Lets Get Started, Shall We?
If you're building IT hardware, at some point, you reach the limit of individual components (disks, processors, memory, etc.) and you start wondering how you might lash things together so groups of smaller things act like bigger things: more speed, more capacity, more reliable, or all three.
Storage is no exception. When requirements start getting bigger than a single disk drive and a single port, you start thinking about how to connect multiple components together so it starts acting like a bigger thing.
To do this, the components have to communicate and coordinate using some shared mechanism. And the basic decisions around how this is done results in entirely different storage architectures.
By the way, this isn't just a storage discussion -- it applies to servers, operating systems, networks, databases, etc. -- I think it's one of those key architectural discussions that show up everywhere.
So, in a futile attempt to dramatically oversimplify, let's look at:
-- share as little as possible (grid designs)
-- share something (clustered designs)
-- share a lot (tightly coupled designs)
Grid Designs -- Share As Little As Possible?
You'll also here grid described as "shared nothing" architectures, although I believe the term "share little" is somewhat more accurate.
The idea is to break the work into as many small, independent pieces as possible, push them out to independent, presumably low-cost nodes, and aggregate the results. This assumes, of course, that what ever you're trying to do can be reasonably broken down into a large number of independent pieces -- a rather big assumption.
In the compute world, we all have probably seen variants of grid computing. I remember a time when it was cool to download desktop code to help with the search for extraterrestial life, or help decode proteins, or something similar. There was a cool layer of software that broke the problem into little bits, farmed them all out to zillions of desktops, and aggregated all the results.
In the storage world, you'd need the same thing -- a layer of software that broke things into useful independent chunks, farmed them all out, and aggregated the results.
If I looked into the EMC portfolio, two products (Atmos and Centera) might be considered grid-like. In both cases, their physical architectures are composed of independent nodes that don't really share all that much.
Each has a software layer that presents what looks like a big space, but is actually composed of smaller chunks that are aggregated. Interestingly enough, both have the property that that you really don't have to know exactly where your data might be to get it back (unlike file systems and SANs).
A key aspect that's usually overlooked in grid storage discussions is that you'll usually have multiple copies of your data. Each node can only access the storage behind it, so if it fails, you're toast. Almost all grid approaches scatter redundant data behind multiple nodes for this very reason.
I believe that grid storage architectures are well suited to massive amounts of content that doesn't change frequently. I don't think they're inherently well suited for OLTP at scale, high performance computing or stuff that's constantly changing.
Grid-style approaches can also be called "scale out", simply because the way that you get bigger (but not necessarily faster) is that you add more commodity-style storage nodes, rather than buying bigger and faster nodes.
Don't assume that grid-style approaches are always cheaper. Besides the redundant data discussion above, there's the sobering fact that sometimes many smaller boxen can be more expensive than a single, purpose-built one. But the scaling model is definitely sweet.
Clustered Design -- Share A Little More?
I guess the next stop on our tour would be clustered storage devices -- those that do a bit more sharing. We're still building things out of commodity-style nodes, but we've got some sort of interconnect that allows the nodes to coordinate work back and forth as needed.
In the server world, many of us are familar with failover clusters: if one server fails, another one picks up the work and carries on. We've also got load-balancing clusters -- if one server gets too busy, work can be moved around: VMware's DRS is an excellent example of this concept. Most all of them required shared storage to work.
Since most storage controllers are essentially purpose-built servers, the same is true in a storage cluster -- if one storage controller gets busy (or fails!) they can easily pass the work off to another node in the cluster.
Cluster architectures usually don't share memory, but often share physical storage. Every node can (usually) see all the storage in the pool -- which means you're not forced into multiple copies of data. Cluster nodes have to do a lot of communication between themselves about current state, data that's being updated, and the like so that they don't step on each other, and are always prepared to take on another node's work.
One great example of clustered storage you might be familiar with is the Ibrix file system (EMC partner) -- take a bunch of commodity servers, plug in shared storage (like CLARiiON) and you've got yourself one honkin' parallel file server with smokin' bandwidth. If I stayed entirely in EMC's portfolio, the bigger Celerras could be described as a flavor of clustering, especially if you're using MPFS with it.
Clustered approaches are also described as "scale out" as the way you get more performance (bandwidth) is by adding more nodes.
Because of their architecture, I think clustered storage approaches are great at high-bandwidth storage applications -- video streaming, high performance computing, signal capture, large-scale file serving and the like.
Conversely, I don't think they're the greatest fit for large amounts of relatively static content, nor exactly what you'd like in a demanding OLTP environment with high write content and varying workloads.
Tightly Coupled -- Share A Lot!
Perhaps the final step on our journey would be tightly coupled designs -- servers and storage controllers that share big memory pools, or act like they do.
If you're a server geek like me, you probably remember the era of SMP (symmetric multiprocessor) UNIX boxes like the Sequent and similar. Big shared memory pools and lots of processors that could see it all.
Storage arrays have the same concept -- think EMC DMX and HDS's USP -- these are the big boxes that can do many things, but are especially proficient at harder workloads like big OLTP and write-intensive environments that tend to vary rapidly.
Some people call these "scale up" (a misnomer IMHO) since they typically come packaged in very large frames, but -- at least from my perspective -- it's just another flavor of scale-out: you get more performance and capacity by adding more I/O and memory controllers.
Now, Nothing Is Simple, Especially In Storage
As you look across the landscape of different storage products from different vendors, they sometimes don't fall neatly into each category. For example, one could describe the XIV as essentially grid-like with a small bit of clustering, but IBM is trying to position it for serious workloads.
If one were to consider a mid-tier dual-controller array such as a CLARiiON, it might be described as halfway between a clustered and a tightly-coupled design -- it shares enough memory to enable fast failover, for example. NetApp filers (in HA configurations) are probably more clustered than tightly coupled -- and so on.
To reiterate an earlier point, it all depends on what you want to do. Mismatch architecture and workload, and one of two things will likely happen: you'll either have unsatisfactory performance, or you'll spend more money than you have to.
What Will The Future Bring?
Storage designers are always trying to build better products, and I believe the trend we'll see in the next few years is hybridization -- storage architectures that combine aspects of both.
For example, the very latest high-speed interconnect and server technologies have the distinct potential of blurring the lines between clustered storage and tightly coupled designs. Intelligent software and faster networks has the potential do the same between grid and clustered designs, I believe.
We'll see, won't we?
Where Does That Leave Us?
Maybe we've spent too much time explaining concepts that really don't matter that much.
Every time my wife gets that itch for a new car (thankfully that hasn't happened in a while), I've learned not talk about differences between turbochargers, superchargers or adaptive cylinder engagement schemes. Nor do I wax on about the different approaches to transmissions or drive trains.
Nope, I don't do any of that.
I simply ask "honey, what kind of car were you thinking about"?
:-)
Comments