Symmetrix V-Max: Storage Architecture Redefined

Today is a very big day for EMC -- and for the broader storage industry. 

EMC has announced an entirely new approach to high-end storage architecture -- nothing more, and nothing less.

I think it's going to take a while for people to fully appreciate what's been done here, and what it means going forward.

Say what you will about EMC, but we're not afraid to make very big bets on very big ideas -- and that's what we've done here.

A Short Primer On High-End Storage

People unfamiliar with the storage marketplace may look at high-end storage arrays and ask "what's the big deal?", not fully appreciating the storage needs of larger enterprises.

These devices first became very popular in the mid-1990s as computing and storage demands began to explode in larger enterprises. 

Perhaps their most distinguishing feature is that they use multiple controllers and large caches to deliver excellent performance as well as the utmost in availability.

When I first came to EMC, my job was to convince Big Unix types that an intelligent cached disk array was a good thing.  Indeed, the whole idea of sharing storage betwen servers was relatively new at the time. 

I think EMC succeeded in that regard :-)

Many of the storage features most people take for granted -- snaps, remote replication, etc. -- first become popular on high-end arrays.

Big, high-end arrays still represent a sizable multi-billion dollar chunk of the market each and every year. 

Why?  I think there will always be IT environments that need the utmost in storage scale, performance, functionality, and so on.

The market has essentially narrowed to two stalwart vendors: EMC and Hitachi.  Most people would exclude IBM from this category as -- technically speaking -- none of their products fit this architectural definition.  And, besides, they've been out of the game for quite a while :-)

And, just to be complete, both Fujitsu and NEC offer products that could theoretically compete in this segment, but -- again -- we don't see much of them, either.

Thinking Outside The Box

For as long as I can remember, high-end storage arrays followed a basic architectural theme: multiple I/O controllers, big cache cards, and some sort of local interconnect -- a big bus (e.g. early Symmetrix 5000), or an internal switch (e.g. Hitachi USP) or perhaps a mesh fabric (e.g. Symmetrix DMX)

In many ways, the architecture defined many attributes about the array -- what was a reasonable entry point, how big it could scale, how fast it could go, how linear was the scaling, how redundant it was, how cost-effective, and so on.

Perhaps the most significant innovation in the V-Max is its unique architecture.  Its intelligence is assembled from rather modest building blocks, each composed of industry standard components, and each containing its own self-contained I/O and cache memory, much like you'd find on a high-end CLARiiON array controller.

What makes it special is that -- thanks to the new virtual matrix -- multiples of these building blocks appear and behave exactly like they were one giant array.  Underneath the covers, there's a shared memory architecture that unifies them into a single virtual "box".

Modular? Monolithic? Clustered? Tightly Coupled? Hybrid? 

Symmetrix V-Max defies neat categorizations, although I'm sure people will try to hang some sort of inevitably incorrect name off of it. 

The lines have been permanently blurred.

But Hasn't That Been Done Before?

Yes and no.  You might be familiar with other mid-tier storage designs that appear to do something similar, but there are some very key differences.

First, none of these smaller players are using a coherent-memory architecture.  Instead, they're doing simple message-passing to coordinate their activities and communicate state.  This is a potentially interesting approach for certain mid-tier environments, but not nearly fast enough or robust enough for the high-end.  I'm sure one of the other EMC bloggers did a post on the interconnect technology used in V-Max -- it's not garden-variety ethernet or Infiniband.

Second, none of these smaller players can get really, really big like the V-Max can.  The overall architecture can eventually grow into literally thousands of ports, thousands of processing cores, tens of thousands of disks, and many terabytes of cache memory -- all operating as a single, tightly-coupled array.

Simply put, there's nothing remotely like V-Max in the marketplace today, so -- if you're an IT architect -- it's probably worth your time to understand some of the key differences it brings to the table.

So, What's The Big Deal?

Actually, there are several "big deals", so let's go through them.

First, the most obvious difference is entry cost.  For roughly the price of a big CLARiiON, a customer can get into a roughly equivalent Symmetrix V-Max.  The "high-end storage is more expensive than mid-tier storage" canard is gone.

The difference between the two?  The V-Max can grow and grow from that modest starting point -- still capturing the economics of mid-tier arrays and linear cost scaling.

Second, the V-Max array isn't limited to a single cabinet with short wires.  Think in terms of multiple cabinets, separated by many meters or -- in the future -- longer distances using optical connections -- all behaving as a single, giant array.

Third, the scaling is utterly linear and modular.  As more building blocks are added, the array gets bigger and bigger, and there's no need to replace the frame, buy a bigger cabinet, etc. -- nothing gets thrown away.

Fourth, there's no assumption that all the building blocks are exactly the same.  Some may be big, some may be small, some may be old, some may be new. 

The entire concept of "upgrades" takes on an entirely different (and attractive) characteristic.

And that's just for starters ...

Rethinking Storage Virtualization

Any storage functionality can appear in one of three places: at the server layer, in the storage network, or in the array itself.

Storage virtualization is no exception.  If you want multiple arrays to appear like a single big one for purposes of pooling and/or migration, you have similar choices.

You could use either clustered file systems or volume managers to achieve the desired effect at the server level.  Many people do this.

You could put something in the data path -- an appliance (SVC), an intelligent switch (Invista) or another storage array (USP) to achieve the same effect.

But -- up to now -- there were few good examples of storage virtualization being provided within the array itself -- multiple arrays "peering" to create a single, enormous logical entity.

In one sense, V-Max introduces yet another distinct and unique flavor of storage virtualization into the industry discussion that can now be compared and contrasted with other approaches.

No, V-Max can't be used to pool a bunch of legacy heterogenous storage -- and there certainly are a lot of people stuck with all sorts of junk on their data center floors -- but it can be used to solve many of the management and migration challenges that show up in large-scale environments, and do so in a uniquely elegant and seamless manner.

And I should point out that using this internal approach, it's a whole lot cheaper and easier than trying to do it externally -- homogenous storage virtualization can now be thought of as an integral feature of the architecture, rather than an add-on.

I'm sure we're going to be having some interesting discussions on this aspect of the V-Max :-)

FAST Permanently Redefines Storage Economics

As if there wasn't enough cool stuff to talk about, we've got to spend some time on the forthcoming FAST capability -- fully automated storage tiering.

Simply put, it's one of those things that permanently changes the game in storage economics once it's fully understood.  And changes it in a very big and meaningful way.

By intelligently and dynamically mixing enterprise flash, FC and low-cost SATA into virtual drives, enterprise storage has all at once become much faster and much cheaper -- and with almost no effort on the part of the storage administrator.

The beauty is that its value isn't limited to special situations like dedupe and spin-down -- this approach works across the board on almost any use case.

I'll leave it to others (such as The Storage Anarchist) to do the in-depth explanation of how FAST works (definitely worth the investment in your time), but its impact is clear -- the market will quickly become segmented into arrays that have this feature, and those that don't.

Everything Changes, And Nothing Changes

Sure, you can look at Symmetrix V-Max, and appreciate that it's a radical departure from what's come before it.

I think at this point it's fair to remind people that the V-Max is still a Symmetrix: it runs the full Enginuity code stack, and supports all the myriad and cool features that have made EMC #1 in this market.

I think it's also important to point out that the current DMX-4 architecture isn't going away -- it's still the market leader in this category, and is doing quite well, thank you.  Not everyone needs a V-Max.

The Symmetrix V-Max is in addition to -- and not a replacement for -- the current DMX-4 line.

And The Inevitable Private Cloud Discussion

If you follow this blog, you might appreciate how the new V-Max fits into the broader private cloud concept.

Simply put, data centers need to look more like clouds, and clouds need to look more like data centers -- hence the concept of a private cloud.

Paul Maritz talks about the ability of virtualization to create a single, giant computer that can take hundreds (or eventually thousands) of ordinary server components and make them look and behave as one.

Cisco's UCS takes that same thinking and applies it to large-scale blade farms with unified connectivity and management.

And -- in one sense -- EMC takes that same thinking and applies it to high-end storage arrays: multiple, independent storage controllers that behave and operate as a single, giant storage array.

Operating system.  Server.  Storage.

I love it when a plan comes together ...

See more at www.overtakethefuture.com

Comments

[nate]

What can the DMX-4 do that the V-MAX cannot? Or another way why might someone today buy a DMX-4 instead of a V-Max?

Are you just giving the V-MAX some burn in time so people become more confident in the system before the older DMX(and Clariion?) architecture goes away entirely?

Does the V-MAX use Intel CPUs entirely for data movement or are there a bunch of ASICs in there too(outside of the ASICs on the HBAs).

Certainly sounds like an impressive system, though probably way outside my price range, just finished a storage migration project, and what we have today will suit us for quite a while.

[Chuck Hollis]

Hi Nate

Technically speaking, there's no feature or capability degradation from DMX-4 to V-Max.

Right now, the DMX-4 is at the top of its game and compares well to offerings from competitors. No real reason to shoot it anytime soon if customers still want it.

CLARiiON is *not* going away -- different design point, different set of starting assumptions.

To the best of my knowledge, there are custom ASICs or FPGAs on the V-Max only for the implementation of the Virtual Matrix.

Otherwise, industry-standard hardware throughout.

Thanks for your kind thoughts ...!

-- Chuck

[Alexei Sapozhkov]

Why didn't you mention IBM XIV as an Hi-End storage? I think the Grid technology that has been implemented into XIV is much more scalable than EMC V-Max that gives just 8-node cluster like IBM SVC.

[Chuck Hollis]

Hi Alexi

External to IBM, the XIV is seen pretty much as a joke in the storage industry these days. Forget high-end, people are wondering if it's basically competent as a mid-tier array.

I could go down a long list of architectural features present in modern high-end storage that are missing in XIV's architecture.

But that wouldn't be any fun.

-- Chuck

[Alexei Sapozhkov]

Hi Chuck, I'm just curious of this long list of features that differentiates hi-end from all other stuff. Is it top secret of EMC or you can share it with me?

[Kirill Ponomarev]

Chuck, my personal view of what EMC is now doing is takingover HDS strategy, doing "bigger and faster" without providing some kind of innovations and new views on storage perspective.

I wouldn't put XIV and V-Max on the same line, because it seems like comparing bananas and apples.

But for me personally - V-Max is interesting solution, but it is not the product which would explode the storage markt with some interesting innovations.

[Denis]

Yes, Chuck!

Please, go down with "long list of architectural" ""features"", but remember that software outside features and options cannot be the "architectural" ones.

You know, progressive storage community is very interested in such statements like "there are only EMC and HDS hi-ends exist" and very confused with no definite replies and explanations.

BR, Denis

[Chuck Hollis]

Sure, every storage vendor would love to claim that their product is "high end" or "enterprise class" or similar.

But claiming it and being it are two different things, no?

So, in general, most storage architects would agree that high-end storage has some common attributes:

- multi-controller architecture (not dual controller -- sorry, 8000)
- large, global shared caches (sorry, XIV)
- powerful local and remote replication (sorry, XIV)
- ability to manage multiple QoS (sorry XIV and to a certain extent 8000)
- 24x7xForever uptime (sorry, XIV)
- ability to incur multiple component failures and still run (sorry, again)

I would suggest that those of you who work for IBM look outside your company and get a good sense in how the storage market is working these days.

Best regards --

[Alexei Sapozhkov]

Chuck, thank you very much.

That was interesting discussion but I don't agree with your point about shared cache or whatever shared resource because in this case the firmware is getting more complex and becomes the real hot spot and eats up all system resources. Also your comments about XIV for the 24x7 forever uptime and resistance to the component failures are incorrect because XIV has N+1 architecture and resilient to the failure of any unit in the system.

And finally, the real life when looking outside depends on sales and technical guys.

Best regards, I wish you gook luck.

[Nikhil Bagalkotkar]

--------- you may choose not to post this on the site however I would be interested in your response --------

Hi Chuck,
I was going through your post on the Enterprise class storage. And found a few inconsistencies

- multi-controller architecture (not dual controller -- sorry, 8000).
Does this really matter when the controllers under consideration are Enterprise class servers configured in a HA cluster supporting concurrent firmware updates. More over I believe these servers perform primarily cache management while the RAID calculations is done via the IO modules itself giving much more scalability and room to increase performance.

- large, global shared caches (sorry, XIV)
By this logic even the Vmax doesnot fit into the Enterprise storage category. Having a Large Cache might simplify the management algorithm however it does limit the cache that can be inserted into a system.

- powerful local and remote replication (sorry, XIV)
Updated XIV does offer Sync and Async modes of mirroring :)

- 24x7xForever uptime (sorry, XIV)
XIV's data distribution algorithm (unlike conventional RAID) ensures the vulnerability window for the system upon drive failure or expansion failure is extremely low (less than 30 minutes incase of drive failures) and the effect of a drive failure on the entire systems performance is negligible. While on the other hand the conventional RAID systems show substantial degradation of performance on drive failures

- ability to incur multiple component failures and still run (sorry, again)
XIV's data distribution algorithm (unlike conventional RAID) ensures the vulnerability window for the system upon drive failure or expansion failure is extremely low (less than 30 minutes incase of drive failures) and the effect of failure on the entire systems performance is negligible. While on the other hand the conventional RAID systems show substantial degradation of performance.

However some components of XIV which redefine enterprise class are
1. No additional licenses to be purchased for any functionality( point in time copy or replication or multipathing or ...)
2. Automatic data migration from older systems to XIV
3. An energy footprint which is at least half that of any other enterprise class storage of similar capacity and performance characteristics
4. Point in time copies that use redirect on write technology, thus ensuring that no OLTP freeze is required.

(at the end of the day TCO sells more than any other tech spec)

to explain it further some one has put an interesting video on youtube
http://www.youtube.com/watch?v=xtGdffB4iBc

[Chuck Hollis]

Nikhil, while I appreciate your sincerity, I really don't want to have a pointless debate with an employee of XIV/IBM.

Regardless of any architectural discussion, the XIV team has so far failed to deliver a credible offering in the marketplace.

Your stuff doesn't do what you say it can do. That's a big problem, in my book.

Show the industry that you can execute, and then we'll get into the discussion. Otherwise, it's a complete waste of time.

-- Chuck