Storage Virtualization and Invista 2.0

You might have seen EMC's announcement of Invista 2.0.  I thought this topic deserved a bit of context.

Storage virtualization has turned out to be one of those long-simmering topics in this industry.  And it's still going to simmer for a while, I'd predict.  This game isn't anywhere near from being over yet.

Best as I can recollect, although the industry was talking about at the end of the 1990s, the discussion got very hot at EMC in the 2002-2003 timeframe.

We knew that this was going to be an important storage technology, but we had some hard choices to make.

If anything, one of the things I like about working for EMC is that we study the problem, and are willing to make the hard choices -- and stick with them -- to get to where we want to be with our customers.

And with the recent announcement of Invista 2.0, I think it's a good time to reflect on the journey, where EMC and the industry has ended up, and maybe a bit of what happens from here.

A Few Basics

Storage virtualization is nothing more than a convenient abstraction of storage between the device that has it (usually a storage array), and the application that uses it.

Like other virtualization abstractions, it can potentially offer benefits around consolidation, moving things around, certain aspects of management, and so on.

But, I'd offer, there's no single *best* way to implement it.  "Best" defines on what you're looking for.  And, since there's such incredible diversity in customer requirements, it's a hard argument to win.

Implementation Options

From a pure technology perspective, there are generically four ways one could implement such an abstraction.

Server-based storage virtualization (aka volume managers) have been popular for over a decade.  VMware's recent announcement around Storage VMotion will probably renew interest in this approach.

Another approach is to add an appliance in the I/O path.  IBM and others have had some success with this approach for certain use cases.

There's a pure network approach, where the network device is taught how to support virtualization concepts (EMC's Invista and RainFinity fall into this category, for SAN and NAS respectively).

And, finally, an array can support many virtualization capabilities, either with storage located within the array (EMC's CLARiiON and DMX, among others), or with the array attaching to external storage (Hitachi's USP-V).

And, just to confuse matters, you'd make one sort of product choices around storage (or block) virtualization, and another set of choices around file virtualization.  Since many organizations are finding they have bigger challenges with their file-oriented storage, rather than their block-oriented storage, it's not always obvious where the best place to start is.

Lots of choices, aren't there?

The Case For (and Against) Storage Virtualization In The Server

Doing storage virtualization in the server is relatively cheap and easy to implement compared to other alternatives.

The good news is that -- depending on your choice of "volume manager", you can get a certain degree of storage consolidation, somewhat easier management, and -- with Storage VMotion -- the potential of non-disruptive movement of storage objects from one array to another.

Does these approaches scale well?  Perhaps not.  With most traditional volume management solutions, there's no single point of control -- most everything is done on a server-by-server basis.  And every server needs some piece of functionality to be loaded, configured and managed.

And if you're really into moving information around without disrupting service levels, let's face it, there's no free lunch between using server cycles for application processing, and using cycles for moving information from place to place. 

Think about the performance impact of a backup, as an example, and you'll get the idea.

The Case For (and Against) Storage Virtualization In An Appliance

Many small players here, and one big one -- IBM.  The idea is to create a server-class appliance that sits in the I/O stream, terminates I/Os, processes them, and sends them along to wherever they need to go.

IBM has poured a ton of R+D into this approach (perhaps the only storage technology they're heavily invested R+D) and made a go of it.  They have delivered decent levels of performance and availability for moderately difficult use-cases.

Is it better than server-resident virtualization?  At a minimum, you don't have to put software on every server you own, there's one place to go manage it (at moderate scale), and you have the benefit of the appliance's CPU cycles to do data movement, so service level impacts should be more moderated.

There's a thorny issue around caching write I/Os in an appliance, and maintaining integrity with the "real" image on the array.

But the only substantive criticism I'd offer would be the challenges of larger scale.  Have lots of servers and lots of storage?  You'll have lots of server virtualization appliances to go with it. 

The Case For (and Against) Using A Storage Array As A Storage Virtualization Device

Most external storage arrays do a fair amount of virtualization-like functionality for the devices they contain.  Hitachi took this idea one step further by allowing external storage to be attached behind an enterprise-class storage array, and extending the paradigm.

On one hand, this is an interesting approach.  Conceptually, you're extending array functionality to subordinate arrays.  Array technology (and its firmware) is relatively mature.  And if you're managing the array, you're also managing the storage behind it.  Performance and availability is decent as well.   After several years, HDS can now point to customers who've implemented it and have it working.

But there's another side to this.  First, an enterprise-class array can be an expensive device by any metric -- cost per port, footprint, cache etc.  And when you're talking real scale for storage virtualization (lots of arrays), pooling and moving across domains isn't easy -- it's not a flat, pooled space where everyone can see everything.

And, since the most common use case for this approach seems to be repurposing stranded storage assets that are on the books and have to be there for a while, I have difficulty comparing the utility of reusing three-year-old+ storage devices as compared to the newer stuff that's faster, cheaper, more reliable and more energy efficient as well.

It's hard to build IT strategies when finance imposes some funky rules -- but that's the world many of us live in.  And HDS has a potential answer for you if that's your situation.

What EMC Did

This isn't widely known, but one of the most free-wheeling (and extended) debates I've ever seen at EMC was to answer the question of "what do we do in this space?"

Lots of people looked at all the alternatives above, some very passionately.  There were proposals to build an extended volume management capability that ran on servers.  There was a proposal to build an SVC-like appliance.  There were even proposals to enhance the Symmetrix to do what Hitachi eventually ended up doing. 

Everyone had an intelligent and passionate opinion.  It went on for a very long time.

But finally, some clear thinking emerged.

First, we believed that storage virtualization had its strongest value proposition at scale. 

Yes, we could see the potential for customers of all sizes to potentially take advantage of what storage virtualization could do, but the more we talked to customers, the more we realized that the bigger you were, the more you needed this stuff.

Now, scale is not just capacity -- that's way too simple.  Scale is performance, number of devices and ports, sophistication of requirements, complexity, etc.  It's a multi-dimensional concept, at least for us.

And once we internalized the "scale matters" concept, one by one the alternatives became less and less attractive.

We began to see that the only candidate technology that could possibly scale economically was storage networking technology. 

At the time, Cisco / Brocade / McData (as well as a few others) were proposing building SAN devices that would support the hardware enablers needed for storage virtualization at scale.  Yes, there'd be a need for external control and coordination of these new ports, but -- at least through an engineer's eyes -- there was the "right" solution to the problem, as we saw it.

And, at the same time, we realized that this would be a long journey.  The technology foundations were new.  We wouldn't be first to market, as other approaches (use an appliance, repurpose an array) were easier to develop and get to market.

It was a hard choice -- either take the time to build something you believe in, or take the shortcut and get to market sooner.  And there was another extended debate around that topic as well.

But our decision to build an entirely new storage virtualization platform on intelligent storage networking technology was fundamentally the right one in the long term -- and, one thing you've got to understand about EMC is that we do think in longer timeframes than most vendors.

Where We Are Today

I'd offer that we're starting to see some market validation.

Sure, it's easy for competitors (and a few smarmy journalists) to poke fun at Invista being "invisible".

It's not. 

We've got enough production customers now who are genuinely happy with what we've delivered that we feel we're seeing a bit of vindication in our very difficult (and expensive) decision.

We've built them a next-gen network storage virtualization platform that they can run their business on.  Not some repurposed appliance, and not some feature on an existing storage array.

We know it will scale -- today, and in the future. 

We know that -- as it scales -- the economics will win out over other alternatives. 

We know that -- at scale -- it's easier to manage and support than other alternatives. 

We know that we can deliver integrated management solutions that make large storage environments easier, and not more complex.

We know that -- as protocols change to 8Gb, 10Gb, FCoE or iSCSI -- we've got them covered.

We've convinced ourselves -- and more than a few customers -- that we've made the right architectural decision, and we're starting to see it play out in more and more customer environments -- especially environments that are operating at scale.

Maybe we haven't wone the battle (for the time being!) in the popular press.  So be it. 

It's not always about being the most popular kid on campus.  Sometimes the geeks are right.

Does that mean you'll never see additional server-based storage virtualization capabilites from EMC? 

I wouldn't take that bet.  Nor would I bet against never seeing an appliance-oriented storage virtualization device, or even arrays that connect to other arrays. 

It's not hard to see that -- over time -- customers will have different use cases, want them in different packages, and so on. 

When you put our entire portfolio on the table for other storage functionality: arrays, tiering, replication, etc. -- you'll see a broad range of offerings with no one "right" solution that solves all problems.

But, at the same time, it's much easier to take lessons you've learned at large scale, and apply them to smaller environments.  As opposed to taking small-architected solutions and trying to make them scale. 

Congratulations to the Invista team -- it's been a long journey over the last few years.  My thanks for keeping with it, and delivering a storage virtualization capability to our customers that gives them a differentiated, sustainable competitive advantage.

So, guys, what are you thinking about for Invista 3.0?

;-)

Comments

[Dave]

Chuck,

Good piece. Well written as usual. Some points could use clarification.

It seems that the big 3 all face similar challenges today with respect to scale of these devices. Once a customer hits the PRACTICAL limits of the virtualization engine they have to install another one. How does the capacity behind each engine participate in a single virtual pool? Am I mistaken in implying Invista has this challenge along with SVC and USPV?

As well, it seems all three approaches face similar challenges when it comes time to upgrade/replace the virtualization engine. How do customers do that non-disruptively?

In my Wikibon interactions I've communicated with and personally interviewed dozens of SVC, USPV and even some Invista customers-- All seem to be effectively solving the problems they were asked to address. But for now, to be frank, the "Invista scales better" rap is not resonating.

I'll be the first to admit mine is a limited sample but EMC's own Invista 2.0 press release quotes a university customer...not typically the reference model for large scale, complex storage (with all due respect to Purdue University). Thanks. -dave from Wikibon.

[Chuck Hollis]

Hi Dave:

Thanks for your great questions.

First, when it comes to "scale", I usually have to remind myself to think in two dimensions -- capacity and performance.

With regards to capacity, I think all of the "big 3" will be working to increase the number of volumes that their products can address as part of a single domain.

I personally believe that EMC's approach has an architectural advantage here as the numbers get larger over time, but I don't think that's the important issue, nor do I think it is especially relevant.

I think for many customers a more important aspect of "scale" is throughput and - yes - additional latency added by the virtualization device.

Comparing normal I/Os routed through a SAN device vs. an appliance or a storage array isn't even close to being a fair contest.

Keep in mind, with Invista's approach, the vast majority of I/Os are kept on the SAN, and not touched by the virtualization device, unlike array and appliance approaches.

Think microseconds vs. milliseconds.

Sure, this won't be an issue for everyone, but as you know there's a certain segment of customer that looks at this sort of thing very carefully.

You probably are aware that Invista's architecture separates data path from control path. If a customer uses dual pathing from the host (most do), the upgrade of the data path (e.g. the SAN switch) is almost identical to any other SAN upgrade. Takes a bit of planning, but this is done non-disruptively on a routine basis thousands of times a year.

I think this is important as technology evolves from 4Gb to 8Gb to whatever lies ahead.

The other component that's subject to upgrading is the control processor. I am not close to the details regarding Invista 2.0, but I believe that it supports fail-over upgrade scenarios, allowing you to upgrade one element of control-processor pair at a time.

Since you've been in the industry a while, you're probably aware that the vast majority of larger customers are unwilling to be part of a vendor's press release -- especially the really interesting ones.

This has nothing to do with technology or vendor relations, and has more to do with legal and PR concerns by most large enterprises.

To use vendor-generated press releases as an indicator of a product's market or technical success wouldn't be among my suggested best-practices for analysts. I'd bet it'd be easy to be led to some erroneous conclusions that way.

Thanks for writing, Dave -- and best of luck with Wikibon.

[Dave]

Touché Chuck. Time to do some more homework for sure. Good luck with 2.0-- hearing lots of good things from customers and expectations are high.

[Barry Whyte]

Chuck, I must agree with Dave, a well written piece. I guess you are admitting that Invista is aimed much more at the enterprise level, where you are happy to help protect the investment (I think I called it a 'cash cow' in the post on my blog) of copy services in the controller.

While I will admit that a very large scale solution that simply cracks the packet at wire speed will mean you can use very little processing power to provide 'redirection' of packets - it most definitely lacks the power and ability to 'snoop' the data. i.e. cache it, manipulate it, copy it, thin provision it, replicate it, mirror it, de-dupe it, whatever you want to do with it.

I may have been a little harsh in my write up of the recent re-announce (wasn't this announced back in August?) but I really believe that unless you have the processing power available, that is now in abundance in an appliance (Quad cores, Octal cores not far off) that is yet unavailable in a line card (not to mention the ability to very quickly sync cluster wide updates) that the flow-throw design is fundamentally flawed in its potential.

Correct me if I'm wrong, but I believe the point in time implementation in Invista is actually just a clone process, and the destination is not available until the source has been fully copied? What about incremental point in time copies, cascades, space efficient?

For those that want to really role out heterogeneous storage virtualization that provides vendor neutral feature rich copy services, an appliance or controller based solution are the only currently viable options?

Thoughts?

[Chuck Hollis]

Hi BarryW -- good to hear from you.

I have to apologize regarding my tardiness in posting your comment and replying -- I've been completely unplugged from the grid, and -- let me tell you -- it's a refreshing experience.

That being said, I'm going to take the easy way out here and punt. There are others at EMC who could probably do a better job of replying and debating the points you raise.

Best wishes for a successful 2008!

[Dave Vellante]

Okay Chuck...in the spirit of doing some homework we went out and contacted Purdue directly (without any knowledge or assistance from EMC) to find out what was really going on there with Invista 2.0. While it's early in the production phase, I have to say we were very impressed that this example was actually representative of real commercial operations.

Still waiting for that banking reference but here's the case study writeup along with my apologies for dissing your reference account!!
http://wikibon.org/Case_Study:_Storage_virtualization_at_Purdue_University