vSphere as an I/O Engine

As more and more people consider bigger and bigger workloads running in virtual containers, the discussion inevitably turns to I/O throughput and latency.

Now, VMware is no slouch at driving big I/O workloads.  I shared one proof-point discussion from testing done last year in this regard on a previous post.

But today, VMware took a big leap forward in this regard with EMC's announcement of PowerPath/VE.

So, why should you care?

A Short Primer on MPIO

The acronym stands for "multi path I/O", and the concept has been around for many years.

From an over-simplified perspective, imagine a server making I/O requests to a storage device.  If I only use one  I/O path, every I/O request has to take their turn.  Use 2, 3 or 4 I/O paths, and the situation gets better -- if, that is, you've got a piece of software that can schedule the right I/O for the right path at the right time.

As with most foundational ideas, it appeared first on mainframes (doesn't everything?) and then made its way over time to other environments.

Most operating systems (VMWare, Windows, AIX, HP-UX, Solaris, etc.) have some base MPIO functionality that's more oriented towards availability rather than performance, e.g. if a path fails, reschedule the I/O for a surviving path.  A few may have a simplistic round-robin scheduling algorithms in an attempt to load balance, but these generally aren't too effective.

Enter PowerPath

EMC has been shipping PowerPath (advanced MPIO software) for many, many years.  There are literally hundreds of thousands of licensed copies out there.  PowerPath supports virtually every operating system, and arrays other than just EMC's. 

If there ever was a de-facto standard for MPIO in the storage marketplace, it'd be PowerPath.

Lots of nice features with PowerPath, and they work pretty much the same way everywhere you use it -- the ability to automatically restore failed paths, the ability to (optionally) support things like in-line encryption, and data migrations -- very handy stuff to have in your utility belt.

But -- far and away -- the best part of PowerPath is that is makes I/O bound applications run really fast.

PowerPath/VE == No Waiting

I/O profiles can be wild and unpredictable beasts.  It's hard enough to nail down an I/O profile for a given application, but start consolidating multiple applications on a single server (think VMware), or -- better yet -- start moving workloads between servers using DRS, and -- well -- attempting to take a static, pre-determined approach to I/O optimization is almost futile.

That's where PowerPath shines -- it has a wide variety of optimization algorithms, and adjusts dynamically based on what's happening Right Now.  All I/O paths are perfectly balanced at all times.  Keep in mind, this isn't about bandwidth, it's mostly about response time -- which is what users see.

Better yet, if you determine that there's a bottleneck between server and storage, simply configure another path.  PowerPath figures out the rest -- it's just about that easy.

So, What's The Impact?

YMMV (your mileage may vary), but a good rule of thumb is that we can see roughly 2x I/O performance in a serious, I/O-bound workload that's using 3-4 paths.  Now, not everyone has serious, I/O-bound workloads that require this sort of aggregate plumbing, especially in VMware environments.

But, consider that vSphere is now capable of sporting a virtual machine that has 8 virtual CPUs, 256GB of RAM and can drive 200k+ IOPs.  I think that would qualify as a serious, I/O-bound workload?

VMware Grows Up

Sure, there's plenty of VMware usage that's comfortable with iSCSI, or NAS, or whatever -- no problem there.

But, if you've signed up to the "virtualize everything" mantra (we at EMC most definitely have!), you're going to want the ability to drive very serious workloads very efficiently.

And we're more than pleased to offer PowerPath/VE to help support people who really want to push the technology.

My Personal Wishlist?

For those of you familiar with how PowerPath works, you're aware that it uses a loadable module architecture that allows all sorts of nifty extensions to be dropped into the I/O path, usually non-disruptively.

An important one of those extensions is encryption -- using RSA engines and key management, you can encrypt storage on a per-server basis, which turns out to be incredibly useful in a wide variety of situations.

Now, how cool would it be if you could selectively encrypt virtual machines -- and their information stores -- as they move around and share resources with other workloads?

Here's hoping that we can do this sooner than later :-)

 

Comments

[Stephen Foskett]

Chuck,

Great intro to MPIO! I'm really pleased to see more use of this tech across platforms!

Some questions:
1) Does PowerPath/VE require an Enterprise Plus license for vSphere 4? It looks like it does from the outside...
2) Does vSphere 4 include basic multipathing in the Standard edition?
3) What's the compelling value of moving up to PowerPath/VE from native MPIO?

Thanks!
Stephen

[Chuck Hollis]

Hi Stephen!

1) I don't know -- Barry Burke is chasing that one down, I think -- on your behalf. It might, which would not be ideal.

2) I don't know. Chad would know the answer to that -- I'll reach out to him. (See, marketing guys can be really useful!)

3) In terms of value props, I'd offer (1) dramatically improved performance for I/O request-heavy workloads, (2) the option of using fewer channels/paths to achieve suitable performance, (3) less "burp" when paths fail, smoother recover when they're restored, and (4) probably a few other things that I've forgotten.

Simply put, PowerPath has become the defacto MPIO standard in demanding Windows/Linux/AIX/HP-UX/Solaris environments. If you're considering heavy workloads in vSphere, you should consider PowerPath for many of the same reasons people considered it for other OS environments.

-- Chuck

[nate]

Certainly something I will look into once we decide to go to vSphere, assuming of course it will work on the version we end up with(likely essentials or standard)

Our I/O needs are not heavy(in VMs) I'm more interested in fast fail over.

thanks for the info!

[John F.]

Hi Chuck,

Powerpath/VE on the Windows platform leverages Microsoft MPIO. Nothing wrong with that. HP and NetApp and a cast of thousands do something similar. Aside from the garden variety MPIO DDK modules, there's the vendor specific logic in the custom Device Spoecific Module or DSM that differentiates one MS MPIO implementation from the next. I think digging in a little deeper into EMC's PowerPath/VE on WIndows DSM implementation could be enlightening, or at least educational. Do you have a technical paper or link to one with some meat on it that you could share?

Enquiring minds want to know.

Thanks in advance,

John

[Chuck Hollis]

Hi John

I'd have to flip you over to someone else to track this down. You're right, I do remember the "vendor specific logic" discussion from Windows MPIO -- I don't remember how it all ended up.

[John F.]

THat would be great if you could do that Chuck. The DSM is definitely where the rubber meets the road.

John

[Stephen Foskett]

John,

Yes, PowerPath on Windows is a DDK for Microsoft's native MPIO. This is a Very Good Thing, and is exactly why Microsoft was brilliant to build an extensible path management platform into Windows!

And in fact VMware PSM is really very similar to MS MPIO, and is therefore also a Very Good Thing.

I don't want to have EMC ripping up the Windows or VMware I/O chain just to insert their path management logic. Why doesn't every OS offer something similar?

This is not to say that PowerPath's value is diminished. EMC basically does everything they should do (choosing paths, balancing IO) and nothing they shouldn't (mucking with the kernel).

Stephen

[Chuck Hollis]

Thanks, Stephen -- I owe you one!!

-- Chuck

[John F.]

Hi Stephen,

You, Chuck (well for obvious reasons I can't speak for Chuck, but hopefully he'll roll with this one), and I all would clearly like to see similar functionality across the OS continuum. Standards like MS MPIO that leave room for OEMs to implement their specific value added features in a non-disruptive way are clearly a good thing. NetApp does their own ONTAP DSM, EMC has thier own DSM for PowerPath, Veritas their DSM, HP, and all the other vendors for that matter. That's how we differentiate ourselves from each other. I'm very interested to know what those features are for EMC that would differentiate them from say NetApp; that's all.

Thanks

John

[Chad Sakac]

The vSphere PSA (Pluggable Storage Architecture) has three module types - SATP/PSP (which work with the native NMP module). An MPP (multipathing plugin) does the function of the NMP/SATP/PSP (which is roughly analagous to the DSM model of the Windows MPIO framework). Both the NMP and MPP use the MPIO framework using in vSphere.

The NMP/SATP/PSP model in vSphere is a big step up vs. VI3.5 for everyone - heck, basic round robin is now an option (not the default for any SATP - ergo any array, but can be manually configured)

VMware has made the API model open and available to all - but only EMC (so far) has done the work to build an MPP.

So - how this helps customers:

- Automated path discovery and setup – without needing rescans. For any large environment this is huge.
- Killing flaky paths proactively.
- Automated reconfiguration of paths.
- More comprehensive load-balancing (weighted, and variable depending on host conditions). This is particularly important for ALUA configurations (which most Mid-range arrays, including NetApp John F, use) - where the first "A" in ALUA stands for "asymmetric" - there is a distinctly GOOD and distinctly LESS GOOD path (due to transversing lower bandwidth internal busses). Simple round robin there can actually hurt - as the inequal path get used equally.
- Predictive load balancing (when used with EMC arrays) – this is HUGE.

Why? Remember – on an ESX cluster – when a path on one host gets busy – what’s the likely root cause? The array port being busy. Which ESX hosts in the cluster will this affect? ALL OF THEM.

First of all – round robin won’t do anything to use other paths more heavily, so will do nada (the busy paths get used the same as the less busy ones). PowerPath/VE (with any array) will do a better job of balancing. But PowerPath/VE with an EMC array will predictively move workloads based on the array ports starting to get busy.

long and short, NMP (Native Multipathing) helps when using Round Robin, but pales in comparison (both in terms of automation and really squeezing the most out of the connectivity).

The goal is to have PP/VE support be heterogenous (same as PP is today). The NetApp team has reached out to the EMC team to discuss this (in spite of all blog back n' forth - the customer comes first.

[John F.]

Thank you Chad for your informative post.

John F.

[John F.]

Looks like Stephen Foskett took the time to expound on the details as well http://blog.fosketts.net/2009/04/22/emc-powerpath-vmware-hyperv/

Thanks Stephen.

John

[Jumper]

Some students pass the duty to professional writers because they miss the ability to compose a satisfactory paper about vSphere in that the reason why you need to use plagiarism detect, but such people like author don't do that. Thanks a lot for the topic