Creating A Pool Of Paths

So much of what we're doing around large-scale virtualization, private cloud, etc. revolves around the pooling of infrastructure so it can be used in a dynamic and intelligent fashion.

VMware intelligently pools server resources.  Storage products like VMAX and VPLEX and FAST illustrate intelligent storage pooling concepts.

But how do we create a "pool of paths" between servers and storage?  Especially when we're talking demanding block protocols?

A Bit Of History

People familiar with EMC's portfolio will recognize the obvious setup to a discussion around EMC PowerPath, and -- especially -- the newer PowerPath/VE (virtual edition).

Over a decade ago, my phone rang, and there was this guy (Ric Cavillo of Conley Coporation) who had written this software that basically emulated mainframe path management for an open systems environment.

If you're not familiar with I/O path management on a mainframe, I'll forgive you, but -- in a nutshell -- it's very smart stuff.  It can continually sense available paths and utilization between server and storage, and optimize accordingly.

At the time, people were starting to put some seriously hairy workloads on big UNIX boxes using Symmetrix.  Think multiple SCSI channels going from server to array.  Paths would get congested, or fail, and the available options were quite primitive at the time.

Would EMC be interested in putting mainframe-class I/O path management on open systems?  

I said "very interesting" and hooked him up with our Symmetrix software engineering team.  This resulted in us acquiring Conley in 1998, and producing PowerPath.

By any standard, it's been one of those smashing success stories in the storage industry, with over 30,000 named customers using the product, and currently installed on well over 1 million servers.  It has become so ubiquitous that many of us had stopped really paying close attention, and were focusing on other things.

Things didn't really change much until PowerPath/VE was introduced, and -- voila! -- the product is now experiencing surprisingly dramatic growth.

So, What's This Path Management Thing?

Perhaps the simplest analogy is checkout lanes at a supermarket.  Ideally, as many people would get through as quickly as possible.

But it's not that easy.  Some people have a few, quick items -- others have multiple shopping baskets.  Different checkout clerks operate at different speeds.  The register might malfunction (or the clerk goes on break!) meaning that the work has to be redistributed.

Not only that, your customers expect different levels of service, e.g. express or priority lanes.  Without someone or something intelligently directing traffic through the available lanes, it's usually a suboptimal experience.

Now, substitute "application I/O" for "customers", "storage devices" for "checkout clerks" and "storage paths" for the queues behind each.  

In this example, PowerPath would be the "intelligent director" (per host) making things flow smoothly.

Most alternative MPIO offerings (multi-path I/O) offerings can detect a failed path, but struggle to use a restored one.  Or they use a simplistic round-robin approach that does not take into account the nature of the I/O, existing path contention, priority classes, etc.  

Good -- but there's room for far better.  And that's where PowerPath comes in.

What Does PowerPath Really Do?

It's quite a long list.  Our own Doc D'Errico owns many of the fundamental patents around portions of PowerPath, but -- in a nutshell:

• adaptive load balancing (for Symmetrix/CLARiiON): EMC proprietary load balancing algorithm which adapts to the number of  live paths and the load on the paths.
• proactive death: if a device becomes inaccessible along a specific path, proactively mark the path as dead for all other devices that are accessible via this path. 
• proactive restore: when one path on a bus (array port) is restored the other paths on the bus are tested (and possibly restored)
• periodic auto-restore: periodically test dead paths to see if they are now live.
• reactive auto-recovery: when no live paths are found while routing an io, all paths are tested (and possibly restored) one more time before failing the io
• live path testing: periodically test all the paths to see which ones are live/dead.
• bus testing (array port testing): periodic testing of the array ports.
• intelligent trespassing for CLARiiONs: no unnecessary trespasses

If you want the full boat PowerPath presentation (also covers encryption options, migration enabler, etc.) I'd invite you to check out this link.

Virtualization At Scale Makes The Case

Now, consider what's happening when most people virtualize their workloads.  You're now driving much more consolidated I/O from a single server and its I/O paths.  To make matters more interesting, VMware is doing its own form of load-balancing based primarily on CPU utilization, so workloads are frequently moving around as well.

Someone once described PowerPath as "DRS for I/O", and that's not too far off -- intelligent load balancing across available resources.  The more consolidated, the more value it brings.

Rather than have me attempt to tell you how wonderful it is, I've copied and pasted some of the user comments from ECN (EMC Community Network) around PowerPath.

Their words, not mine (light editing for typos, etc.)
-------

User: janetmason13 

Post 1: 4 posts since Jan 21, 2010 10:34 AM

PowerPath/VE performance gain 

Adding to my other discussion where I described my testing/configuration between VMware native multipathing and PowerPath/VE, I completed more comprehensive testing between the two products and wanted to highlight it here (and see if others are seeing similar resuls). 

I performed this latest round of testing in another one of our data centers with both EMC (DMX and CLARIION) and HP storage.  We have a range of between 2 and 30 to 35 VM's per hosts and run them between 40 and 90% (average is ~ 70%) all the time.  

When I installed and tried to configure NMP the best I could, we didn't really see a performance improvement (over no multipathing) since we needed to keep updating the system when VMachines moved or changed - maybe that's the biggest headache.  

But when we pulled native MP out and put in PowerPath we immediately saw at least a 20% jump.  We hadn't even configured it yet.  And now that we've tweaked it I think we're seeing over 33 or 34% better performance.  

We're even thinking that when we upgrade to VMAX's later this year we will beable to use fewer HBAs since Powerpath will take care of load spreading.

Now if we could just get EMC and VMware to change their licensing of PowerPath/VE from vsphere enterprise plus to include enterprise too, I'd been much happier!!!

Post 2: May 5, 2010 8:02 AM in response to: mrdenny 

Re: PowerPath/ve performance gain 

A longer term review has shown that our performance (IO throughput) with PowerPath/VE has peaked at about 32% to 35% (better than with NMP or no multipathing at all)!!!   We haven't measured before and after CPU use changes.  We might even be able to reduce some hardware costs during an upcoming upgrade!

Post 4: Jun 24, 2010 2:13 PM  in response to: PawelO 

Re: PowerPath/ve performance gain 

Not 100% sure if you would see same performance gain with HP.  I'm using CLAROPT and SYMOPT on EMC storage.  We actually havn't had to tweak much.

Although we initially had some NMP-owned paths mixed in with Powerpath/VE-owned paths and have subsequently removed the NMP ones, and made sure that Powerpath/VE can see all channels, we haven't done anything - definitely a plus in Powerpath's corner.  

For testing with iometer, we ran it on 8 VMservers running Exchange with ~30VMs total and I think ~2500 users per VM with each file at ~500M.  

PowerPath/VE showed significant improvements in throughput, less CPU use (by avg 31% over NMP).  Actually, the more load we put in test, the better the results.  Our first test was 4 servers with 10VMs and 1000 users per. 
------------
There's more, but -- in a nutshell: better availability, better manageability -- and honkin' performance boosts if you're pushing serious I/Os.  

Thanks to all for their comments!

Stepping Back A Bit

Just about everyone is aggressively consolidating around virtualization: fewer servers, fewer arrays, and fewer paths between them.

Features like VMware's DRS takes care of load-balancing across servers.  

Features like EMC's FAST do something similar around pools of storage media: flash, SATA, etc.  

And there's a clear role for something like PowerPath to pool and optimize across the connections between the two.

So, the next time you're chatting with your friendly neighborhood storage vendor, why not ask what they're doing about path management (especially in aggressively virtualized environments), and -- how does it compare to PowerPath?

Comments

[Itzik Reich]

Itzik Reich (EMC) here, the best thing that happened since vSphere 4.1 is that now customer can leverage vSphere Enterprise to buy PowerPath/VE, no need to purachase the Enterprise Plus for it..