Managing Block Storage At Scale: Introducing EMC ProSphere

Personally, I think the whole topic of storage management doesn't enough industry attention.  Far more ink seems to get spilled on more prosaic topics like debating storage network protocols.

By all rights, storage management ought to be a topic of vibrant discussion and debate.

There's so much is in play here:

• dramatically growing information volumes and associated storage farms
• new storage technologies, like flash and storage federation
• the growing predominance of virtualized workloads
• the rapid shift to IT-as-a-service models (e.g. "cloud")
• and, as ever, the need for efficiency, control and choice

I think we collectively spend too much time debating the technologies themselves, and perhaps far too little time discussing how people might use them effectively.

So it seems almost precognitive that today EMC is launching an entirely new storage management environment targeted at large scale environments with demanding block-oriented storage workloads.

EMC And Storage Management

Having been at EMC since 1994, this isn't the first time the topic of storage management has come up.  We've been swinging away at it for almost 15 years.  I can track the somewhat challenging evolution of the discussion, almost year-by-year.  It's been a frustrating journey.

On one hand, customers seem to be generally saying the same sorts of things thing, e.g. make it easy, automated, powerful, cheap, scalable, extensible, etc.  All the buzzwords sound largely the same, even if they're polar opposites :)

The real challenge starts when you start diving in to the details, and realize that just about everyone wants something very different, depending on who you're asking, and when you're asking them.  

It gets even harder when you have to decide whether to build a product that works the way people work today (however inefficient that might be) vs. creating a product with an order-of-magnitude more efficient management paradigm.  

Change is hard for people, no matter what the benefit might be.

The foundational approach to EMC storage management is easy to understand: all EMC products expose their functionality through a wide variety of APIs and CLIs.  As an example, we were early proponents and advocates of the SNIA SMI-S standard; largely in hopes that there would eventually be a vibrant market of third-party storage management solutions for customers to choose from.

No such luck.  

Today, the vast majority of enterprise storage management software is sold with -- and oriented to -- the primary arrays that the customer is using.  It's rather rare to see software from Vendor A managing storage from Vendor B.  Not ideal, but that's the way the market has evolved.  Since EMC is #1 in storage, it's no surprise that the market analysts put us at #1 in storage management software as well.

EMC first entry into the enterprise storage management market -- ControlCenter -- has now been out there for over a decade.  It was very ambitious and innovative for its time.  It started by doing Symmetrix very well, and over time branched out to managing non-EMC storage arrays.  You'll find it in shops large and small.  To be frank, it has had its challenges.

More importantly, the world has changed a lot in the last decade -- and we think it's time for a newer approach.

For starters, in more modest environments, EMC's Unisphere has done a good job in replacing the legacy CLARiiON and Celerra management environments with a nifty and slick UI that's easy enough for occasional use, but can drill down and do powerful things when needed.  

Unisphere wasn't simply a repackaging of the older environments; it was a clean-sheet-of-paper approach to a clear use case -- the IT generalist.

Such is the nature of EMC ProSphere: a clean-sheet-of-paper approach to a clear use case -- the IT storage specialist.

Scale-Out Is Different

We've come to understand that notions of "ideal enterprise storage management" align closely with the storage model being used.  There is a clear and compelling case that can be made that there is no "one size fits all" storage management model that is feasible across the industry. 

Where the storage management discussion gets interesting is when the focus shifts from modest multipurpose dual-controller arrays to the newer generation of scale-out storage architectures.  Any time you're talking storage at significant scale, management efficiency becomes a front-and-center concern, as it should be.

Today, EMC has one general purpose (e.g. "unified") storage produc line (VNX and VNXe) and three scale-out storage architectures: one targeted at object models (Atmos), one targeted at file systems (Isilon) and one targeted at block devices (VMAX).   To claim that any single storage architecture can excel at all four use cases would be, well, disingenuous.

If your storage needs are relatively modest - and you'd prefer an all-in-one storage platform - that's where VNX fits in.   Indeed, as you look at Unisphere for VNX and VNXe, you'll see how the storage management model closely aligns with the use case. If you haven't given the UI a whirl, here's a link to a simulator you can download and try for yourself.

At the other end of the storage management spectrum, you'd probably agree that both Atmos and Isilon have large-scale management models that are logical extensions of their storage model and use cases.  It's fair to describe them as built-for-purpose vs. all-things-for-all-people.

By extension, the best way to understand the ProSphere use case would be as "storage management for scale-out block models", e.g. VMAX and related technologies.

Imagine an IT professional that's responsible for an overwhelming number of block devices, ports and associated applications.  Not only multiple arrays, but multiple locations as well.  Physical hosts, as well as a fast-growing pool of virtualized hosts.  A storage media mix that's progressively using more flash to boost performance and lower costs.

I think it's best not to think of this person as a traditional old-school "storage administrator"; more along the lines of a "storage service delivery manager" -- organizationally responsible for delivering a wide range of storage services predictably and efficiently.

So how does the new EMC ProSphere do things differently for this increasingly important use case?

Start With Ease Of Installation

If I'm being blunt, early versions of ControlCenter were a royal PITA to install and configure.  Later versions sanded off many of the rough edges, but it still was a project that you planned for.  

Part of the pain was the dependence on agents running on hosts, part of it was the need for beefy physical services for the management applications to run on, part of it was the painful length of time it took to discover a storage/server landscape.

All orders-of-magnitude different and better with the new ProSphere.  

For starters, it's built on an agent-less model, and uses standard interfaces to go beyond simple storage/network/HBA detection, and can give you decent insight into what's happening on the servers -- without agents.  The back-end components are designed to run in virtual machines.  End-to-end discovery of one of the largest existing production ControlCenter environments took under eight hours with ProSpehere.  Most early users share that their environments were fully discovered in under an hour or less.

Put it all together, ProSphere gets up and running with an absolute minimum of fuss and resources.  A big departure from ControlCenter, and markedly better than competitive alternatives.

Designed For Scale

Of course, any management product for storage at scale has to boast impressive numbers for how many objects a management environment can handle.  EMC ProSphere is no slouch here, the raw stats are impressive: manage up to 1.2 million volumes, 36,000 ports and 18,000 hosts as part of a single logical domain.

If you're sitting there, shaking your head, and wondering "who the heck would need that?" the answer is a) more than few people today, and b) even more in the near future.  Scale-out block environments can get really, really big, just like scale-out file and object.

It's one thing to boast big numbers; it's another thing entirely to provide logical constructs for efficiently managing groups of related objects at scale.  And that's what you can see here: new constructs to manage at scale.

The key innovation here is called "Smart Groups": any arbitrary collection of management objects that share properties -- horizontal classes like ports or storage devices, or -- more importantly -- vertical classes like "silver class storage services for VMware".  

Any managed object can be a member of an arbitrary number of groups.  They can be created, edited or re-grouped at any time as your environment evolves -- no need for massive amounts of up-front planning that turns out to be wrong anyway :)

By progressively classifying -- and re-classifying -- storage management objects into overlapping and evolving groups, storage service delivery managers can iteratively template-ize their environments to gain great efficiencies, while still retaining the flexibility to do one-offs as needed.

From Trees To Search

Another example of the "designed for scale" thinking shows up in how you find things.  Old school: everything in a heirarchical tree of one sort or another.  New school: wildcard searches that can leverage whatever naming conventions you've decided on to quickly find objects of interest.

For example, a search on "*exch*" can give you all the VMs, hosts and HBAs associated with your Exchange environment -- that is, if you're using that naming standard.  Or "*FC*1??" can get you all the FC ports in the 1XX range.  Pick the items you're interested in, create a Smart Group, and you're off to the races.

.. And Immediate Drill-Down

Click on any object, and you'll immediately see the end-to-end context for that object.  EMC ProSphere claims that two clicks not only gets you an end-to-end view of a specific server/storage relationship, but can also compare both the server-side as well as the storage-side performance metrics. 

There's a slick panning and pivot capability that has to be seen to be appreciated.  Put the ProSphere UI up against anything (from EMC or anyone else) and the visual experience is like night and day.  If you're in to storage management at scale, you owe yourself to see a demo.  

To me, it looks like power tools for storage pros managing at nontrivial scale.

Agent-less?  Yes!

Between WMI, SSH and the VMware API, there's no real need anymore to have dedicated agents running on physical or virtual hosts as there were in the past.  Indeed, all the rich functionality and data that used to mandate an agent-based approach can now be achieved simply by probing the host using a documented interface.

That being said, there are specific customer environments that prefer a dedicated agent in some situations for security reasons.  The EMC ProSphere architecture allows for selective use of host-resident agents, which will ostensibly be coming in a near-term release.

Tight Coupling With Advanced Storage Features

Although ProSphere does EMC block devices today (e.g. VMAX, CLARiion, VNX) it makes no bones of doing a super-good job of tightly integrating with VMAX array technology.

As an example, if you're familiar with FAST VP on the VMAX, you'll quickly appreciate that it demands a service-oriented paradigm: create pools of resources associated with different service levels, and make them available to hosts.  

ProSphere works hand-in-glove with features like FAST VP to extend the management paradigm even further -- to ports, switches, HBAs as well as both physical and virtual hosts.  Put differently, FAST VP lets you set up service levels within the array itself; ProSphere extends the domain to the physical or virtual operating system with everything in between.

Ready For Storage Federation

As more and more IT domains start spanning multiple physical locations, it's important that the storage management model inherently understands multiple locations -- and the relationships between objects across these locations.  

Not only is this a great convenience for the storage service delivery manager managing across multiple locations, it becomes downright essential in a future world where you're moving hot workloads from location A to location B, using something like VPLEX Geo for example.

Early Feedback From Customers?  Extremely Positive, Thanks

One of the advantages we've gained from chipping away at storage management for so long is key: don't go off into some ivory tower and build something.  Instead, engage your target users at each and every stage of the process.  Iterate fast, get feedback often, and quickly incorporate it into your thinking.

That's what the EMC ProSphere team has done now for quite a while.  Customer thinking is baked in vs. bolted on.  If you'd like an insider's perspective, check out this blog post from our own David Spencer.

While I don't expect everyone to be raving fans of the initial product, the early customer feedback is quite encouraging indeed; for example, this video.  Or maybe you'd be interested in ESG's take.

Current And Future Capabilities

The current GA version of ProSphere is quite compelling in itself and worth an evaluation, but -- understandably -- not everything could be done on Day 1.  The team -- rightfully so -- thought it more important to get the foundational aspects right vs. simply larding on all sorts of point features.

If I had to point to the current strengths of the 1.0 product, I'd highlight easy installation, quick discovery, great visualization and end-to-end infrastructure performance monitoring.  The current version of ProSphere also has moderate integration with two other important modules of note.

One is the Symmetrix Performance Analyzer.  Nothing does better at doing a gory, deep drill-down into VMAX performance.  At this point, it's a link-and-launch integration with moderate context integration.  I know the team plans a deeper integration at some point in the future, but the current offering will tell you more about VM-to-array performance than probably anything else out there.

The other module of note is the EMC Ionix Storage Configuration Advisor

At its essence, this product keeps you out of trouble, especially in larger environments.  

As your infrastructure evolves with new requests, new technologies and new configurations, it is continually checking three sets of knowledge bases: EMC's e-Lab interoperability matrices, a continually evolving set of EMC's best practices, as well as any locally defined configuration policies.  It too is destined to get subsumed by ProSphere in a future release.

As I look at the broader roadmap schedule, the ProSphere team intends to be very busy indeed.  For example, broader support of non-block EMC devices, as well as non-EMC arrays.  More complete monitoring and alerting vs. the performance-oriented monitoring and alerting in the current version.  Extended capacity forecasting, planning and reporting tools -- not only physical storage capacity, but performance capacity as well.  

All intended to be delivered in a relatively timely fashion.  If you're understandably skeptical, I fully understand that.  The good news?  Most of the required storage management componentry already lives in the EMC portfolio -- it just needs to be integrated into a useful user experience.  

And that's where the heavy lifting will come :)

Advanced Tools For The Storage Professional?

I think industry interest in storage management undoubtedly has to grow in the near future: there's just so many forces at play here that will consipire to make new storage management frameworks -- and new service-oriented storage operational models -- an increasingly important topic.  

EMC ProSphere reflects much of this new thinking.

I also would like to re-iterate my case that there will never, ever be a "one size fits all" intergalactic storage manager that covers the majority of use cases -- just like there's no single storage architecture that does this.  

You don't build a killer management product by trying to be all things to all people: better to pick your target audience and nail it.

That's reflected in EMC's current storage management offerings: Unisphere for the VNX in general purpose environments, and specialized storage management environments for scale-out block, file and object respectively.  

Bottom line: if you're managing block storage at scale, you need to take a look at the new EMC ProSphere.

Comments

[Dvellante]

You are an awesome communicator Chuck. As one who sorta writes for a living I have a great appreciation for your writing skills. Nice overview.

Where I'm stuck in your assessment - and you touched on this a little - is heterogeneity. I've always had the impression that the paucity of storage management software from vendor A managing storage from vendor B was, in a large part anyway, the fault of storage vendors. It seems none of you ever wanted to be Veritas-like - you know the old line about a "Hardware Agenda?"

In reading your press release, again it seems to be 100% focused on managing EMC-only products. And in addition to all the things you list that customers want...the ability to manage other vendors' storage is downplayed but always comes up top of the list for customers. My sense was always that the business case wasn't there for emc and your competitors.

But is that really the case? Why leave that to Symantec. Or could VMware play a role there? Or are you planning to obtain Cisco-like market share so it won't matter :-)

Would love for you to double click on the heterogeneity issue a bit - specifically - even lacking a SNIA standard, why can't EMC do a better job providing support for other vendors' arrays?

[Storagezilla]

Hi Dave.
Symantec traded in "No Hardware Agenda" for "No Hardware Experience" with Huawei over the past few years so I wouldn't dust that quote off too often.

Around heterogeneity, ProSphere uses standardised interfaces such as WMI, SSH and SMI-S as well as Restful services for interoperability. What's in the current release is what we have ready for market today, there will be expanded support beyond what Chuck listed in the future.

[Chuck Hollis]

Hi Dave -- as always, thanks for the kind words.

I think it's rather commonplace for industry observers to blame "the vendors" for any issue that the watchers feel should be addressed.

For me, the root cause is not vendors, but customers.

Look at it this way, if there was strong customer demand for rich, heterogenous storage management tools (and backed by real purchase orders), trust me, you'd see all sorts of great offerings out there.

Sure, everyone says they want it. Who wouldn't? Next time, ask them how much they'd pay for it, and you'll probably get a different picture indeed :)

We, as vendors, tend to spend our R+D dollars on what our customers tell us they'd spend money on. And you don't see a lot of RFPs out there for rich, heterogenous storage management environments.

The beauty of capitalism at work.

That being said, there's more heterogenous support in our portfolio than I think you're giving us credit for. Familiar examples include RecoverPoint, VPLEX, PowerPath, the previous ControlCenter, Ionix, etc. etc.

In the case of ProSphere, the team fully intends to broaden and deepen support of non-EMC arrays -- but not as a first priority, as you'd might expect. Again, capitalism at work.

This time around, it should be a bit easier -- but not much.

Thanks to all of us vendors, there's half-decent SMI-S support in many of the popular storage platforms as a starting point, which will inevitably have to be extended by all sorts of one-off APIs, CLIs, scripts, etc. Vendor-by-vendor SMI-S implementations are sort of hit-and-miss these days, unfortunately.

Expensive stuff to develop; more expensive to support.

I think there's an opportunity for you and your team to dig deeper into this issue, and shine some independent light on the purchasing dynamics at work here, especially in larger storage environments.

We've got our views -- it'd be useful to see what Wikibon uncovers.

Thanks again ...

-- Chuck

[Thanh]

Was Prosphere developped from scratch or will it used some Controlcenter code or is it an aptare-based software ?

A guy having pita with ECC.

[Chuck Hollis]

Thanh --

The team tells the the vast majority of code (as well as architecture, design philosophy, etc.) are brand new. There was no mandate or incentive to re-use existing functionality.

I don't know what "aptare" is, but I doubt it was used.

-- Chuck