« My Friday Rant | Main | A New Chapter Begins In The IT Stack Wars »

January 26, 2010


Vaughn Stewart


You bring up some excellent points in this post around enabling secure solutions for cloud providers. Restricting access to content, Delegated Management, Verified solutions are critical components required to make secure multi-tenant solutions a reality in the marketplace.

However, these technologies and operational practices are meaningless without the infrastructure to deploy on. It is on this point that I find your closing comments rather interesting.

Maybe it's just me, but the last lines of this post change the topic from one of requirements for secure infrastructures to one of dismissing technologies and a solution set that EMC doesn't offer.

In particular your backhanded compliment to the Cisco-NetApp-VMware SMT solution. Lines like, "Back to our competitor's proposed solution -- a somewhat-secure-but-not-really-all-that-secure approach."

May I suggest that you might have been better served by striking said line and just delivering the message that, "EMC should probably get busy and focus a bit more on this topic."

Mike Riley

"...a somewhat-secure-but-not-really-all-that-secure approach."

How so? Something amiss in the joint whitepaper (http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/Virtualization/securecldg.html) from Cisco, VMware, and NetApp?

"Does it solve a useful customer problem for people concerned with security? Perhaps, but I'm incredibly skeptical."

I'm not sure why. EMC partners with Cisco and owns VMware. Have you voiced this concern with Tony Bates from Cisco or Paul Maritz from VMware? Were they not credible resources to comment on this solution? Is there something they missed or are missing? Probably not a fair question for either this blog or the 30 minute event today since there are, as you mentioned, a myriad of details one needs to consider as part of this solution. Still, I'm not sure how, with the companies agreeing to design, implementation and support and current customer adoption (Jared Wray, Tier 3 video - check it out), one can be skeptical of something that already exists.

In sum, you had 3 executives spend 30 minutes in a collaborative discussion giving you the high-level design and business drivers for the solution; you had a customer step up and say, "Yep - it works great and I'm using it today" and for more detail you had a whitepaper with low-level design considerations and a 24 hour support agreement announced.

Now, I think it's good for customers to have a healthy bit of vendor skepticism. Quite frankly, I too am a bit surprised that EMC hasn't put a lot of thought into secure multi-tenancy, but I get the sense from this blog that because EMC hasn't, readers should be skeptical? I didn't see any supporting points on why they should be.

While EMC mulls this idea over, I'd encourage readers to look at what has already been put into practice by joint Cisco, VMware, NetApp customers. As Tom Georgens put it in the video, this isn't a theory. It's already being put to good use. No "perhaps" about it.

Chuck Hollis

Hi Vaughn

I was hoping for some answers to the questions I posed. I didn't get any. So I guess they'll remain open questions for the time being.

As far as "solutions", I'll give you credit for the work done. That doesn't mean you've solved a useful problem yet. And we certainly wouldn't label something "secure" unless we felt it truly was. Guess that's a difference between our companies.

Best of luck with this going forward!

-- Chuck

Chuck Hollis

Hi Mike -- congratulations on your event today!

But when the adrenaline wears off, then what? Cisco's networking capabilities works with all devices ; VMware's security capabilities are famously agnostic, which leaves us with the inevitable "why NetApp?"

We are pointed to MultiStore, which creates a virtual array. A neat trick, but "secure"? That's where I have an issue.

I was also hoping for some credible responses for the issues I raised, but didn't get any yet. So the topic is still open.

Best of luck going forward with this!

-- Chuck

Jonathan Donaldson

Hi Chuck,

My team, along with teams at Cisco and VMware worked long and hard to deliver an architecture that solves the customer issues of transferring isolated app / tenant silos into a shared cloud infrastructure. We truly feel that this solution offers the security and SLA protection that these environments require.

You state in your comment :

"why NetApp?"
We are pointed to MultiStore, which creates a virtual array. A neat trick, but "secure"? That's where I have an issue."

Here is a link to an independent 3rd party security audit done on MultiStore:


We are in the late planning stages right now for a full stack 3rd party security audit of the solution.

All of that being said, it is not just MultiStore that brings strength to the solution to answer the 'why NetApp' question, it is the SLA protection with FlexShare, the efficiencies inherent in NetApp controllers, and our willingness to roll up our sleeves and get it done for the customers that answers the 'why NetApp'.

Chuck Hollis

Hi Jonathon (and the rest of the NetApp team)

Yes, there's some fine work here. And I'm sure you all worked long and hard.

It's all about how you frame the problem. I'm reasonably convinced that tenants are isolated from each other. But how are tenants isolated from the service provider?

I brought up three simple examples. You are the third person from NetApp to talk about many interesting things -- none of which even begin to answer the questions I raised.

This security business is not based on feeling and opinions. At some point, there's a certain rigorousness of approach that's needed, and all the "independent third party" opinions do little to address fundamental questions such as the ones I -- and many others -- will raise.

More work to do, guys ... much, much more.

-- Chuck

Mike Riley

Hi, Chuck,

As I mentioned, I doubt the execs were going to get into the weeds on the technical details nor was I in a 4 minute YouTube video. Details are available in the joint whitepaper (link provided previously). I'd suggest understanding the security aspects of IPspaces, RBAC, and at least reading the whitepaper before commenting.

Encryption and secure disk replacement - all handled. I would reference specific applications in the intelligence community but then I can't do that, can I?

Delegation of roles, separation of church & state - yep, the solution has that too. That's the very nature of RBAC and Multistore. Again, I'd refer you to the whitepaper and NetApp documentation. (I can see the e-mail traffic now inside EMC: "What in blazes is RBAC?"

Passing an audit - done. Not only for secure public sector/military installations but commercial applications as well. Remember, this solution is already installed and not on a whiteboard somewhere. I'd also refer you to the Multistore analysis at www.matasano.com.

The answers to your questions were already available in public documentation regarding data access, RBAC, IPSpaces, logging and auditing. All easily accessible in the same amount of time it took to pen this blog but, then, what fun would that have been, right? Kind of kills your blogging momentum when you have to go off and do a little fact checking.

Now that I've answered your questions. Would you care to answer mine (previous post)?

Jonathan Donaldson


First off, good discussion.

So if I understand your security question it boils down to "Quis custodiet ipsos custodes?"

Translation for the Latin challenged: http://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes

I would argue that if your security posture is so extreme that you need to guard your data from your trusted providers, you are probably not the type of organization that would ever allow that data to move beyond the organizational firewalls (or one that employs any type of non-vetted contractor/partner/employee). This would be an extremely niche customer, and thankfully for both NetApp's efforts and the vBlock efforts, not the norm. You are more likely to loose data from an authorized disgruntled employee than a service provider whose business model relies on customer trust.

Like any outsourcing endeavor, you should look to those trusted providers with both a good reputation, audits, and the controls to prove it. Possibly the existing SAS 70-II audit reporting could be implemented on the logging from the infrastructure. This would be a strictly forensic measure, but could over time prove that the provider's procedures are effective for the 95% of the organizations that will use this type of solution.

For those organizations that have extreme data security requirements, they will most likely implement internal private cloud environments.

Over time you will see more and more additions to the security of this solution, but remember, security one of four key concepts in the SMT, all are critically important for the solution to be realized.

Chuck Hollis

Hi Mike

I did download the whitepaper, and spent some time reading it. I hit the "you gotta be kidding me" meter at about 10 minutes, and stopped reading.

Just to be clear, I was addressing the marketing claim of "secure multitenancy" in presumably service provider environments (as marketed by NetApp), and not environments where shared facilities are under the control of a trusted entity, as you are suggesting.

Here's what I picked up -- please correct me if I'm wrong.

1) Encryption of virtual machines is done at the physical server level, rather than virtual machine level. Any key management is presumably under the control of the cloud administrator, and not controlled by the tenant. Again, encryption and key management were not covered in detail, so I'm inferring here.

2) The document is very clear that "best practices" is that the cloud administrator has complete and unfettered access to all resources in the domain. Tenants are not given the choice as to what roles can be played by the "cloud administrator". I can cut and paste here if you'd like.

3) I believe the white paper made no reference to procedures for restricting data intermixing on physical storage devices, nor secure erasure procedures, nor securing access to storage administration, nor securing access to service personnel. Please correct me if I'm mistaken.

4) There was no provision that I could see in the solution for providing tenant-visible audit trails of surrounding infrastructure: configurations, changes, security events and the like. Again, I am inferring from the lack of discussion in the document.

As I'll repeat again, the solution seems to do a decent job of separating logical tenants, but not protecting the tenant's interests vs. the service provider. In many people's minds, that's what "secure multi-tenancy" is all about. If you have another interpretation, we should discuss in detail.

Now, for some suggestion on style points that might help towards a more productive industry discussion:

!) The solution was promoted as "cloud", implying an external service provider who may or may not be trusted. None of your examples correspond with the use case marketed.

2) My understanding is that you passed an "evaluation", and not an audit -- based on the links provided. There's a difference, as we both know. If there is some separate audit for the solution that wasn't originally published, please share the results.

3) We too work extensively in government secured environments around the world. Ditto financial services, health care, retail, etc. The required procedures are well documented, and not a secret in themselves -- as you should well know.

4) Forgive me if I blog on what appear to be gaps in "solutions", especially when it comes to security. We can quibble over performance, availability, economics, etc. -- but security (like high availability) requires more precision and less hand-waving.

5) I don't want to get into a duel with you and the other NetApp fanbois, e.g. "you didn't answer my questions, so why should I answer yours?". We didn't publish and promote a "secure multi-tenancy solution" -- you did. So be prepared to take a little heat if it comes up short in important areas.

Mike, security is an important topic, as we both know. It deserves an open, honest and transparent discussion of the risks, threats and gaps.

No security solution is perfect. But if you're going to offer up solutions in that space, be prepared to enter into an open, honest discussion of what threats you *do* protect against, and which ones you *don't*.

Again, a good effort by the joint team. Much, much more work to do.

-- Chuck

Chuck Hollis

Hi Jonathon

You can't have it both ways.

Either (a) the world needs a secure multi-tenancy solution it can trust, or (b) there will be no such thing, and we can skip talking about cloud in a meaningful way for enterprise IT.

I happen to agree with (a) above. Such solutions do existing in other domains, not just in the general case we're discussing here. And, based on what was shared yesterday, there's still a lot of work to get there.

Your comments regarding disgruntled employees made me smile -- simply because service providers get them as well!

-- Chuck

Joint tenants

The latest cloud storage-focused alliance between major vendors takes the form of reference architecture for secure realistic server secure multitenancy drawn up by NetApp with VMware and Cisco. Obviously, shared hardware effectuation greater utilization and efficiency along with equipment, operations, and utilities cost savings. In the past, security was the wrench in the works — would Citigroup really want to host applications on the same server hardware as Bank of America? Secure multi-tenancy may not solve all the cultural and IT problems, but it goes a long way toward doing so. Large organizations will be especially pleased –I can only imagine that agent federation Vivek Kundra and the GSA will be all over this announcement.

Ryan Michaels

Hey Chuck and Co,

Fantastic discussion from all parties concerned, I have found the reading informative and constructive.

I think both the VCE and CVN alliances are fantastic for the market today and will be suitable for 90% of the market.

The only comment i would make is that VCE was a great announcement to the market and im sure there are customers that will adopt it and love it.

However, deciding to post on another alliance pitched at secure Multi-tenancy(something that most people feel as core to cloud services from a provider point of view) and making comment on stuff that is pure heresay cheapens the whole discussion and looks poorly on you Chuck. Those people who read industry blogs know that Mike Riley is one of the most gifted IT professionals in the market along with guys like Chad Sakac.

I love reading your blogs but you have let yourself down with this one.

Chuck Hollis

Hi Ryan --- you need to go re-read this discussion. At some point, you entered the Twilight Zone ...

This has never been a personal attack on Mike or anyone else. This hasn't even been a diss on the work done by NetApp per se.

Any security discussion of any value has to be open, forthright and transparent. That's what you're seeing me do.

I raised several points that I and others found almost immediately.

The response to the points should be either (a) we think we've got that covered, here's how, or (b) we don't think that vuln is going to be a problem, here's why.

You, and everyone else associated with NetApp, has failed to offer up a response in either the (a) or (b) form. Mike pointed me at the exact same documentation that raised the initial questions, for example.

So, in many people's minds, this still remains an open question. Rather than try and discredit me, why not answer the questions?

-- Chuck

Chuck Hollis

"Joint Tenants" -- going forward, I'm considering simply deleting commentors with obviously faked credentials.

Either make a decent attempt to identify who you are, or I'll simple press the "delete" key next time.

-- Chuck


Security is a concern for few and still there are many customers willing to put their data on the cloud by taking that risk. I am not saying secure multi-tenancy not providing the security , what I am saying is that, it will take time for customers to move their data however there are some customers ready at this movement to move their by taking this risk. Its all about time and importance of the data.
Definitely it is not easy to move the data which is at most important to the client.



Kudos to EMC for providing a nice discussion platform for my company
where we have a fair chance to explain 'why netapp'

Tom Rafter

What I find highly amusing is that consultants I'm working with from EMC/Isilon describe Isilon version 6.5 as being capable of providing a secure multi-tenanted solution. Obviously they're not drinking the same kool-aid as Chuck is here and version 7 of the Isilon software will provide a secure multi-tenanted solution more akin to what their competitors provide.

Chuck Hollis

Hi Tom

I'm not sure I follow your train of thought here. The current capabilities of the Isilon platform are fairly well documented. Whether that meets your personal standards for secure multi-tenancy (or not) is subjective at best.

I'm not sure what NetApp offers in this regard these days, but at one time they had published a "secure multitenancy architecture" where (no joke) a single super-user had access to all resources and data, which probably wouldn't pass muster for many. Hopefully, they've adopted more of a segmented role model as found in the current Isilon release.

You say nothing about your requirements here -- I'd be interested to understand what you might find lacking.

Best regards

-- Chuck

The comments to this entry are closed.

Chuck Hollis

  • Chuck Hollis
    Chief Strategist, VMware Storage and Availability Business Unit

    Chuck works for VMware, and is deeply embroiled in all things software-defined storage these days.

    Previously, he was with EMC for 18 years, most of them great.

    He enjoys speaking to customer and industry audiences about a variety of technology topics, and -- of course -- enjoys blogging.

    Chuck lives in Holliston, MA with his wife, three kids and four dogs when he's not traveling. In his spare time, Chuck is working on his second career as an aging rock musician.

    Warning: do not buy him a drink when there is a piano nearby.
Enter your Email:
Preview | Powered by FeedBlitz

General Housekeeping

  • Frequency of Updates
    I try and write something new 1-2 times per week; less if I'm travelling, more if I'm in the office. Hopefully you'll find the frequency about right!
  • Comments and Feedback
    All courteous comments welcome. TypePad occasionally puts comments into the spam folder, but I'll fish them out. Thanks!