« Sticky Is Good | Main | Defining Tenants In The SP Environment »



Feed You can follow this conversation by subscribing to the comment feed for this post.


You claim to be able to argue the capabilities or perhaps the lack of, in your opinion, of NetApp SMT, but at no point do you make an argument. You just state that you don't think NetApp SMT is all that secure and reference @texiwill.

How can you possibly argue, "a) the security, audit and compliance capabilities should -- at a minimum -- meet the defined requirements of the tenant."? In your own words, it's the requirements of the tenant. You can't argue what the tenants requirements are. You simply don't know what the requirements are.

I would love to read or hear real arguments around this, but you just don't make any here. And there is barely a mention of what EMC is doing to capture on this opportunity.

I hope you follow this post up with another that has details, arguments and specifics.


Mike Dunn

Chuck Hollis

Hi Mike

Well, let me put it this way.

If I were to market a solution as "secure multitenancy", I'd be darn sure it was (a) secure, and (b) multitenant.

Not everyone needs security. Not everyone needs multitenancy. But for those that do, I would argue that whatever is proposed should be fit for purpose. Truth in advertising, don't you know?

Having worked extensively with many corporate IT groups, I know that the proposed solution would be a non-starter for many pragmatic use cases. Again, as I point out in the blog post, everyone's needs are different.

Had the solution been marketed as "simple multitenancy", I'd have no argument whatsoever, since that is what it does.

Thanks for the comment

-- Chuck

Edward L. Haletky (@texiwill)

There are 3 Administrators within an Multi Tenant solution. The application Admin, The Cloud Admin of the Tenant, and the Provider Administrator.

The Application Admin is concerned only with their application(s), the Cloud Admin of the Tenant is concerned with all the virtual bits that make up the cloud environment they are renting. This could be Software, Platform, or more likely Infrastructure. Which means this Admin lives within the tenant as well as within the provider space. They can often see everything within the tenants environment.

The last administrator is the Provider Administrator and he can see ALL data within the cloud environment. I do not care if it is written to an encrypted disk or not, if the tenant is running within a virtualization host at all, this administrator can if they want to, see/manipulate/destroy all data as it is being written to the disk, all data that is encrypted within a virtual machine, all data as it goes over the network, and all in memory data of a virtual machine. In effect, they can see, manipulate, and by their very nature ACCESS all data in motion and most if not all at rest. If you question this capability please listen to the Forensics discussion on The Virtualization Security Podcast (http://www.virtualizationpractice.com/blog/?page_id=4852).

So my question is how do we protect against this administrator accidentally deleting, accessing, and manipulating tenant data without the tenant knowing about it. In other words Confidentiality and Integrity.

Availability is well covered, but Integrity and Confidentiality is ONLY covered once it hits the disk, not before, not when the data is in motion.

And since the Cloud Admin of the Tenant also lives half in the space of the Provider Admin, how do you protect this user from seeing other Tenants data?

Yes, I know that there are tools such as HyTrust, vCenter, and others that try to protect against this but at the moment they can all be bypassed one way or another. What controls are in place or required to prevent this? What auditing is required?

While Chuck mentioned nearly all the EMC/RSA/VMware products, my thought is that they are inadequate without a good base to build upon and a proper 'secure' architecture that accounts for the most secure of tenants and allows those not interested in all that security to pick and choose what maps to their existing policy. I have yet to see such an architecture. I see a start at handling Availability and security of data at rest, and some guidance from CloudAudit.org and the Cloud Security Alliance.

When you enter the cloud today, you really need to ask yourself "Do you trust the Cloud Administrator with your customer and employees PII?" I certainly would not.


@Chuck Thanks for the response, but you still don't make an argument or case for why you believe it to be un-secure or able to support many tenants. SMT may very well be a non-starter for many pragmatic use cases, but then again you could say that about any product, including your own.

Several times in this piece you say, "I could argue." I am asking for this argument. You don't provide any details. You said you could pick apart the "multitenancy" aspect, but you don't. And you just said that you wouldn't have had a problem if it was marketed as "simple multitenancy" just not secure... so are you in agreement with NetApp that this does provide a multitenant environment?

Where's the beef, Chuck?

Chuck Hollis

Hi Mikedunn

Since you're a NetApp reseller, I don't know if you sincerely don't understand, or whether you're just toeing the partner line.

I did my best to explain the primary concerns: (a) there is no protection for the tenant's data from the service provider -- hence not really secure, and (b) there is no mechanism to isolate workloads on the shared array proposed -- hence not really multitenant.

For many enterprise workloads, being unable to protect sensitive data, or guarantee a given storage performance level -- well, that makes it a non-starter.

If I sold someone a data recovery solution that "sort of" did data recovery, it wouldn't be very good, would it? If I sold someone a business continuity solution that "sort of" protected against disasters, the same thought would apply.

And if I sold you a "secure" solutions that wasn't really secure? And a "multitenant" solution that wasn't really multitenant?

See @texiwill's comments above -- basically saying the same thing, only with more words :-)

Sorry I can't be more clear for you ...

-- Chuck


Hi Chuck Hollis
Hardly toeing the partner line... And I sincerely don't understand what your argument is, since you have yet to make one. I am asking that if you are going to write a blog blasting a product, provide some explanation why. And if you are going to say you could do a "take-down" of a product, then provide some detail as to how. You haven't done any of these. You claim to have knowledge of a product, but don't provide any details of this knowledge. What you said, and what Edward said, aren't even "basically" close to the same thing.

Again, I will reference your earlier comment that you wouldn't have a problem if NetApp marketed SMT as "simple multitenancy." So, again, I will ask, do you take issue with the capability of SMT to deliver multitenancy or not? If you do, please explain why.

Edward L. Haletky (@texiwill)

Actually Mike, what I stated is the overall problem with all current SMT solutions. They are just not secure with 'data in motion'. If the 'data is only protected at rest', what happens once I read that data, that data NOW is in motion through the virtual environment and therefore subject to being read, manipulated, destroyed in transit by the cloud provider's administrator, the Tenant's Cloud Administrator, or even the application administrator (depending on the application).

The problem with many storage folks thinking about SMT, is that they only concentrate on the storage aspect of data at rest. In a true Multi-Tenant Cloud, data is hardly ever at rest and as such is subject to attack as it makes its way through the layers from disk to the application.

You may argue this is a 'hard' nut to crack, but unfortunately, this is not true. The most simple of security suggestions are ignored wholesale by current virtualization administrators, and therefore their environments are seriously under-protected. I speak of isolating the management of the 'environment' from other networks. I know of at least one Cloud company that has zero isolation between this all important network and its subsequent components making it a nice juicy target.

Just protecting at the storage layer, DOES not imply SMT. It is only one small part of the entire picture.

More food for thought, how do you protect the data in motion from 'escape the VM' and other side channel style attacks as well? Not sure you can at the moment either. You can only mitigate the possibilities of such attacks, but if you do at least some of those mitigations you WILL drastically affect performance (VDI) solutions.

As I stated earlier, availability is well understood and fairly well solved, but Integrity and Confidentiality has yet to be.


Thanks Edward for providing some additional clarity.

Could the problem be more related to virtualization as whole? And the difficulties in securing VM's? I forget what the exact stats are that Gartner estimates, but something along the lines of 70-80% of virtualization projects are done in a less than secure manner. Why is this?

As you point out companies like Hytrust are popping up to solve this to a certain degree, but I think until virtualization admins and corporate IT take virtualization security seriously, it isn't a top priority for them currently. That's not to say that it shouldn't be or that security isn't important, as it obviously is, but I think as whole the security of VM's and virtualization is not well understood.

You may disagree, but I think there is always going to be some risk involved in regards to protection from the provider admin. I don't know that this risk will ever go away regardless of advances in technology.

Chuck, would still like to hear from you on some of my questions.


Mike Dunn

Edward L. Haletky (@texiwill)


You are correct, it is a continual issue within any virtualized environment, however there will soon be technology that makes it much harder for a Administrator to damage the system, whether or not it will be used within a traditional data center is up for grabs. The why it is this way is more historical than anything else. People feel that if they virtualize they are more secure, this is a blatant false statement but someone sold it to them, and it keeps popping up when I hear from marketing types.

But we are really talking about Secure Multi-Tenancy here, outside the traditional data center. Specifically within the Cloud whether private or public. Until the 'designs', 'architectures', and 'solutions' look at solving this problem they are frankly not Secure and I for one would hesitate before suggesting them without a bunch of attorney's reviewing all contracts, requirements, liabilities, etc.

Yes I know people use the cloud today, and I even do to a small extent but I go in with my eyes wide open, knowing full well that anything I put there is viewable by all administrators.

Any Secure Multi-Tenancy solution absolutely requires protection for the data from ALL including the administrators. How much protection is something dial-able by the Tenant not the Cloud Provider.

Where do we need to start this: with a workable definition of 'Tenant'.

Chuck Hollis

Edward rightly points out that it all starts with a definition of "tenant" -- and there's a reason I tend to choose that word.

Most the desired attributes and properties of "tenant" in the SMT world have strong analogs in the real world, so it's a useful starting point for describing differences, rather than similarities.

When I lease a property from a landlord (service provider), there is set of clear expectations around rights and responsibilities. The landlord provides a building in usable condition for the intended purpose. The landlord does not have any access to the property in the building. If the landlord needs access, there's a process to do that.

I, as tenant, can sub-let the property if allowed, can use it for a wide variety of purposes without disclosure and/or approval, have certain protections from other tenants as well as the landlord, and more.

The physical model has strong corollaries in the idealized SMT world.

So, join in with me here, people. If we're going to define "tenant" in this cloudy world, should we start with a workable real-world model as a starting point?

-- Chuck

Edward L. Haletky (@texiwill)

More food for thought: Defining Tenants for Secure Multi-Tenancy for the Cloud -> http://www.virtualizationpractice.com/blog/?p=5793


Either of you happen to be in Chicago this week? Looks like a pretty good panel discussing some of what we have been going at on here.

I am going to try and submit some of our questions from here to the moderator. Feel free to add your own ideas and I will submit all of them and see what these folks think.


Mike Dunn

Chuck Hollis

Nope, not traveling much these days. I'll be interested to see how these -- and other topics -- are handled. My suspicion is that -- generally speaking -- it's early days on the discussion.

Thanks for the offer, though ...

-- Chuck

Edward L. Haletky (@texiwill)

Neither will I be. :} I will be at vForum in Boston however.

The comments to this entry are closed.

Chuck Hollis

  • Chuck Hollis
    Chief Strategist, VMware Storage and Availability Business Unit

    Chuck works for VMware, and is deeply embroiled in all things software-defined storage these days.

    Previously, he was with EMC for 18 years, most of them great.

    He enjoys speaking to customer and industry audiences about a variety of technology topics, and -- of course -- enjoys blogging.

    Chuck lives in Holliston, MA with his wife, three kids and four dogs when he's not traveling. In his spare time, Chuck is working on his second career as an aging rock musician.

    Warning: do not buy him a drink when there is a piano nearby.

My Official Blog

Blog powered by Typepad
Member since 10/2006