Towards A Serious Discussion On Secure Multitenancy

This whole topic has been bubbling for a while as we consider fully virtualized (and inherently oversubscribed) pools of resources which will form the basis of so many service provider (cloud?) models going forward.

A small skirmish erupted when NetApp started promoting a solution they dubbed SMT -- secure multitenancy.  While there were aspects that did address some concerns around security (but not all), and some concerns around multitenancy (but not all), many of us felt it was being over-marketed in several regards.

Indeed, I saw a nasty take-down by @texiwill at GestaltIT's Field Day about the security aspects. I could do a similar take-down on the multitenancy aspects as well.

The purpose of this post isn't to slam any competitor, it's to drive towards two key discussion points -- (a) what is needed, and (b) how do SPs monetize it? 

And I don't think I can do all of this in one post.

To Begin With

Enterprise IT folks are concerned with security: identifying and mitigating risks.  They do this in their traditional data centers using largely traditional approaches.

If a service provider comes to them and offers to take some of that work away, they have a rightful concern that security measures (and reporting) are at least as good as -- if not better -- than what they've got today.

Introduce a pooled resource model (shared servers, storage, network, back-end services, etc.) and that concern is heightened considerably.

A parallel discussion arises with service delivery, considering performance as an example.  Enterprise IT are in charge of service delivery, and have existing processes and capabilities to do so in their largely physical IT world.

Move to a world where everything is dynamically pooled (and presumably under the control of a service provider), the concerns are heightened.

Adding Balance To The Discussion

Not every enterprise IT function is created equally.  Not everything requires high levels of security, or robust service delivery.  There's plenty of market opportunity for SPs that don't want to invest in this stuff.

However, all the interesting enterprise IT workloads generally require high levels of auditable security and compliance, and generally require a robust approach to service delivery.  "Interesting" is code for "profitable" in this context.

And there's the halo effect -- even though your tenant may not need these things today, there's comfort in knowing that their SP can deliver these things if needed.  Today's pilot program is tomorrow's mission critical application, for example.

So, more often than not, I believe that most SPs offering IT infrastructure, platforms or applications as a service will want to invest in these capabilities, and become proficient at them.  It makes a certain sense to have premium capabilities, and then down-feature as needed to meet price points in the market.

So, how might we think about this?

Digging Into Security A Bit

Bring up the topic of security in any IT discussion, and opinions and perspectives will flow like booze during happy hour.  But how do we create a framework for sorting out what matters, and what doesn't?

My current thinking (with the help of others) has evolved to the following perspective.

a) the security, audit and compliance capabilities should -- at a minimum -- meet the defined requirements of the tenant.

b) all aspects of security, audit and compliance capabilities should be completely transparent and under the control of the tenant, and not the service provider.

c) the security capabilities must be able to protect against an untrusted service provider -- because everyone has a bad day once in a while.

If that's the baseline, there's some extra-credit available as well ...

d) ideally, the security capabilities available should exceed what the customer has (or needs) today.

e) there should be provisions for forensic support.

The last one (forensic support) I owe to Edward Haletky at www.virtualizationpractice.com, also known as @texiwill. 

He made the insightful point that -- from time to time -- tenants of a service provider run into legal difficulty.  Standard practice is to confiscate the computer equipment as evidence. 

Well, if the offending party happens to be a single tenant of a multitenant SP environment, that could create an ugly situation for everyone else, no?

The answer lies in the ability to provide a legally compliant "snapshot" of the tenant's information footprint on demand -- not only apps and data, but access logs, etc. and do so following chain-of-custody evidence rules.  At EMC, we've done that sort of thing for law enforcement agencies for a while, so no biggie -- *if* we think of it ahead of time, that is :-)

Now, if we could do all of this security stuff in a pooled, dynamic, virtualized (and oversubscribed) environment -- without resorting to "old school" physically isolated infrastructure, so much the better.

Going back to what was being proposed by another vendor, we could argue about (a), but (b) and (c) were clearly lacking.

Digging Into Multitenancy

At one level, when discussing multitenancy, the real issue becomes around having the capability to guarantee certain performance levels for individual tasks, applications, tenants, etc.

These core capabilities, in turn, form the basis of so many different business models that underpin various SP offerings.

In the physical world, this was straightforward.  You needed more speed, you bought faster server, more storage, faster storage, more pipe, etc.  Welcome to the world of physical hosting, co-lo, outsourcing et. al.

In this new pooled, dynamic and virtualized world, it's not so straightforward. 

Everyone is sharing the same pool of goodies, so there are some new issues to consider.  SPs make good money if they get good at managing oversubscribed services.

More importantly, it's possible to vary service levels up and down quickly -- if needed -- creating a fertile space for all manner of interesting pricing models.

So, if I were putting together a list of IaaS multi-tenancy asks, it'd go something like this.

a) the ability to set minimum thresholds of services delivered, cast in whatever terms make sense to the customer -- IOPs, GHz, MB/sec, users, response time, transactions, etc.

b) the ability to dynamically vary these service levels on short notice, pending available shared resource.

c) all aspects of service delivery transparent and visible to tenant (including costs!)

d) protection against unruly tenants who exceed their allocated resources.

Going for extra credit, I'd add:

e) the ability to establish maximums as well as minimums

f) the ability for tenants to "sub-let" available resources (and manage associated service deliver) to multiple applications or sub-tenants.

Again, going back to what a certain vendor was proposing, I could argue that the server and network components did a decent job at (a), (b) and (d) but didn't address (c).  And the storage vendor couldn't do any of (a), (b), (c) and (d) as far as I could see.  They had a piece of software that would "suggest" to the shared resource what should be done, but that was about it. 

Hence my rant.

Secure Multitenancy -- Enabling Technology

Frankly speaking, I think most of the pieces are coming together to nail most of this during 2010 and 2011.  It's not all here yet, but it looks like it will be soon.

The underlying premise around security is simple -- as virtualization establishes a "new perimeter" for security, it can form the basis of security, auditing and compliance measures that are potentially far more secure and streamlined than anything we've seen in the physical world.

One fascinating piece is the "trusted hardware root" technology coming from Intel, VMware and RSA.  It arguable can be used to create a more trusted -- and verifiable -- security environment than *anything* available today -- physical and virtual.  Complement these with more standard integrated security offerings for fully virtualized worlds, and it becomes interesting.  Add in the potential future capability to encrypt on a per-VM basis (where the tenant owns the keys), and it becomes downright compelling.

Coming in from the other side, there's increasing capabilities to put "tenants in control" of security, auditing and compliance.  Security event information management (or SEIM) is an example of a high-level security control point.  As is IT compliance.  As is DLP. As is advanced identity management and verification.

All of these feeding into a GRC framework (such as Archer) that gives enterprise tenants far better control over their risk profile than they have today.

When it comes to multitenancy -- and delivering dynamically managed service levels from shared infrastructure -- there are some strong capabilities today, but more to do.

Generally, our friends at VMware and Cisco do well for their part.  Plenty of ability to segment and adjust workloads using the hypervisor, or converged network.  By and large, I think they've done their part.

On the storage front, perhaps the best capability in the industry can be found on Symmetrix VMAX -- which has not only the ability to dynamically manage different levels of storage media performance (think FAST), but has sophisticated controls around related resources: cache, processor and port bandwidth.  More to do here, though -- it's not as perfect as we'd like yet.

On the management front, the challenge is being able to carve off the required virtual resources needed to deliver a given service level, expose their components, put them under tenant control and let the tenant subdivide those resources as need be.

You can see a promising start to this in the next version of EMC Ionix UIM, but it's got a ways to go before it delivers on the potential.  Related efforts are underway at VMware as well.

Again, don't let anyone tell you that all the pieces are completely in place today, but -- as far as I can tell -- there's enough to get started with, and more on the way.

The Bottom Line

It's an embryonic discussion within the industry, so your comments and thoughts are appreciated.

One example of this happened at EMC World where @texiwill, @basraayman and @storagenerve joined me for a two hour discussion on these very topics. 

Add that to about a dozen others I've had, and some consensus is starting to slowly emerge.

What do you think?

Comments

[Mikemdunn]

Chuck-
You claim to be able to argue the capabilities or perhaps the lack of, in your opinion, of NetApp SMT, but at no point do you make an argument. You just state that you don't think NetApp SMT is all that secure and reference @texiwill.

How can you possibly argue, "a) the security, audit and compliance capabilities should -- at a minimum -- meet the defined requirements of the tenant."? In your own words, it's the requirements of the tenant. You can't argue what the tenants requirements are. You simply don't know what the requirements are.

I would love to read or hear real arguments around this, but you just don't make any here. And there is barely a mention of what EMC is doing to capture on this opportunity.

I hope you follow this post up with another that has details, arguments and specifics.

Thanks,

Mike Dunn

[Chuck Hollis]

Hi Mike

Well, let me put it this way.

If I were to market a solution as "secure multitenancy", I'd be darn sure it was (a) secure, and (b) multitenant.

Not everyone needs security. Not everyone needs multitenancy. But for those that do, I would argue that whatever is proposed should be fit for purpose. Truth in advertising, don't you know?

Having worked extensively with many corporate IT groups, I know that the proposed solution would be a non-starter for many pragmatic use cases. Again, as I point out in the blog post, everyone's needs are different.

Had the solution been marketed as "simple multitenancy", I'd have no argument whatsoever, since that is what it does.

Thanks for the comment

-- Chuck

[Edward L. Haletky (@texiwill)]

There are 3 Administrators within an Multi Tenant solution. The application Admin, The Cloud Admin of the Tenant, and the Provider Administrator.

The Application Admin is concerned only with their application(s), the Cloud Admin of the Tenant is concerned with all the virtual bits that make up the cloud environment they are renting. This could be Software, Platform, or more likely Infrastructure. Which means this Admin lives within the tenant as well as within the provider space. They can often see everything within the tenants environment.

The last administrator is the Provider Administrator and he can see ALL data within the cloud environment. I do not care if it is written to an encrypted disk or not, if the tenant is running within a virtualization host at all, this administrator can if they want to, see/manipulate/destroy all data as it is being written to the disk, all data that is encrypted within a virtual machine, all data as it goes over the network, and all in memory data of a virtual machine. In effect, they can see, manipulate, and by their very nature ACCESS all data in motion and most if not all at rest. If you question this capability please listen to the Forensics discussion on The Virtualization Security Podcast (http://www.virtualizationpractice.com/blog/?page_id=4852).

So my question is how do we protect against this administrator accidentally deleting, accessing, and manipulating tenant data without the tenant knowing about it. In other words Confidentiality and Integrity.

Availability is well covered, but Integrity and Confidentiality is ONLY covered once it hits the disk, not before, not when the data is in motion.

And since the Cloud Admin of the Tenant also lives half in the space of the Provider Admin, how do you protect this user from seeing other Tenants data?

Yes, I know that there are tools such as HyTrust, vCenter, and others that try to protect against this but at the moment they can all be bypassed one way or another. What controls are in place or required to prevent this? What auditing is required?

While Chuck mentioned nearly all the EMC/RSA/VMware products, my thought is that they are inadequate without a good base to build upon and a proper 'secure' architecture that accounts for the most secure of tenants and allows those not interested in all that security to pick and choose what maps to their existing policy. I have yet to see such an architecture. I see a start at handling Availability and security of data at rest, and some guidance from CloudAudit.org and the Cloud Security Alliance.

When you enter the cloud today, you really need to ask yourself "Do you trust the Cloud Administrator with your customer and employees PII?" I certainly would not.

[Mikemdunn]

@Chuck Thanks for the response, but you still don't make an argument or case for why you believe it to be un-secure or able to support many tenants. SMT may very well be a non-starter for many pragmatic use cases, but then again you could say that about any product, including your own.

Several times in this piece you say, "I could argue." I am asking for this argument. You don't provide any details. You said you could pick apart the "multitenancy" aspect, but you don't. And you just said that you wouldn't have had a problem if it was marketed as "simple multitenancy" just not secure... so are you in agreement with NetApp that this does provide a multitenant environment?

Where's the beef, Chuck?

[Chuck Hollis]

Hi Mikedunn

Since you're a NetApp reseller, I don't know if you sincerely don't understand, or whether you're just toeing the partner line.

I did my best to explain the primary concerns: (a) there is no protection for the tenant's data from the service provider -- hence not really secure, and (b) there is no mechanism to isolate workloads on the shared array proposed -- hence not really multitenant.

For many enterprise workloads, being unable to protect sensitive data, or guarantee a given storage performance level -- well, that makes it a non-starter.

If I sold someone a data recovery solution that "sort of" did data recovery, it wouldn't be very good, would it? If I sold someone a business continuity solution that "sort of" protected against disasters, the same thought would apply.

And if I sold you a "secure" solutions that wasn't really secure? And a "multitenant" solution that wasn't really multitenant?

See @texiwill's comments above -- basically saying the same thing, only with more words :-)

Sorry I can't be more clear for you ...

-- Chuck

[Mikemdunn]

Hi Chuck Hollis
Hardly toeing the partner line... And I sincerely don't understand what your argument is, since you have yet to make one. I am asking that if you are going to write a blog blasting a product, provide some explanation why. And if you are going to say you could do a "take-down" of a product, then provide some detail as to how. You haven't done any of these. You claim to have knowledge of a product, but don't provide any details of this knowledge. What you said, and what Edward said, aren't even "basically" close to the same thing.

Again, I will reference your earlier comment that you wouldn't have a problem if NetApp marketed SMT as "simple multitenancy." So, again, I will ask, do you take issue with the capability of SMT to deliver multitenancy or not? If you do, please explain why.

[Edward L. Haletky (@texiwill)]

Actually Mike, what I stated is the overall problem with all current SMT solutions. They are just not secure with 'data in motion'. If the 'data is only protected at rest', what happens once I read that data, that data NOW is in motion through the virtual environment and therefore subject to being read, manipulated, destroyed in transit by the cloud provider's administrator, the Tenant's Cloud Administrator, or even the application administrator (depending on the application).

The problem with many storage folks thinking about SMT, is that they only concentrate on the storage aspect of data at rest. In a true Multi-Tenant Cloud, data is hardly ever at rest and as such is subject to attack as it makes its way through the layers from disk to the application.

You may argue this is a 'hard' nut to crack, but unfortunately, this is not true. The most simple of security suggestions are ignored wholesale by current virtualization administrators, and therefore their environments are seriously under-protected. I speak of isolating the management of the 'environment' from other networks. I know of at least one Cloud company that has zero isolation between this all important network and its subsequent components making it a nice juicy target.

Just protecting at the storage layer, DOES not imply SMT. It is only one small part of the entire picture.

More food for thought, how do you protect the data in motion from 'escape the VM' and other side channel style attacks as well? Not sure you can at the moment either. You can only mitigate the possibilities of such attacks, but if you do at least some of those mitigations you WILL drastically affect performance (VDI) solutions.

As I stated earlier, availability is well understood and fairly well solved, but Integrity and Confidentiality has yet to be.

[Mikemdunn]

Thanks Edward for providing some additional clarity.

Could the problem be more related to virtualization as whole? And the difficulties in securing VM's? I forget what the exact stats are that Gartner estimates, but something along the lines of 70-80% of virtualization projects are done in a less than secure manner. Why is this?

As you point out companies like Hytrust are popping up to solve this to a certain degree, but I think until virtualization admins and corporate IT take virtualization security seriously, it isn't a top priority for them currently. That's not to say that it shouldn't be or that security isn't important, as it obviously is, but I think as whole the security of VM's and virtualization is not well understood.

You may disagree, but I think there is always going to be some risk involved in regards to protection from the provider admin. I don't know that this risk will ever go away regardless of advances in technology.

Chuck, would still like to hear from you on some of my questions.

Thanks,

Mike Dunn

[Edward L. Haletky (@texiwill)]

Hello,

You are correct, it is a continual issue within any virtualized environment, however there will soon be technology that makes it much harder for a Administrator to damage the system, whether or not it will be used within a traditional data center is up for grabs. The why it is this way is more historical than anything else. People feel that if they virtualize they are more secure, this is a blatant false statement but someone sold it to them, and it keeps popping up when I hear from marketing types.

But we are really talking about Secure Multi-Tenancy here, outside the traditional data center. Specifically within the Cloud whether private or public. Until the 'designs', 'architectures', and 'solutions' look at solving this problem they are frankly not Secure and I for one would hesitate before suggesting them without a bunch of attorney's reviewing all contracts, requirements, liabilities, etc.

Yes I know people use the cloud today, and I even do to a small extent but I go in with my eyes wide open, knowing full well that anything I put there is viewable by all administrators.

Any Secure Multi-Tenancy solution absolutely requires protection for the data from ALL including the administrators. How much protection is something dial-able by the Tenant not the Cloud Provider.

Where do we need to start this: with a workable definition of 'Tenant'.

[Chuck Hollis]

Edward rightly points out that it all starts with a definition of "tenant" -- and there's a reason I tend to choose that word.

Most the desired attributes and properties of "tenant" in the SMT world have strong analogs in the real world, so it's a useful starting point for describing differences, rather than similarities.

When I lease a property from a landlord (service provider), there is set of clear expectations around rights and responsibilities. The landlord provides a building in usable condition for the intended purpose. The landlord does not have any access to the property in the building. If the landlord needs access, there's a process to do that.

I, as tenant, can sub-let the property if allowed, can use it for a wide variety of purposes without disclosure and/or approval, have certain protections from other tenants as well as the landlord, and more.

The physical model has strong corollaries in the idealized SMT world.

So, join in with me here, people. If we're going to define "tenant" in this cloudy world, should we start with a workable real-world model as a starting point?

-- Chuck

[Edward L. Haletky (@texiwill)]

More food for thought: Defining Tenants for Secure Multi-Tenancy for the Cloud -> http://www.virtualizationpractice.com/blog/?p=5793

[Mikemdunn]

Either of you happen to be in Chicago this week? Looks like a pretty good panel discussing some of what we have been going at on here.

I am going to try and submit some of our questions from here to the moderator. Feel free to add your own ideas and I will submit all of them and see what these folks think.

http://www.itm.iit.edu/data/CloudComputingMay3rd2010.pdf

Mike Dunn

[Chuck Hollis]

Nope, not traveling much these days. I'll be interested to see how these -- and other topics -- are handled. My suspicion is that -- generally speaking -- it's early days on the discussion.

Thanks for the offer, though ...

-- Chuck

[Edward L. Haletky (@texiwill)]

Neither will I be. :} I will be at vForum in Boston however.