Thoughts on Secure Multi-Tenancy

All of the sudden, everyone at EMC was asking me about this topic.  Where the heck did this come from?

Doing a bit of digging, I figured out why -- one of our competitors has started promoting a solution in this space.  Lovely.  I dutifully found the source documents, and realized that there was a good deal more to this discussion than white papers and marketing videos I was able to find.

So, without too much competitive rancor, please join me in a quick discussion of secure multi-tenancy, because it shows all signs of being an increasingly popular topic over time.

The Basics

At first blush, the concepts of "secure" and "multi-tenancy" appear to be polar opposites.

Security is inherently about isolation: keeping your information, applications and resources  isolated from others.  It's also about clear separation of administration roles as well.

Multi-tenancy, on the other hand, is all about sharing: especially when it comes to sharing physical resources (compute, network, storage), but, ultimately, operational roles as well -- fewer people doing more.

Why Is This Important?

Service providers and cloud operators depend on high degrees of multi-tenancy to make their value propositions work from an economic perspective.  You can't be cheaper than traditional IT unless you're aggressively sharing resources and operational roles.  That's what any cloud is about.

Most customers, on the other hand, are concerned about security and isolation.  They look at these shared resource models, and justifiably have hard questions.  Indeed, addressing this fundamental issue -- at something more than a “security-washing” level -- is essential for enterprise-oriented external clouds to be something more than a novelty.

I don't believe the problem is insolvable: for example, secure multi-tenancy has been the norm in data networks for many years.  Multiple networking tenants can share the same physical network infrastructure, and there's little concern around security.

But we're not in the same situation when it comes to shared IT infrastructure.  I've picked out only three issues among many, simply because they illustrate the real challenge here.

Issue #1 -- Preventing Service Provider Access To Your Information

If you're renting shared resources from a service provider, you want a very clear understanding as to what things the server provider can do, and what things they are prevented from doing.

In a storage centric discussion, this can boil down to a simple question: can the service provider get access to your data without your permission?  I'm not talking policy and procedure, I'm talking simple physical access.  And we're not just talking protecting primary storage here -- we're talking snaps, backups, archives -- the whole enchilada.

If you've ever been in a hosting facility, you may have seen "locked cages" that prevent physical access to equipment.  In a multi-tenancy environment, we need a virtual equivalent to that concept.

The most complete answer, in many people's minds, is strong encryption complemented by tenant-owned key management.  In a virtualized environment, this means that encryption can be done granularly on a per-VM basis, with key management owned by the tenant and not the service provider.

If you think that I'm being too stringent here, keep in mind that in many proposed solutions, data from multiple tenants is freely intermixed on available storage devices.  Sure, logical access is mediated, but it's all living on the same rotating physical rust.

Now, if I'm imagining virtually anything, let's imagine a disk appears to be going bad, and needs to be replaced.  Your sensitive data may be somewhere on that disk, in plain view.   Can you even tell which drives might hold some of your data?

In EMC land, that's one of the reasons we created a Secure Erase function that doesn't allow disk drive removal without erasure, as well as using RSA technology to provide role-specific temporary administrative credentials to arays.  Not to mention PowerPath-based encryption for guest images, Avamar-encryption and RSA’s key manager.

Because drives do fail, equipment gets upgraded, and technicians occasionally do have to work on the equipment.

Issue #2 -- Separation Of Church and State

The most secure environment in the world is useless unless there's an administrative model that provides separation of roles and responsibilities.  You don't want everyone running around with the "root" or "admin" password, do you?  Or granting the “cloud administrators” omnipotent privileges?

Even in classical IT environments, this definition and enforcement of role separation and delegation is a really big deal.  Taking that a step further, if we introduce multi-tenancy into the equation, the tenant should have full control of the administrative model for the resources they control.

No back doors, and no access -- unless granted by the tenant.  Imagine you had a lockbox down at the local bank.  You'd assume that you would be the only one with access, wouldn't you?  Same for security in mulit-tenant environments.

Put differently, I believe that this core delegation of roles and responsibilities lies at the heart of making secure multi-tenancy something more than a vague marketing concept.

No matter what you do, it ain't secure unless you can manage it in a secure way.

Issue #3 -- You're Not Secure If You Can't Prove It

Early on, I realized that all the security stuff in the world was relatively useless unless you could prove it to an outside auditor.  Secure multi-tenancy is no exception.

If you're an enterprise IT user, and you're using an external service provider, you'll probably have to pass the same sort of audit as you would for any infrastructure you owned.  

That means, among other things, you'll have to provide audit and change control logs (including potentially those from your service provider!) showing that controls are in place and are effective.  And, of course, it wouldn't do to have the audit logs on the same multi-tenancy infrastructure, would it?

Any viable 'secure multi-tenancy solution" in my book would have to have infrastructure audit logs available to all tenants at a minimum.  This level of transparency and accountability isn't the norm in most SP models, but will likely be before too long.

Issues #4, #5, #6 ...

There's more, but I think I've begun to make my point.   I haven't even begun to delve into issues like denial of service, or certifying IT infrastructure compliance, or a whole bird's nest of related topics.  And that’s just poking at the “secure” part of multi-tenancy.  Even within simple multi-tenancy (forget security for a moment), there’s a lot still yet to be said.

Here's the message: if this topic is important to you, there's far more to this secure multi-tenancy thing than integrating a bit of networking technology and emulating a logical array.  There's data, for one thing.  And administrative models.  And being able to credibly pass an audit.

Now, back to our competitor's proposed solution -- a somewhat-secure-but-not-really-all-that-secure approach.  Does it solve a useful customer problem for people concerned with security?  Perhaps, but I'm incredibly skeptical. 

However, this does make me think that EMC should probably get busy and focus a bit more on this topic.  Fortunately, we've got a lot of very interesting pieces that could result in a very compelling solution.

Just off the top of my head: guest OS level encryption w/key management, for example.  Ability to encrypt snaps, real backups and archive copies.  Separation of tenant data on isolated media if needed.  External configuration and event auditing with per-tenant reporting.  Automated IT compliance management with tenant reporting.  Tools that support an administrative model with clear separation of roles and responsibilities.  A mechanism for secured temporary administrative credentials.  Secure media erasure.  

And a few more potential bits as well :-)

Just to be clear, though, there are lots of enterprise use cases for service provider multi-tenancy that don't require high degrees of security.  The proposed approach might work there.  And, on the other hand, there are environments where the security requirements are so daunting it's hard to imagine them running in a generic multi-tenant environment -- ever!

By the way, this stuff ain't exactly nuclear physics, either.  Just take a look at how most large-scale data centers in sensitive industries do their business (e.g. financial services) and you'll see all that I described here, and more.

That being said, we collectively as an industry have to do a bit of legwork to bring this to service providers in a way that doesn't require dedicated, separated physical infrastructure.

We're not there yet, however.

Comments

[Vaughn Stewart]

Chuck,

You bring up some excellent points in this post around enabling secure solutions for cloud providers. Restricting access to content, Delegated Management, Verified solutions are critical components required to make secure multi-tenant solutions a reality in the marketplace.

However, these technologies and operational practices are meaningless without the infrastructure to deploy on. It is on this point that I find your closing comments rather interesting.

Maybe it's just me, but the last lines of this post change the topic from one of requirements for secure infrastructures to one of dismissing technologies and a solution set that EMC doesn't offer.

In particular your backhanded compliment to the Cisco-NetApp-VMware SMT solution. Lines like, "Back to our competitor's proposed solution -- a somewhat-secure-but-not-really-all-that-secure approach."

May I suggest that you might have been better served by striking said line and just delivering the message that, "EMC should probably get busy and focus a bit more on this topic."

[Mike Riley]

"...a somewhat-secure-but-not-really-all-that-secure approach."

How so? Something amiss in the joint whitepaper (http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/Virtualization/securecldg.html) from Cisco, VMware, and NetApp?

"Does it solve a useful customer problem for people concerned with security? Perhaps, but I'm incredibly skeptical."

I'm not sure why. EMC partners with Cisco and owns VMware. Have you voiced this concern with Tony Bates from Cisco or Paul Maritz from VMware? Were they not credible resources to comment on this solution? Is there something they missed or are missing? Probably not a fair question for either this blog or the 30 minute event today since there are, as you mentioned, a myriad of details one needs to consider as part of this solution. Still, I'm not sure how, with the companies agreeing to design, implementation and support and current customer adoption (Jared Wray, Tier 3 video - check it out), one can be skeptical of something that already exists.

In sum, you had 3 executives spend 30 minutes in a collaborative discussion giving you the high-level design and business drivers for the solution; you had a customer step up and say, "Yep - it works great and I'm using it today" and for more detail you had a whitepaper with low-level design considerations and a 24 hour support agreement announced.

Now, I think it's good for customers to have a healthy bit of vendor skepticism. Quite frankly, I too am a bit surprised that EMC hasn't put a lot of thought into secure multi-tenancy, but I get the sense from this blog that because EMC hasn't, readers should be skeptical? I didn't see any supporting points on why they should be.

While EMC mulls this idea over, I'd encourage readers to look at what has already been put into practice by joint Cisco, VMware, NetApp customers. As Tom Georgens put it in the video, this isn't a theory. It's already being put to good use. No "perhaps" about it.

[Chuck Hollis]

Hi Vaughn

I was hoping for some answers to the questions I posed. I didn't get any. So I guess they'll remain open questions for the time being.

As far as "solutions", I'll give you credit for the work done. That doesn't mean you've solved a useful problem yet. And we certainly wouldn't label something "secure" unless we felt it truly was. Guess that's a difference between our companies.

Best of luck with this going forward!

-- Chuck

[Chuck Hollis]

Hi Mike -- congratulations on your event today!

But when the adrenaline wears off, then what? Cisco's networking capabilities works with all devices ; VMware's security capabilities are famously agnostic, which leaves us with the inevitable "why NetApp?"

We are pointed to MultiStore, which creates a virtual array. A neat trick, but "secure"? That's where I have an issue.

I was also hoping for some credible responses for the issues I raised, but didn't get any yet. So the topic is still open.

Best of luck going forward with this!

-- Chuck

[Jonathan Donaldson]

Hi Chuck,

My team, along with teams at Cisco and VMware worked long and hard to deliver an architecture that solves the customer issues of transferring isolated app / tenant silos into a shared cloud infrastructure. We truly feel that this solution offers the security and SLA protection that these environments require.

You state in your comment :

"why NetApp?"
We are pointed to MultiStore, which creates a virtual array. A neat trick, but "secure"? That's where I have an issue."

Here is a link to an independent 3rd party security audit done on MultiStore:

http://www.netapp.com/us/library/white-papers/wp-multistore-analysis.html

We are in the late planning stages right now for a full stack 3rd party security audit of the solution.

All of that being said, it is not just MultiStore that brings strength to the solution to answer the 'why NetApp' question, it is the SLA protection with FlexShare, the efficiencies inherent in NetApp controllers, and our willingness to roll up our sleeves and get it done for the customers that answers the 'why NetApp'.

[Chuck Hollis]

Hi Jonathon (and the rest of the NetApp team)

Yes, there's some fine work here. And I'm sure you all worked long and hard.

It's all about how you frame the problem. I'm reasonably convinced that tenants are isolated from each other. But how are tenants isolated from the service provider?

I brought up three simple examples. You are the third person from NetApp to talk about many interesting things -- none of which even begin to answer the questions I raised.

This security business is not based on feeling and opinions. At some point, there's a certain rigorousness of approach that's needed, and all the "independent third party" opinions do little to address fundamental questions such as the ones I -- and many others -- will raise.

More work to do, guys ... much, much more.

-- Chuck

[Mike Riley]

Hi, Chuck,

As I mentioned, I doubt the execs were going to get into the weeds on the technical details nor was I in a 4 minute YouTube video. Details are available in the joint whitepaper (link provided previously). I'd suggest understanding the security aspects of IPspaces, RBAC, and at least reading the whitepaper before commenting.

Encryption and secure disk replacement - all handled. I would reference specific applications in the intelligence community but then I can't do that, can I?

Delegation of roles, separation of church & state - yep, the solution has that too. That's the very nature of RBAC and Multistore. Again, I'd refer you to the whitepaper and NetApp documentation. (I can see the e-mail traffic now inside EMC: "What in blazes is RBAC?"

Passing an audit - done. Not only for secure public sector/military installations but commercial applications as well. Remember, this solution is already installed and not on a whiteboard somewhere. I'd also refer you to the Multistore analysis at www.matasano.com.

The answers to your questions were already available in public documentation regarding data access, RBAC, IPSpaces, logging and auditing. All easily accessible in the same amount of time it took to pen this blog but, then, what fun would that have been, right? Kind of kills your blogging momentum when you have to go off and do a little fact checking.

Now that I've answered your questions. Would you care to answer mine (previous post)?

[Jonathan Donaldson]

Chuck,

First off, good discussion.

So if I understand your security question it boils down to "Quis custodiet ipsos custodes?"

Translation for the Latin challenged: http://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes

I would argue that if your security posture is so extreme that you need to guard your data from your trusted providers, you are probably not the type of organization that would ever allow that data to move beyond the organizational firewalls (or one that employs any type of non-vetted contractor/partner/employee). This would be an extremely niche customer, and thankfully for both NetApp's efforts and the vBlock efforts, not the norm. You are more likely to loose data from an authorized disgruntled employee than a service provider whose business model relies on customer trust.

Like any outsourcing endeavor, you should look to those trusted providers with both a good reputation, audits, and the controls to prove it. Possibly the existing SAS 70-II audit reporting could be implemented on the logging from the infrastructure. This would be a strictly forensic measure, but could over time prove that the provider's procedures are effective for the 95% of the organizations that will use this type of solution.

For those organizations that have extreme data security requirements, they will most likely implement internal private cloud environments.

Over time you will see more and more additions to the security of this solution, but remember, security one of four key concepts in the SMT, all are critically important for the solution to be realized.

[Chuck Hollis]

Hi Mike

I did download the whitepaper, and spent some time reading it. I hit the "you gotta be kidding me" meter at about 10 minutes, and stopped reading.

Just to be clear, I was addressing the marketing claim of "secure multitenancy" in presumably service provider environments (as marketed by NetApp), and not environments where shared facilities are under the control of a trusted entity, as you are suggesting.

Here's what I picked up -- please correct me if I'm wrong.

1) Encryption of virtual machines is done at the physical server level, rather than virtual machine level. Any key management is presumably under the control of the cloud administrator, and not controlled by the tenant. Again, encryption and key management were not covered in detail, so I'm inferring here.

2) The document is very clear that "best practices" is that the cloud administrator has complete and unfettered access to all resources in the domain. Tenants are not given the choice as to what roles can be played by the "cloud administrator". I can cut and paste here if you'd like.

3) I believe the white paper made no reference to procedures for restricting data intermixing on physical storage devices, nor secure erasure procedures, nor securing access to storage administration, nor securing access to service personnel. Please correct me if I'm mistaken.

4) There was no provision that I could see in the solution for providing tenant-visible audit trails of surrounding infrastructure: configurations, changes, security events and the like. Again, I am inferring from the lack of discussion in the document.

As I'll repeat again, the solution seems to do a decent job of separating logical tenants, but not protecting the tenant's interests vs. the service provider. In many people's minds, that's what "secure multi-tenancy" is all about. If you have another interpretation, we should discuss in detail.

Now, for some suggestion on style points that might help towards a more productive industry discussion:

!) The solution was promoted as "cloud", implying an external service provider who may or may not be trusted. None of your examples correspond with the use case marketed.

2) My understanding is that you passed an "evaluation", and not an audit -- based on the links provided. There's a difference, as we both know. If there is some separate audit for the solution that wasn't originally published, please share the results.

3) We too work extensively in government secured environments around the world. Ditto financial services, health care, retail, etc. The required procedures are well documented, and not a secret in themselves -- as you should well know.

4) Forgive me if I blog on what appear to be gaps in "solutions", especially when it comes to security. We can quibble over performance, availability, economics, etc. -- but security (like high availability) requires more precision and less hand-waving.

5) I don't want to get into a duel with you and the other NetApp fanbois, e.g. "you didn't answer my questions, so why should I answer yours?". We didn't publish and promote a "secure multi-tenancy solution" -- you did. So be prepared to take a little heat if it comes up short in important areas.

Mike, security is an important topic, as we both know. It deserves an open, honest and transparent discussion of the risks, threats and gaps.

No security solution is perfect. But if you're going to offer up solutions in that space, be prepared to enter into an open, honest discussion of what threats you *do* protect against, and which ones you *don't*.

Again, a good effort by the joint team. Much, much more work to do.

-- Chuck

[Chuck Hollis]

Hi Jonathon

You can't have it both ways.

Either (a) the world needs a secure multi-tenancy solution it can trust, or (b) there will be no such thing, and we can skip talking about cloud in a meaningful way for enterprise IT.

I happen to agree with (a) above. Such solutions do existing in other domains, not just in the general case we're discussing here. And, based on what was shared yesterday, there's still a lot of work to get there.

Your comments regarding disgruntled employees made me smile -- simply because service providers get them as well!

-- Chuck

[Joint tenants]

The latest cloud storage-focused alliance between major vendors takes the form of reference architecture for secure realistic server secure multitenancy drawn up by NetApp with VMware and Cisco. Obviously, shared hardware effectuation greater utilization and efficiency along with equipment, operations, and utilities cost savings. In the past, security was the wrench in the works — would Citigroup really want to host applications on the same server hardware as Bank of America? Secure multi-tenancy may not solve all the cultural and IT problems, but it goes a long way toward doing so. Large organizations will be especially pleased –I can only imagine that agent federation Vivek Kundra and the GSA will be all over this announcement.

[Ryan Michaels]

Hey Chuck and Co,

Fantastic discussion from all parties concerned, I have found the reading informative and constructive.

I think both the VCE and CVN alliances are fantastic for the market today and will be suitable for 90% of the market.

The only comment i would make is that VCE was a great announcement to the market and im sure there are customers that will adopt it and love it.

However, deciding to post on another alliance pitched at secure Multi-tenancy(something that most people feel as core to cloud services from a provider point of view) and making comment on stuff that is pure heresay cheapens the whole discussion and looks poorly on you Chuck. Those people who read industry blogs know that Mike Riley is one of the most gifted IT professionals in the market along with guys like Chad Sakac.

I love reading your blogs but you have let yourself down with this one.

[Chuck Hollis]

Hi Ryan --- you need to go re-read this discussion. At some point, you entered the Twilight Zone ...

This has never been a personal attack on Mike or anyone else. This hasn't even been a diss on the work done by NetApp per se.

Any security discussion of any value has to be open, forthright and transparent. That's what you're seeing me do.

I raised several points that I and others found almost immediately.

The response to the points should be either (a) we think we've got that covered, here's how, or (b) we don't think that vuln is going to be a problem, here's why.

You, and everyone else associated with NetApp, has failed to offer up a response in either the (a) or (b) form. Mike pointed me at the exact same documentation that raised the initial questions, for example.

So, in many people's minds, this still remains an open question. Rather than try and discredit me, why not answer the questions?

-- Chuck

[Chuck Hollis]

"Joint Tenants" -- going forward, I'm considering simply deleting commentors with obviously faked credentials.

Either make a decent attempt to identify who you are, or I'll simple press the "delete" key next time.

-- Chuck

[jagan]

Security is a concern for few and still there are many customers willing to put their data on the cloud by taking that risk. I am not saying secure multi-tenancy not providing the security , what I am saying is that, it will take time for customers to move their data however there are some customers ready at this movement to move their by taking this risk. Its all about time and importance of the data.
Definitely it is not easy to move the data which is at most important to the client.

-Jagan

[engineer]

Kudos to EMC for providing a nice discussion platform for my company
where we have a fair chance to explain 'why netapp'