Creating The Virtual Runbook

Discussions move fast in this industry, especially with customers.

Yes, server/desktop virtualization is cool.  We're in, they say.

Yes, we get that the supporting infrastructure may have to change -- servers, storage, backup, replication, management, security, etc.  We're evaluating that one.

But what about the most important part of IT -- the people and process?

A Very Subtle (But Important) Challenge

Cool technology that changes how things are done are great.  We've seen lots of examples over the last decade or so.  And certainly virtualization is one of those.

But there seems to be two related effects.  One, the supporting technologies around the new whizbang thing have to change and adapt to support it.  But more importantly, how people use the new tool has to change as well.

And sometimes, that's the long pole in the tent -- especially at scale.

The Email Example

I like to use email as an example of new technology coming into our lives that changes a lot, but in a very subtle way.

As people started using email heavily, all sorts of surrounding technology became an issue.  Servers, storage, backup, replication, management, security, etc ... the popularity of email rippled through the rest of the IT infrastructure.  Some people saw it coming, others didn't.

But the people angle was even more interesting.  I vividly remember a senior executive (not at EMC, thankfully) who scribbled a communication down on paper, gave it to his administrative assistant, and asked her to type it up for his review, and then send it via email as well as physical mail.

Hardly an effective use of the technology, right?  But old habits die hard.

Surrounding Technology Infrastructure for Virtualization

I've written before how much of the EMC technology portfolio has been targeted at advanced VMware use cases.  Why?  Server virtualization creates an abstraction that tends to impact adjacent areas.

• Storage design can be different in a VMware environment.
   
• Backup and recovery certainly is, hence the interest in EMC Avamar.
   
• People want to make local and remote copies of individual virtual machines, rather than entire VMFS images, hence the interest in RepliStor and RecoverPoint.
   
• They want to be able to have storage management that gives an accurate view of utlization and configuration in the virtual environment, hence the interest in ControlCenter.
   
• They want to be able to discover their applications and supporting assets in near-real-time -- and how they all relate -- both prior to moving to a virtualized environment, and certainly afterwards, hence the interest in Smarts ADM

They want to be able to provide predictable service levels and provide real-time root cause analysis for problems in the new environment (as well as the old one!) hence the interest in Smarts.

Even the server guys aren't immune -- there's been a significant shift in demand for the types of servers that people prefer to run VMware on -- one thing causes changes in another adjacent area.

And there's more .. but you get the idea.

Some people will consider this an up-front issue -- backup seems to be a "solve it now" problem, as well as storage design.  Others will want to be aware of the potential concern, but don't feel the need to move on it soon.  That's fine as well.

But What About Process?

Every large IT organization has some sort of run book that describes how IT runs in a day-to-day environment.

And behind that, dozens (or hundreds, or thousands) of people who run IT day-to-day and keep things humming along.

The interesting challenge is that -- in a virtualized world -- that all changes.

Think about it for a minute.  Take an arbitrary IT process, and see how it changes in the VMware environment.

Chargeback?  Think differently.  Provisioning?  Much better with VMware, but it's a new process.  Patch management?  Subtly different again.  Help desk?  Different again.  Change control? Yep, that's different too.  Problem resolution?  Performance issues?

The list goes on, and on, and on.  Sure, there are candidate answers to all of this, but -- taken collectively -- that's a lot of reengineering of people and process. 

Yes, you don't have to do all of that to see some benefit, but if you're really going to leverage what you've done, it's certainly a consideration.

Now, if you're a smaller IT organization where people where multiple hats, this is no big deal.  You get in a room, figure it out, and press on.

But if you've got multiple IT specialties and process that's been baked over a decade or longer, that's a different story altogether.

Maybe you can hammer out the process.  But what about getting everyone to stop doing things the old way, and start doing things the new way?

A Simple Example

You probably know I get to meet with lots of customers, and I really enjoy the interaction.

One guy was very proud that he'd virtualized several hundred physical server images onto VMware, and it was working really well.

I asked him "how long does it take you to provision a server?"

He answered "about six minutes".

I asked, "No, how long does it take from when someone requests a machine image that the user actually gets what they asked for?".

He looked at me and said "well, that still takes six months".

The technology had changed, the process hadn't. 

And, as a result, the business wasn't getting the full benefit of the investment they'd made.

I bet I wouldn't have to look too hard to find similar examples ...

It's No Surprise, When You Think About It ...

Server and desktop virtualization is a really big thing.  Many of us see it as changing how computing will be done in the future.  It's that significant.  There's no denying it.

But, like any Really Big Thing, there's going to be two adjacent discussions:

What are the adjacent technologies in the infrastructure that are going to need to change?

and

How will we change our people and processes to maximize the impact of our investment?

I think it'll be interesting to see how all this evolves, won't it?

Comments

[John McArthur]

Chuck, You've got this one right. Virtualization changes everything, which is why I think affordable and easy-to-use management and monitoring tools are going to be so important. Who makes sure that the virtual servers that are requested are actually used? When a virtual server is no longer used, who makes sure that it is decommissioned, and who ensures that the allocated storage is returned to the pool? When things are too easy, like writing an e-mail, sometimes we forget about the downstream impact and unintended consequences. Let's face it, some e-mails should never be written, and some virtual machines never deployed.

[Steve Chambers]

Hey Chuck,

At VMware we are starting a new extension to our vmware.com community, called VI:OPS (Virtual Infrastructure Operations).

Although it's early days, we are populating it with proven practices for Strategies, Standards, Automation, Management and Security.

Run books are a key part of this, whether automated or not, and for example we are currently uploading standard operating procedures from VMware's professional services org... see http://vpp-dev-1.vmware.com/home/docs/DOC-1024

We are actively seeking contributors who recognize that getting involved with this community will drive out better practices around virtualization.

Cheers!
Steve