The Dark Side Of Clouds

In my travels, I'm starting to see a disturbing trend.  I don't like it one bit.

The setup is easy.  Business is demanding more from IT.  IT wants to respond by being able to deliver cloud-like services internally.  A group of smart people assemble to try and tackle the challenge.

And then it all goes horribly, horribly wrong.  

The assembled team decides to invest many months and piles of money to go hand-craft their own "cloud" out of commodity components.  Their management team, for whatever reason, allows them to go do it.  Based on experiences so far, almost no one seems to be able to reach the finish line with a satisfactory result.

The first time I saw this at a large financial institution, I thought it was an anomoly.  I've now seen this for the 7th or 8th time, so there's a pattern forming.  And it's grim.

I've been giving some serious thought as to why some IT groups decide to go down this particular road, and others take a more pragmatic route. 

And, to be honest, it's not a pretty picture.

Meet The Cloud Building Team

A classic example was about six weeks ago -- I had a few hours with the "cloud team" at a relatively large financial services institution.  They had started to assemble a home-grown cloud environment, and they wanted to know what commodity-like storage bits and pieces EMC might have that could help.

Not a good starting position, but I was sort of curious as to why they were going down their chosen path.

My first question -- why are you folks hand-crafting a commodity cloud vs. simply buying a pre-integrated and supported solution like a Vblock?  

Obviously, to save money they replied.  They felt that by using open source and commodity components, they had the potential of saving their company a "boatload" in vendor costs.  Amazon had done it, Google had done, why couldn't they?

My second question -- how was it going?  

A bit of a pause before the response.  It was taking longer than they had originally promised their management, and they were facing more challenges than they had originally anticipated.  Nothing that more time and more resources couldn't fix.

My third question -- well, how far had they gotten?  

After six months, they had a modest environment of a few hundred virtual machines up and running, and it was sort of working on good days.  They'd be unable to convince too many people to use the new environment -- something about stability problems, application compatibility issues, a less-than-optimal user experience, and so on.  Nothing that couldn't be sorted out.

The big question -- how much money had they put into it so far?  

Not much, they'd reckoned -- some commodity servers, storage, networks, etc.  

Hold on -- what about all the time your team had spent so far?  Six months -- how many people?  Long pause.

And what about the opportunity cost?  Wasn't the team six months in with nothing really usable to show so far?  And no firm estimate as to how much more it was going to cost, or how long it was going to take, or even if the results would be usable?  A much longer pause ensued.  

I was calling their baby ugly.  You could feel the tensions start to rise.

I thought for a moment as to whether or not I should bring up their perceptions of ongoing support costs associated with their pet project if it ever made it into a production environment.  

I wisely decided to hold that thought, and pursue a different course.

Perceptions About The Alternative

OK, I said, you guys probably know about what VCE is doing with Vblocks, and what HP is doing, and so on.  Why didn't you go down that sort of route?

The first objection was predictable: it costs too much.  Really?  How did the cost-to-serve numbers stack up?  Well, they really hadn't done a detailed comparison, but they just felt that by going down the open source and commodity component route, they'd end up saving money.  

After all, that's what the big cloud boys did -- Facebook, Twitter, et. al.

The second objection was also predictable: they wanted to avoid vendor lock in.  They felt that if they started with something like a Vblock, they'd be forced to stay with something like a Vblock, especially if they wanted to introduce another flavor of storage, server, network, hypervisor, etc.  They felt that utter flexibility to mix and match components was essential.

That's interesting, I thought.  How did they feel about their mainframe environment?  Their corporate network?  Their email and database environment?  Their purchased applications?  Weren't those largely vendor-specific enterprise-class environments?  

Yes, the claimed, and that was precisely what they wanted to avoid this time around.  

I kind of sat back and thought about what they were saying -- I was quiet for a while.  I wasn't quite sure where to take the discussion, really.  I needed time to regroup.

One of the senior guys spoke up after a bit, maybe being a bit too honest.  

Look, he said, there are going to be a lot of financial institutions that are going to want to run on commodity clouds in the future.  If they could figure out how to make it work here, there'd be strong demand for that skill set and expertise elsewhere.

Wow.  I really had to think about that one.  I was largely dumbstruck -- ultimately, this wasn't necessarily about doing what was best for the business.  Now it seemed that part of the agenda was about helping the participants create a marketable set of in-demand skills.  Wow.

What Could I Say?

There comes a time in some customer discussions where you assess the situation, and the logical course of action is to beat a hasty retreat.  This particular conversation wasn't going anywhere fast.

They had ostensibly brought EMC in to talk about some of our specific storage products that might work in their environment.  They were curious about what Atmos was all about.  They wanted to know what they upper-end of the Iomega range could do for them, and they had a passing interest in low-cost, high-density and high-bandwidth modular storage arrays. I told them what I knew.

One of the guys in the room kept interrupting me and wanted to know what the effective price per usable gigabyte was for each of the products I was talking about.  I guess I frustrated him, because I just didn't know what the current answer was -- the prices drop routinely as the components change.  

After about 15 minutes of that, I did come up with one last proposition to try and salvage the original thread.

The Proposition

Look, your team has been chartered to deliver a cloud-like environment to IT so they can start delivering variable IT services back to the business.  You're under the gun to deliver something usable, and soon.  And you've convinced yourself that the right path is to roll your own.

Since there's a lot at stake, why don't you understand your competition?  

Bring in a small Vblock, see what it can do and use it as a benchmark to compare your internal efforts.  Compare costs, compare functionality, compare stabilty, compare integration, compare compatibility, compare support, etc.

If you can do better, faster and cheaper with your own home grown efforts, great.  You'll have accomplished something substantial. But at least you'll know what competitive alternatives can do -- you'll have the facts.  And if you can't do better on your own, you'll know that sooner than later.

They politely thanked me for my suggestion.  I knew that they'd never consider it seriously.

Later Reflections

As I was walking out, I was thinking about all the Vblocks that went in shortly after the Big Homegrown Cloud Project had visibly failed.  I could have shared that with the team, but I don't think it would have mattered.

I could have made the case that Amazon, Google, Facebook, Twitter, et. al. have a very different business model (not to mention application profile!) than large financial institutions.  I don't think that would have mattered either.

I could have made a case that changing how IT operates -- and how it interacts with the business -- is the *real* heavy lifting around clouds and IT-as-a-service.  And, instead, they were spending all their precious resources on re-inventing the wheel.  I don't think that one would have worked.

I could have made a case that getting something workable and usable on the floor probably mattered more to the business people who were paying the bills than their elongated and uncertain approach.  I'd be just talking to the wind on that one ...

The sales rep was looking for some ideas on what to do next at the account.

The first thing that came to my mind was simple: go find some grown-ups, and tell them what's going on.  At some point, a legitimate business requirement had obviously turned into a self-satisfiying science project for the participants.  Their cloud project wasn't going well.  However, their defense and rationalization mechanisms were working quite well indeed.

Based on my experience, the prognosis was not good.  The sooner someone who was paying the bills realized it, the better.  

And, no, you weren't going to earn any brownie points with the "cloud team" in the process.

The bottom line?  Had they chosen a different path, they would have been using their cloud in production for several months.  They'd be moving on to the next challenge.  Or, they could look at what VCE had done, and make an informed case to do better themselves.

Instead, it looked like all they had a pile of assembled components, and nothing really useful to show other than some interesting experiences and some more stuff to put on their resume.

Sure, there are cases I've seen where -- yes -- it could make sense to invest in a big team and a big project to create a hand-crafted cloud.  But that's the rare exception, and only after a sober analysis of the costs and benefits of doing so.

The first few times I saw this, it was unusual.  But it seems to be happening more.

And I think that's not too good for anyone.

Comments

[Martin G]

The biggest mistake is believing that you need a large team to make this happen. You don't; you can start small and with commodity; you can build a cloud to support testing environments which can be brought up quickly and easily for example. You can learn a lot from this approach; you may still decide to go down the VCE route but you won't waste a huge amount of time either. This is not rocket science!

You can make this as complicated as you like I guess but as soon as it gets complicated; you've probably gone wrong. My advice for most people, start small and don't try to boil the ocean.

If you or your teams can't build a commodity cloud in less than month; you've employed the wrong people. Harsh, I know!

[Chuck Hollis]

Hi Martin

I do agree -- quick, low-cost "experiments" have a role in any new approach to technology, and that includes cloud-building. If you can afford to have something fail, you're in good territory. And there's lots to be learned about the second-order challenges in the process.

But that's not what I'm seeing. We're talking big projects with big budgets and big outcomes at stake.

And that's really scary when you see it.

-- Chuck

[Martin G]

That's because people are being told 'We must have Cloud...*NOW*' often; these might be either external or internal pressures. How many people go into these big projects with little idea of what the outcome should be? Just wanting to build a Cloud; that's not an outcome...that's playing with technology for technology's sake.

There's a lot of confused people wandering around technology departments at the moment; flustered and flapping, threatened by Cloud but not knowing why. Which is pretty much a perfect scenario for vendors...

[Mike Foley]

I suppose if I was told I had to drive from here to the West Coast for an important business meeting I could buy a bunch of parts and put together a car that beats a Honda. Or, I could just go buy or rent a Honda and get on with the trip. (which is what my boss asked me to do!)

I'm hearing similar stories. I ask "How frequently do you build cars?"

[Jo Maitland]

Chuck,

Interesting post. I imagine you have considered that when you have these conversations with customers, they:

a) keep both hands firmly in their pockets holding on to their wallet
b) are not under a big time crunch to build a cloud NOW
c) do not get opportunity cost(IT dept is rarely made aware of how integral they are or could be to the business as the business side doesn't trust them and or has low expectations of them)
d) think your product automates away their job
e) keep both hands firmly in their pockets holding on to their wallet.

That's my experience in talking with your customers.

Jo

[jamal k]

@Jo, you forgot f) are petulant assholes with an axe to grind.

[Wesley W]

Oh the anti-engineer tone of this post and some of the comments is disturbing. Lets turn this negativity around: You want to sell VBlock and the biggest obstacle is the technical staff. So you make a blog post from your perspective as a manager/sales person: technical folks are basically self-serving Dr. Frankensteins and if only you could just bypass them and get a business person to pull the trigger on a VBlock sale...

I know that isn't true. Look, Cloud is a cluster*&#! right now. I'm not even sure what it means anymore. I'm being sold by Cisco network virtualization solutions "that are not VLAN based" because VLANs are not secure. My security folks and our auditors tell us VLANs are not secure. But VCE produces a secure multi-tenant document that says VLANs *are* secure. The "C" in VCE stands for Cisco, no?

And yes, Vendor lock-in is an issue. Its clear VMWare and Cisco want to rule the world. However in recent years we've discovered that other vendors have competing products that are sometimes better. We can negotiate prices down with vendors by being multi-vendor. The results speak for themselves. VMWare and Cisco loathe this greatly and engage in shady sales tactics to get their way. They hijack standards bodies and they refuse to integrate with existing virtualized platforms. Just ask the Service-Providers. The IETF is just now coming around to addressing some of the shortcomings of VM/network integration.

So why should I dole out a pile of cash? Oppurtunity cost? An intangible and largely made-up percentage that I can't quantify that passes me by with 40 other percentages on someone's sales deck?

We're profitable now. Our customers are happy. I think I'd rather wait for the dust to settle. In the meantime, we can address the shortcomings with enhancements to our existing infrastructure/product portfolio using commodity and open source solutions.

[Chuck Hollis]

Hi Wesley -- thanks for the thoughts.

My intent was not to be anti-engineer, or to endorse any one vendor's approach over another.

The decision to invest in a robust engineering function to create a home-grown environment is an important decision. Since there are now alternatives, those must be considered now?

You (and your co-workers) are not free. Nor will be the technical folks who end up supporting whatever it is you create.

Opportunity cost is difficult to measure at the IT level -- that's a business discussion. However, the ability for IT professionals to coldly tick off bounded alternatives for achieving said business goal (pros *and* cons) is useful, and -- it seems -- very rare indeed.

Your vendor rant is understandable. Vendors are vendors. Engineers are engineers. The interests between the two groups are not always aligned.

You didn't mention anything about your organization and what it does -- enterprise IT, service provider, something else. It'd be easier to have a more nuanced discussion with specific context.

Thanks for writing

-- Chuck

[DMR]

This project is suffering from a lack of executive leadership/sponorship. Have seen this many times in the past, as soon as the bravado of open source and commodity HW appears, you can bet with almost complete certainty that there is little connection between the science project and the business itself. The reality that this teams fails to realize is that if this is truly a best use of the firm's scarce resources, then every day the project is not funcitonal represents opportunity costs to the owners of the business. It is very frustrating when you see a train getting ready to go off a cliff and all you hear is that the costs are low and that there is no vendor lock. Really? Is that this project's value to the firm or is the value better, more agile, optimmally aligned I/T services. I wonder which is more important to the executive suite?

[CatchDaRayz]

Not so long ago, rich people bought cars on the basis of how they looked, how they enhanced their prestige and then hired a chauffeur to clean, maintain and drive it. You would have never tried to sell a horseless carriage to the chauffeur.

Taken another way, the majority of folks in IT are there for the technology. Had they been more interested in business they would have gone down that path and probably be earning more money as well.

With outsourcing and solution oriented vendors, the IT folks feel threatened. To the management they are a fungible resource which leads to job hopping and with that their primary investment will be in their technical proficiency.

Judging how the motor vehicles all but drive themselves to a voice given destination, the best technology focused folks will either end up working for a vendor or outsourcer or go the way of the horse buggy whip makers.