My Friday Rant

Over the several years that I've been blogging, I have learned many times that most people don't appreciate a rant -- especially when it comes to one vendor criticizing another.

Now, given the extremely competitive nature of our industry, and the fact that I obviously work for a vendor, this makes for a problem -- how best to share my somewhat pointed comments about what others are doing?  And not annoy everyone in the process?

I see what everyone does, and -- given my obvious biases -- I still think I have a worthwhile perspective that might be interesting to some -- what to do about it?

So, I'm going to go with a "truth in labeling approach".  In this post, I'm going to rant a bit at others in the industry -- including a few rants that necessarily include EMC.  Forgive my crankiness ...

And if that sort of thing isn't your cup of tea, better to skip this post and go on to something else.

</begin rant>

Rant #1 -- I'm The Only One

Let's face it.  Storage arrays do lots of interesting tricks these days.  Pick a given feature, and you'll find multiple vendors doing different flavors of essentially the same thing.

To my way of looking at it, once there are multiple alternatives available, your focus should shift from "I'm the only one who does X" to "you have choices, and here's why I think my approach is better".

Let's look at our first example -- thin (or virtual) provisioning.  This popular feature allows storage administrators to hand over what looks like 1 TB to an application, but only the actual storage written is physically allocated. 

This one-time win for storage capacity savings has turned out to be very popular -- so much so that you'll find it on just about every modern storage array in the industry.

So, what completely escapes me is the logic behind pretending you're the only one who does thin provisioning?  Or, offer a "guarantee" that you can reduce physical capacity 50% by using this approach?  I am puzzled.

The same thing applies to primary storage deduplication.  NetApp came out with it as a free feature first; EMC followed suit a while later with the Celerra. 

Now that there's two flavors -- both from respected industry leaders, both available for free -- I would fully expect the discussion to turn to the relative pros and cons of each approach.

And, when you do that, guess what you find? 

Each approach has its relative strengths and weaknesses -- no one really "wins", it's all dependent on your particular use case.  Both are free.  Both are seamless.  EMC does a lot of stuff with Celerra dedupe that NetApp doesn't do.  I presume there are a few neat bits of the NetApp approach we could learn from as well.  Choice is good.

Anyone who's heard a pitch from more than one vendor would know that.  Again -- respect your customers' intelligence, I'd offer.  Acknowledge that there are alternatives, and make your best argument for superiority for the customer's use case!

A third area is emerging in automatic data tiering.  This important capability (called FAST at EMC) is helping customers dynamically exploit the benefits of mixing storage types: flash, FC, SATA, etc.  Right now, we think we're pretty much the only top-tier vendor delivering this capability.  But as others start to do similar things, we should shift the discussion to the relative merits of our approach vs. alternatives.

And, of course, why make it about one feature or another?  Why not get a storage platform that does *all* of the cool stuff?  It's kind of like arguing the relatively superiority of hammers vs. pliers when what you really want is a toolbox.

Sigh.

Rant #2 -- If It Ain't Easy, Don't Bother

Sure, new IT capabilities would be great if all we had to do is drop a bit of magic in, do so virtually risk free, do nothing to change our processes, and start getting the benefit.

And, to be absolutely fair, there are a bunch of products from EMC and others that do this to various degrees and are marketed as such: DataDomain backup, Celerra's FMA, EMC FAST, and so on.  All notable in that customer implementation is usually quick and easy; benefits relatively immediate. 

I've started to call these sorts of propositions "magic pills" since, compared to many other IT initiatives, they're relatively quick and effective.  And the more goodness that we can deliver to users in this sugar-coated manner, the better.

But, let's face it -- not everything in life is easy, and not every IT problem can be fixed with a magic pill.  There are some in the vendor community that roundly criticize any vendor proposal that might take some work -- especially in the area of process change.  They tend to inherently discredit any approach that involves changing the way you do things -- regardless of the benefits of doing so.

Huh.  If we decompose IT into technology and process, one could successfully argue that we should be continually updating not only our technology, but our processes as well.  And we as vendors should work to be brutally honest as to the relative pros and cons between quick palliative fixes and more substantive heavy lifting -- and let the customer choose, not the vendor.

Not everything in life is easy.  If there was a magic weight loss pill that worked and didn't cause ugly side effects, I'd be eating and drinking a whole lot more :-)

Rant #3 -- If We Don't Have It, You Don't Need It -- Even If It Hurts You

Of course, if you're a vendor, and you have gaps in your portfolio, you try and steer the conversation away from what you don't have, and towards what you do have.  I get that. 

But sometimes that sort of thinking can put a customer at risk -- and that's where I start to rant.

There are lots of descriptive terms in our industry where interpretation is subjective.  And there are a few that are not.  "Secure", I would argue, is one of those concepts where a degree of precision matters.  And when someone markets something as a "secure solution", I tend to apply higher standards than if they were emphasizing "fast" or "cost-effective" or something else.

I don't want to name names, but -- given our RSA DNA -- we're seeing some "secure solutions" out there that deserve a far richer discussion around exactly what *is* secure and -- more importantly -- what *is not* secured by their approach.  Put differently, security and creative license is a potentially dangerous mix.

I'm not a fan of green-washing, cloud-washing or secure-washing.

A more concerning topic is the subject of backups -- another sensitive topic with me these days.

I'm starting to see a vendor meme out there that goes along the lines of "you don't really need backups, just use local snaps on our array".  This scares me, as it should you.  Now, anyone who's been in this industry more than a few months realizes that (1) hardware can occasionally fail, (2) software can occasionally fail, and (3) the humans that touch both can occasionally fail.

This is why we make separate logical and physical copies -- as separate from the source as we can justify.  Separate from primary storage volumes, sorta OK.  Separate from primary storage array, better.  Separate from the primary storage array's physical location, even better.  Multiple separate copies at a distance -- best of all.  And so on.

Now, there's a very rich discussion around (a) how much protection is needed/justified, and (b) the best way to implement the model you've chosen, but to position local snaps as a substitute for a robust backup without full disclosure strikes me as near-reckless.

Well, boys and girls, the worst *does* happen occasionally.  You've been warned.

More Rants?

Nope, those were the big ones for this week.  Thanks for putting up with me on this.

</end rant>

I'll make you an offer, though .. if you've got your own related rant you need to get off your chest (and can do so in a reasonably polite manner), go ahead and leave a comment!

Comments

[Chuck Hollis]

I guess there's a meme floating around on some of this -- see Stephen Foskett's take here:

http://developer.nirvanix.com/blogs/strategies/archive/2010/01/22/mr-backup-is-right-cloud-replication-is-not-backup-but-backup-is.aspx?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+EnterpriseStorageStrategies+%28Enterprise+Storage+Strategies%29

[Preston de Guise]

Hi Chuck,

I think most of the issues presented are vendor neutral, and more indicative of individuals (regardless of sales or engineering) who get so wrapped up in the company they work for that they can't see the forest for the trees. It's usually something I see for instance in:

- Dedicated hardware folks talking about not needing software in a solution
- Dedicated software folks talking about not needing hardware in a solution
- Inexperienced sales folks who think they can massage the products they understand into the solution the customer needs

I agree, in each case, they're annoying as hell and only serve to dilute the customer experience. Thanks for ranting about it!

[ToniArthur]

When I worked as a consultant/engineer at a VAR, I was always asked which product was better by sales people, customers, and co-workers. My answer has always been to ask what the goal is.

Most people don't seem to see storage as a tool, they see it as a solution. To me it's a tool whose utility is entirely dependent on what you want to achieve. You don't use a saw to pound in a nail, you don't use a hammer to drive in a bolt (well, most of the time, anyway).

And, as you pointed out, seeing a feature list with buzzwords doesn't always tell you what the actual capabilities are, someone has to dig through the marketing and dive into the technical parts to figure out what the advantages and disadvantages are of both approaches.

A place I used to hang out on USENET had the philosophy that "all hardware sucks, all software sucks", and in a lot of ways that's true. It's finding which works best (or sucks less, as it were) in your environment that makes the difference. And a lot of people don't understand that.

[Chuck Hollis]

Preston and ToniArthur -- well put!

-- Chuck

[Keith Aasen]

Disclosure - I'm a NetApp Employee

Chuck,
I understand your need for a rant, I often come away from watching certain vendors present and feel the need to rant. Thank you acknowledging (sort of) that this happens with EMC as well but then you seemed to take a rather pointed rant at NetApp. I do have a couple of questions.

You say that EMC followed suit with primary dedupe shortly after NetApp. This might be a failure on whomever caused you to rant to clarify what NetApp was deduping. As far as I am aware EMC does not have a primary storage deduplication solution for VMware or block solutions. Am I wrong? I know you keep saying stay tuned and It's coming but is it here and now?

Second, again this might be due to the material or the presenter you saw but I hope no one suggested local snaps are your only backup. They are a incredible way to steamline your backup, minimze disk space required and nearly eliminate load on the connected servers but of course you should move those backups to a 2nd device for better protection. No fair picking on only half of the solution.

Finally, I agree with your comment that different vendors do things differently. That why we all exist. Yet consistently I feel like you accuse us (other vendors in general) with being dishonest rather than understanding the solution we are suggesting.

Whew, there is my Friday rant and now I too feel better.

[Chuck Hollis]

Keith -- thank you for your thoughtful comments and disclosure, it's refreshing ...

We approached dedupe in order of "where's the most data?"

First target was backups (hence Avamar and DataDomain). Second target was file systems (hence Celerra dedupe and FMA, which is pretty cool). Third target is generic FC block (not here yet!). Fourth target is VMware-encapsulated binaries (something cool coming this quarter, I'm told).

NetApp, by comparison, decided not to go after backup wholeheartedly (is the VTL still around?), and has a few known issues on large filesystems. When performance issues start to rear their head in block environments, the answer was "just add PAM" which ended up in a difficult cost/benefit tradeoff, and makes the proposition a bit more complex, especially in write-intensive environments.

Like I said, strengths and weaknesses from each. That's what makes life so interesting.

Being from EMC, we're quite aware on how local snaps and copies can streamline backup. If you remember TimeFinder (1995) was the first enterprise-class local replication for storage. We did quite well integrating it with downstream backup processes (and do so to this day), but we never, ever confused a local array replica with a backup. And, yes, we see NetApp people making the "you don't need backups, just snaps" pitch day in and day out.

Thank you for sharing your rant!

-- Chuck

[nate]

one thing that drive me nuts is big companies buying their way into new accounts(or at least trying to). Or trying to buy their way into keeping the customer.

Usually involves inferior products from the big corp and smaller competitors with better stuff. I've encountered this at my last two rounds of storage purchases. The big corp realizes they can't compete with the product so they massively slash the price(often times below cost) knowing that all they need is the foot in the door and they can rape the customer for upgrades to "fix" the problem.

All of the anti trust stuff around Intel is a perfect example of this on a large scale anyways. Intel spent billions paying off the big OEMs to limit AMD's adoption, AMD eventually broke into the market and destroyed intel on a technical level for a few years until Intel caught up again. And at the end of the day what happens? Intel spends something like $1.2B to buy AMD off it's back, I believe they even wrote that off in a single quarter. That's a slap on the wrist, years of damage done and Intel writes it off in one quarter, that's not a deterrent to not doing those things again it's encouragement.

If it wasn't for massive missteps by Intel on the Itanium and the early netburst P4 architecture combined with near perfect execution by AMD on their Opteron platform and the x86-64 instruction set, AMD would of never have stood a chance.

Nothing makes me more mad than that kind of behavior, that I can think of anyways.

Which is why I not only gravitate towards smaller companies to be my suppliers and vendors but towards smaller companies to be my employers as well. I like best of breed, in the linux/unix world there's a saying for the CLI "do one thing and do it well".

[Dimitris Krekoukias]

Chuck,

Understanding NetApp's SnapVault product would shed some light regarding confusing a local array replica with a backup.

On the flipside, other vendors should probably learn enough about, for instance, Avamar before berating it.

There's a lot of that going around - most vendors, understandably, only have a very cursory knowledge of the competitors' systems, and even then a lot of the knowledge is based on very old data (like ex-employees from EMC going to NetApp or vice versa, for example).

It's difficult enough keeping up with one vendor's systems, let alone knowing all of them 100%.

There is a place for most technologies, but alone a single technology is usually not enough. Compellent has been doing their Data Progression for ages now and is way ahead of FAST or anything anyone else is shipping - yet never achieved enough traction. Maybe because, alone, just like thin provisioning, it won't take care of the majority of a Data Center's issues.

Complete solutions is what sells systems, and very few companies can claim that they can deliver solutions that, if not complete, at least take care of the majority of the problems.

Just my $.02

Thx

D

[Liem Nguyen from Compellent]

Hi Dimitris (and Hi Chuck),
Thank you for your compliment about Compellent's Data Progression technology. We totally agree with your comment that a single technology is not enough and also with Chuck that a storage platform should do "'all' of the cool stuff.” I just wanted to clarify Compellent's R&D philosophy. At Compellent we've always gone about creating a storage toolbox, not a point product. The goal being to deliver a complete storage solution. Sure, automated tiered storage is important. But so are thin replication, replays (snapshots), thin provisioning, boot from SAN, multi-site monitoring and reporting, and other apps, which are all integrated in the Compellent Storage Center and available to every customer regardless of their size. I’d also add to that toolbox something pretty fundamental but not everyone talks about, and one that’s missing from Chuck’s list: a scalable, open hardware platform. If a platform is truly scalable, it’ll be *persistent*--so that users don't have to throw out their old controllers to get any new functionality (e.g. SSD, FCOE) or more capacity. It’ll last for years, not just through the life of the initial maintenance contract.

Good discussion Chuck, thanks for starting it. As I always say, one guy's rant is another guy's riposte. :-)

[Chuck Hollis]

Hi Liem -- I think you're making my point here.

You offered up a nice list of features and capabilities that aren't unique to Compellent, and can be found in many other storage arrays, including EMC's.

Simply listing off a bunch of features that are now industry standard doesn't really help with meeting customer needs, e.g. "the car I'm selling has headlights, and a steering wheel, and a nice gas pedal". That's nice, but somewhat irrelevant.

Congratulations on having a product that is -- by some measures -- more complete than many. But, once again, you're going to have to bring something more to the table to differentiate and position yourself as the best solution for a customers' requirement.

Cheers!

-- Chuck

[Storagetexan]

Hi my name is Tommy Trogden and I’m a Xiotech Enterprise Architect – this feels oddly like some sort of 12-step program when I do this – less anyone is unsure of where my loyalties lie :)

Chuck,
I agree with you, some people just can’t appreciate a good RANT. I’m not one of those and thoroughly enjoyed your commentary. You should turn this into a weekly or monthly occurrence!!

In particular I love RANT #1 – I couldn’t agree more with you. At the end of the day there are a lot of vendors that have the same “features and functions” whether it’s built in, bolted on, or done a little differently. At the end of the day, it’s all about solving the customers business and technical issues. What’s even more interesting is for every feature a storage vendor puts into their array, the OS/Application manufacture eventually adds it to their solutions. Microsoft rolled in Single Instance Storage (good/bad or indifferent) and native async replication, ZFS has a ton of cool features built into the file system for the various Unix platforms (Native COW Snaps, dynamic striping, dedupe etc), your very own VMWare VSphere 4 has native thin provisioning. It’s getting to the point that the OS and applications vendors are simply adding these features into their platforms almost as quickly as storage vendors are adding “value” into their own. At the end of the day customers are going to be in a great position to figure out just exactly where and how they want to deploy these types of technologies. It’s a consumer “Win-Win” all around. My opinion is eventually it’s going to move away from “bells and whistles” and back to who protects, scales (both capacity and performance) their data the best.

I also really liked rant #3 – “if we don’t have it, you don’t need it” – That’s the greatest one ever!! The reverse is even better, “If we HAVE it, then clearly you MUST need it” approach. I run into that a lot and it’s almost comical. Hmm, then again, maybe it’s me and I’m living your “if we don’t’ have it, you don’t need it” approach. I think I’ll need to grab a beer and contemplate that a little more.

So, again NICE JOB on the blog post. How you crank these out as quickly as you do is beyond me. As a newbie in the blogging arena I’m always in awe of those that have been doing it as long as you have.

Thanks,
@Xio_Tommyt
www.StorageTexan.Com

[Mike Workman]

Chuck - Mike Workman, Pillar Data

I guess the votes are in favor of your rant! I enjoyed it for sure.

Is there an industry this collective set of behaviors doesn't happen in? I think basic commodities are all down to $/pound or euros/kilo, but industries like ours have this property in general. Take for example the Automobile industry. Almost every car will get you from point A to Point B. But the parameters on how they do that, and some of the underlying technologies are different. In some cases the feeds and speeds, like MPG, 0-60, leg room (which I don't need as I am vertically challenged) are enough to make a choice. In others, it might be service and support. But in the end, all we have as vendors is to appeal to our prospects with the things we have that we feel offer a better solution than the other guy will sell you at the same or better price. So we all endeavor along those lines.

What bothers you I think, because it bothers me, is the way in which people misrepresent their solution with careless disregaurd of the facts. While close inspection reveals the deception, it takes work on the part of a prospect to reveal the complete truth.

I think all of our Sales teams engage in a battle around these points of obfuscation and deception, and sometimes they lose, which makes us angry when the lose is perhaps not warranted by the truth. Unfortunately none of us are completely objective in our assessment of the truth, so that adds to the emotion giving way to rant.

So, we endeavor to expose the BS, try and remain objective, and forge ahead. For me, I enjoy the mission, the journey, and am just happy to get to work at it. I can tell from most of your blogs, and those of the other commenters that they are too.

Happy 2010!

Mike