That's what the press release SHOULD have said ...
I thought I had seen it all before.
But my colleagues at NetApp win this week's prize for "creative marketing".
They've just submitted SPC-1 benchmarks on a few of EMC's products.
And written a nice press release as well.
My oh my ...
A Little Background
A while ago, I took NetApp to task for claiming things about their system performance that weren't only not quite true, but could put customers in a very painful situation.
As a result, they appeared to back off a bit on the hyperbole, but I guess they're back with a new angle.
You may be acquainted with the entire SPC benchmark debate in our little storage community, or it may be new to you.
We think it's a bad test, and it can (and often does) mislead customers. We think that the overall approach is subject to all manner of "creativity" by the benchmarketing team. And it's often turned into the worst kind of benchmarketing exercise that ends up causing more harm than good.
Not good for customers, not good for the industry.
And, since we don't think it's particularly a good thing, we've refused to play.
So NetApp took an interesting approach. They appeared to have bought some of EMC's products, ran the tests on our behalf, and submitted the results to the SPC web site.
It's certainly a creative approach, I'll grant them that.
This Opens A Whole New Can Of Worms
Imagine a world where every vendor runs benchmarks on every other vendors' products.
Would you like the benchmarks on Product X?
Would you like it from Vendor A, Vendor B, or Vendor C?
At least there was semblance of order when each vendor ran their own benchmark, and you knew that they were highly motivated to show their products in the best possible light.
Now we have two kinds of tests: those designed to show products in their best possible light, and those designed to show them in the worst possible light.
I would offer that a rather pointless exercise had just become even more pointless, if that's possible.
I Know, I Know ...
Sure, it'd be great if there was (a) a great, independent storage benchmarking test that (b) captured real-world use cases of storage, and (c) was conducted in a fair an impartial way so that customers could easily compare the performance differences between different vendors' offerings.
Well, we don't seem to live in that world. And pretending that we do won't do any good, either.
So, What Does It Mean?
That part confuses me mightily. Look, I'm a pretty adept marketing guy in this industry, and I keep wondering "what were they thinking?".
Here's why I'm puzzled ...
First, the SPC-1 and SPC-2 have been generally discredited in the industry (and by most customers) as mainly a marketing exercise. Sure, there are a few vendor diehards out there (e.g. IBM) who are a bit out-of-sync with this reality, but you'd think NetApp would have known better.
And, as far as customers go, it just never comes up. Never.
Second, it's hard to imagine NetApp positioning themselves as the "fair and impartial tester" in this face-off. The only reason you'd spend the money to buy the equipment and run the tests is to put your competitor in a bad light. I think most reasonable customers would figure that part out.
Third, I wonder why NetApp is picking a fight that's going to be hard for them to (architecturally) win.
I mean, look how NetApp builds their storage arrays, and performance optimization doesn't seem to be the highest on the list. If a customer really, really cares about performance, they're not likely to be considering NetApp products.
Sure, there are other good reasons to potentially consider NetApp, but I'd argue that "blistering performance" wouldn't make the top three. If you want to go fast, there are better ways to do it.
Trying To Look Inside Their Heads
This type of marketing is usually cause-and-effect. You don't spend time and money showing that you're (hopefully) faster than a competitor unless your sales force tells you "hey, we're getting beat up on performance".
Well, if that's really the case, I'd offer that you need to build faster products, if that's the case. Maybe skip the aggressive benchmarketing. Or perhaps put some flash memory in them? ;-)
All posturing aside, I think NetApp's under a lot of pressure these days. In their traditional markets, they've got strong competitors above them (EMC) and below them (Isilon, et. al.). Despite investing substantially in acquisitions (Spinnaker, Topio, Decru and now Onaro), not much has changed.
I'm sympathetic, but up to a point. I keep wondering when they're going to mature as a company, and potentially start building for long-term customer value, rather than the latest flashy campaign.
And this latest effort on their part demonstrates to me that it's probably going to take a while longer before we see that happen.
[more thoughts -- added 1/30]
I was thinking about this more last night, and had even more questions I couldn't answer.
For instance, the SPC and NetApp have both been around for quite a while -- and have studiously ignored each other -- why the big change now? Did IBM (the only other vendor cheerleader) put them up to it? Or did they smell an opportunity for a marketing ploy?
How does Walter E. Baker fit in? Walter is the infamous part-time "administrator" for the Storage Performance Council. He wrote the tests, he runs the web site, he's the "audit committee", and so on. It seems to be a part time gig for him to make a few bucks. Nice little thing he's got going.
The reason this is unusual is that dear Walter prefers to keep a very low profile -- you won't find his name on too much associated with the SPC, but he's the guy. And there he is in the press release, with a nice quote. Maybe Walter likes the idea of lots of vendors running lots of tests on each other's equipment, enough to coax him out of hiding.
I don't know if you noticed this, but NetApp issued their press release, and it pointed to a bunch of results that Howard hadn't gotten around to posting on his SPC web site quite yet. I'm thinking that Howard is sitting in a staff meeting for his day job, his blackberry starts going crazy about this SPC thing, etc. The results didn't get posted until later in the day, which probably caused a fair amount of consternation at NetApp PR.
Finally, why the SPC-1? It's the older of the two tests, and was replaced by the SPC-2, which now being replaced by the SPC-3. Strange, indeed.
[end of late night rant]
Finally ...
So, what do you think?
Is this kind of benchmarketing a good thing, or a bad thing? What's your reaction?
I'd love to hear ...
[postscript on 02/06/08]
Boy, did I take a beating on this one, as well as my previous post on "Does Anyone Take The SPC Seriously?" I guess that's the price you pay for being a corporate blogger!
Whereas much was written in the blogosphere by all sorts of people, two things stand out from all of this.
First, the commentary from real live customers (as opposed to vendor fanboys, industry curmudgeons and other denizens of the blogosphere) was very supportive, in general.
While customer are very entertained by all of this, they told me they don't take vendor benchmarks seriously, certainly don't take the SPC seriously, nor do they lend any credence to NetApp's announcement. Many of them know what a CLARiiON can do, and they know what a NetApp box can do. Says more about the test and the tester, they said.
I thought that was the case, but it's always nice to hear from people ;-)
Second, a few industry commentators weighed in and basically confirmed most of what I said. If you're interested, see what Beth Pariseau and John Webster have to say about all of this, among others.
Finally, after looking at all the documents, I think I've figured out (unofficially, mind you) how they might have wired the test in their favor. It's both subtle and obvious -- can you find it as well?
And, between you and me, I"m done with the SPC debate for the time being. It's getting boring, and it doesn't seem to bring much value to customers. Besides, I dislike the rudeness that this topic seems to bring out in people. It's OK to disagree, it's not OK to be disagreeable.
[end postscript]
Recent Comments