"Independent Analysis" -- Is There Hope?

I try to avoid rants in this blog.  It doesn't make for especially good reading, I've found.

But one of the topics that has elevated my blood pressure over the last few years has been the practice of vendors funding "independent analysis" or "independent research" that -- surprisingly! -- says exactly what the vendor might want it to say.

Someone sent a report from the Aberdeen Group my way that actually gave me a bit of hope that the practice -- and its participants -- might be coming around to a different way of doing things.

Put differently, there might be hope!

I Really Hate This Kind Of Marketing

If you're a regular reader of this blog, you know I've reacted very strongly to -- as an example -- NetApp's practice of funding "independent research" or "independent testing" along these lines.  Whether it's the latest VeriTest "analysis", or torturing the poor SPC, I am of the opinion that these sorts of practices serve neither customers nor the industry well.

I wish EMC was immune from from these temptations.  We're not entirely, although we're getting better.  Occasionally, a product group at EMC will do one of these sorts of things, and get it out into the marketplace before saner heads can prevail.  From my perspective, we're doing far less of this sort of thing than we used to, which is good.  And, generally, speaking, we know it's not a good thing.

The analyst, research and testing community have a key role to play here as well.  Ideally, they'd have a code of conduct regarding what work they take on, how the work is done, and how it's used.  There are many shining examples of people who understand this, but a few who don't.

So, What Happened?

Someone flipped me this link showing an Aberdeen group study showing that (among other things), EMC customers did much better than most dealing with storage capacity growth: power consumption, floor space, cooling, utility costs, and so on.

Not exactly an earth-shattering revelation, if you know EMC's storage products and how we work with customers to use them effectively.

What was suprising -- at least to me -- is that I found out that EMC had absolutely nothing to do with this research.  We didn't know that they were doing it, didn't send them to "special" customers, didn't tell them what questions to ask, didn't review the results, and so on.  Completely hands-off, so to speak.

Actually, someone came across the white paper after it had been published, and we paid a token amount for reprint rights to put it on our website, nothing more.

So, Why Is This Interesting?

Sure, you can be skeptical about how Aberdeen conducted the survey, methodology, interpretation of the results, and so on.  But, given the circumstances, one thing you don't have to be on the lookout for is the annoying flavor of vendor-defined pre-ordained conclusions I find so distasteful.

Is this a better model for independent research in our industry?

I'd like to think so.  Researcher asks a question that's of interest to many people (customers and vendors alike), conducts their own research, and sells the results to multiple parties to cover costs.

Even better would be a community-based approach, like the folks at Wikibon are trying to get off the ground.  I don't think they've really hit critical mass yet, but it's promising, and I wish them the best.

As a vendor, I have no problem living in a world where well-meaning researchers try to get to the bottom of interesting questions in our industry.  I just don't like it when it devolves to a "pay to play" scenario -- no one's interests are served.

What do you think?

Comments

[Scott Quick]

Great post. Makes you wonder if Aberdeen was trying to curry favor with EMC for follow-on work ;-) Nonetheless and as you mention, analysts still play a valuable roll - IF they avoid the preordained conclusions you mentioned.

[alan pelz-sharpe]

As far as I know CMS Watch is the only truly independent analyst firm in the business (we never consult to vendors, we don't write white papers, speak at vendor events - we pay our own travel and expenses, we don't even accept a meal from a vendor etc). Other do a good job too such as Burton Group - but its very very unusual for analyst firms not to be heavily dependent on vendor money.
Aberdeen represents the worst elements of the analyst industry in many peoples eyes (read the WSJ exposes of their practices) - so its highly unusual and frankly hard to believe that there was no vendor funding or involvement at all in the report you mention - if it is true then maybe there is hope.
But its a double edged sword Chuck - vendors (EMC included) buy analyst subscription services and consulting - and this gives the AR managers immense leverage over the analysts - heavily influencing or stopping any negative or critical output. Sad but true.
For truly independent researchers there is no financial leverage - but there are other means to twist the arm, via restriction of access to execs or refusal to provide briefings or fact checking etc.
The analyst community and the vendor AR community is in desperate need of a thorough overhaul - and like you I think we would all benefit as a result.
Best
Alan