Charting The Intel Effect

Two bits of news, oddly correlated.

First, there's EMC's recent SPEC benchmark posting, where a single Intel-based Celerra data mover absolutely *smokes* every other NAS device out there.

Second, there's Timothy Prickett Morgan's take on the IDC server numbers, where -- once again -- the market bought more Intel-compatible servers and fewer of the other kind.

I think I've seen this movie before, and I know how it ends :-)

Remembering The LAN Wars

Once upon a time, there was a vibrant industry debate about what kind of wire and protocol would make sense for a building network.  There were many choices: RS-422, Token Ring, Coax -- even a few more esoteric choices.  And there was Ethernet.

At the time, I worked for a vendor (Convergent Technologies) who had pretty much bet the farm on the RS-422 approach.  We ended up losing that argument.  As did any customer who wired their building with Token Ring, etc.

Ethernet won -- not necessarily because it was the better technology, but simply because the fact that economies of scale started to kick in (as well as a powerful ecosystem) and that was that.  The effect is often called "Ethernet economics" in the industry.

Case closed.

Fast Forward

Listen to me talk, and I'm very clear that we believe that -- over the next few years -- far more enterprise workloads will end up on Intel (or potentially compatibles), and far fewer workloads will end up on other kinds of more traditional RISC processors.

IDC makes the point that -- for the last five quarters -- increasingly more money is being spent on Intel and compatibles, and considerably less is being spent on everything else.

Sure, I can make a technology argument that Intel is delivering amazing price/performance, especially with their latest round of Westmere processors.  But my stronger argument is based on economics and ecosystems -- the same forces that makes Ethernet the de-facto standard it is today.  Call it "Intel economics".

For me -- case closed.

And The Really Fast NAS Part

Our engineering team posted an interesting SPEC NFS benchmark today. 

Contrary to the beliefs of some, EMC is not against standard benchmarks per se, just bad benchmarks.  We've always thought the SPEC has done a good job in characterizing real-world workloads, as well as managing an open and fair process.

If you go look at the SPEC NFS benchmark page, you'll see this single node system (with a spare passive node for failover) handily dusts everything else out there at 135K NFS ops, and by a large margin.  And we didn't need to use flash drives to do it, nor 10Gb ethernet :-)

The VG8 scales up to 8 nodes (actually, 7 active and one failover), so it's reasonable to expect that a fully loaded VG8 might deliver ~945K ops -- and that's before we get into things like 10GbE and flash and pNFS. 

Hey, 1M+ ops might be a good goal to shoot for the next round ...

This Isn't Really About Benchmark Bragging ...

OK, maybe it is -- just a little :-)

I think the real story here is about the new "Intel economics" -- the ability to use technology, scale and ecosystem to deliver price/performance that can't be easily achieved any other way.

It doesn't matter whether you're a vendor of server or storage, or a user of these devices -- it looks like it's quickly becoming an Intel world. 

Comments

[Vaughn Stewart]

There must be a glitch in the Matrix! Did EMC change something? This rig appears to be the same one used in the CIFS Spec results.

http://blogs.netapp.com/virtualstorageguy/2010/02/emc-benchmarking-shenanigans

It was the CIFS results where EMC claimed superior performance yet somehow forgot about economics. Who would buy such a rig when >50% of the performance can be obtained from other arrays for ~10% of the price?

Kudos for building such a config?

[Vaughn Stewart]

Chuck - my apologies for my initial comments, I read the wrong link. I read the December 2009 results (EFD drives) versus the August 2010 results (FC drives).

I still have a number of questions around the viability of such a config to address customer needs, with the most pressing being 304 drives configured into 152, 2-disk RAID-1 mirrors? Why would EMC use such an unrealistic config? RAID-1 reduces the usable capacity by 50% and does not provide enough data protection for today's workloads with the amount of media imperfections found in the large disk drives available today.

What would be very interesting would be to see the performance numbers of the same config with RAID-5 & RAID-6 enabled. We have completed testing with traditional storage arrays (note I'm not implying an EMC array was used in this testing), which demonstrates a significant drop-off in performance when more practical types of RAID are implemented.

http://blogs.netapp.com/virtualstorageguy/2010/08/fact-vmware-vsphere-on-netapp-is-fastr-and-greener.html

I think its great to show how fast one's array can perform, but it sure seems misleading as the config is not practical for most. When it comes to Spec submissions maybe EMC should follow the lead of NetApp and only publish real-world hw architectures for such tests.

[Chuck Hollis]

Hi Vaughn

When it comes to these sorts of benchmarks, the guys usually use what they have in the lab. I don't know why they chose a RAID 1 config vs. the many other alternatives out there (e.g. FAST, etc.) but that's what they did.

For many customers, RAID 1 is still a legit option, even if NetApp isn't able to support it. I'd hardly call it "unrealistic" or "misleading". Lots of it out there still.

A bit of trivia: the Symmetrix has always supported a feature known as DMSP (dual mirror service policy) where the array will adaptively select the drive in the pair that can service the I/O the fastest. It's an enhancement to basic mirroring performance in much the same way PowerPath's algorithms are an enhancement over standard multi-pathing.

You have to admit, though, the SPEC numbers are pretty amazing, and they absolutely blow away just about everything else out there. A single VG8 blade appears to outperform your current high-end multi-node NAS devices.

You and your team have some explaining to do. And complaining bitterly about the use of RAID 1 doesn't strike me as a particularly useful approach.

My impression is that getting good SPEC results has more to do with processor performance and I/O efficiency of the NAS head, and less to do with the backing data store.

Also interesting is that only a modest amount of 1GB ethernet was needed to achieve those results.

-- Chuck

[nate]

Chuck your thought on the good SPEC results is right - just look at the Avere results for proof of this.

Though back end storage can contribute a lot. Back when Exanet was still around one of their big customers demonstrated a 200% increase in SPEC SFS performance using "better" storage vs the LSI stuff the NAS typically shipped with. Same number of disks, same number of NAS heads, double the performance.

If SPEC forced disclosure of the cost of the solutions tested that would be great, I think overall it's a good benchmark(mainly because it provides a level playing field to compare systems, not everyone has the ability to bring everything in house to test).

The V-MAX certainly is a pretty impressive system, 256GB of cache! wow..

[Chuck Hollis]

Hi Nate

I think as technology evolves, the bottleneck moves from place to place -- back-end, NAS head, network, etc. What I took away from this was that a smokin' multicore CPU (properly leveraged) can make a big difference.

The classic debate in benchmarking is the right mix between optimizing for performance and optimizing for cost. I think the team picked a point somewhere in the middle of the continuum. EMC has both more expensive and performance backends (imaging all flash, for example) and cost-effective ones.

You're right, the VMAX has a large, shared non-volatile cache -- it's one of the hallmarks of high-end enterprise storage: those big caches can soak up all sorts of transient I/O bursts very effectively. As the capacity of the Symmetrix has increased dramatically, so has the cache -- the ratio has stayed largely constant over the years.

Thanks for the comment ...

-- Chuck

[Vaughn Stewart]

I think you made my point. While the numbers are large one has to ask is the the test bed used to obtain the numbers either...

A. Realistic and likely to be deployed

B. Unrealistic and unlikely to be deployed

For most I believe the answer is B. As such EMC should be commended for achieving the performance numbers which they posted and they should provide performance numbers obtained from a real-world configuration likely to be implemented by the majority of EMC customers.

While the results in less glamorous results it adds credibility that EMC is more than a storage marketing company.

Again, nice numbers, just seeking substance.

[Chuck Hollis]

Vaughn

You're sounding like a broken record in a digital age.

The configuration *is* realistic for some customers. Just because your company can't support a RAID 1 config doesn't mean isn't realistic for some customers.

And I think your focus on the back-end is a rather clumsy misdirection on the real issue -- your NAS performance is way behind the curve.

Not by a little, but by a whole lot.

Looking forward to a more meaningful response in the near future.

-- Chuck

[Mark Janvier]

If you want to test a new product like the VG8, it seems perfectly normal to me that you try to make the new product the weakest link in the performance chain. Otherwise you will not be able to test the new gear's true ability. Discussing what configuration is used in other parts of the chain is useless, since the test was made to see what the new product is capable of. And it seems the new product is capable of a lot, if placed in the correct surroundings. Thus proving the initial point of Chuck’s blog: when using Intel technology you can scale performance together with Intel.

Discussing if the configuration used in this particular test is practical or real-world or not, again seems useless. For some it might be for others it might not.

Mark Janvier – TC @ EMC

[Tim]

Ok so what I like most about this is the fact that if you do a normal enterprise half loaded ~800 to 1200 spindle 4 head config, with a nice FAST balance, you will have great performance and capacity for both NAS & SAN.

[Visiotech]

Rules of thumb. Take any benchmarks metric and divide it the number of disk used during the tests. You'll be surprise to see they all starts to appear the nearly the same (+- 20%).

This way you have a realistic measure or apples vs apples. More disk you have more likely you have higher throughput, IOPS etc.

Once you hit the physical limitation of the disk then you start to see how the controllers are good enough to hide disk and backend ports weakness. Strange that most vendors do not show where the chart plateau appears.

These benchmarks never reveal the truth about any disk storage. They are a point in time on the chart with careful setup lab environments. Not real customers life workload of installation. It is well known that disk performance degrade as file systems age. This is mainly due to the nature of fragmentation as well as random IO caused by several shares.

There is other benchmarks that EMC never did. Storage Performance at storageperformance.org. I also do not endorse any of these benchmarks except SPC Benchmark 3BR who will be release soon. At least this one is much closer to customer real life situation.

[Chuck Hollis]

Hi Visio

While I appreciate the comment, there are a number of points that I think deserve clarification.

".. divide by the number of spindles ..."

Yes, we've all seen benchmark results that follow that rule of thumb, but keep in mind (a) this does not apply to use of flash as either storage device and/or cache, and (b) at some point, the controller / head / network becomes the limiting factor, hence the large number of spindles in this test.

" ... physical limitations of the disk ..."

Although most disk components are similar, how they're used by the storage array vendor can vary widely: algorithms, back-end loop implementations, cache, etc.

That's another reason we need lots of spindles and a crushing workload -- otherwise, the difference aren't readily apparent in more modest setups.

" ... benchmarks never reveal the truth ..."

Couldn't agree more -- there's no good substitute for real workloads with real operational procedures, e.g. snaps, backups, replication, disk recoveries, etc. Ideally, we'd see multiple workloads on the same storage array, filled to capacity, and run for many weeks.

" ... disk performance degrades over time ... "

Mostly true for environments that defer writes and attempt to come back and defrag later, like WAFL. The degree and rate of degradation varies widely.

" ... other benchmarks that EMC never did ..."

We've never endorsed nor used the SPC test. We haven't liked the tests nor the methodology nor the governing processes. However, there's always room for change -- we'll keep an eye on the 3BR benchmark -- thanks!

-- Chuck