« The Real Story At Pfizer | Main | Meet The Prince of Dataness »

August 25, 2010

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d83451be8f69e2013486779aba970c

Listed below are links to weblogs that reference Charting The Intel Effect:

Comments

Vaughn Stewart

There must be a glitch in the Matrix! Did EMC change something? This rig appears to be the same one used in the CIFS Spec results.

http://blogs.netapp.com/virtualstorageguy/2010/02/emc-benchmarking-shenanigans

It was the CIFS results where EMC claimed superior performance yet somehow forgot about economics. Who would buy such a rig when >50% of the performance can be obtained from other arrays for ~10% of the price?

Kudos for building such a config?

Vaughn Stewart

Chuck - my apologies for my initial comments, I read the wrong link. I read the December 2009 results (EFD drives) versus the August 2010 results (FC drives).

I still have a number of questions around the viability of such a config to address customer needs, with the most pressing being 304 drives configured into 152, 2-disk RAID-1 mirrors? Why would EMC use such an unrealistic config? RAID-1 reduces the usable capacity by 50% and does not provide enough data protection for today's workloads with the amount of media imperfections found in the large disk drives available today.

What would be very interesting would be to see the performance numbers of the same config with RAID-5 & RAID-6 enabled. We have completed testing with traditional storage arrays (note I'm not implying an EMC array was used in this testing), which demonstrates a significant drop-off in performance when more practical types of RAID are implemented.

http://blogs.netapp.com/virtualstorageguy/2010/08/fact-vmware-vsphere-on-netapp-is-fastr-and-greener.html

I think its great to show how fast one's array can perform, but it sure seems misleading as the config is not practical for most. When it comes to Spec submissions maybe EMC should follow the lead of NetApp and only publish real-world hw architectures for such tests.

Chuck Hollis

Hi Vaughn

When it comes to these sorts of benchmarks, the guys usually use what they have in the lab. I don't know why they chose a RAID 1 config vs. the many other alternatives out there (e.g. FAST, etc.) but that's what they did.

For many customers, RAID 1 is still a legit option, even if NetApp isn't able to support it. I'd hardly call it "unrealistic" or "misleading". Lots of it out there still.

A bit of trivia: the Symmetrix has always supported a feature known as DMSP (dual mirror service policy) where the array will adaptively select the drive in the pair that can service the I/O the fastest. It's an enhancement to basic mirroring performance in much the same way PowerPath's algorithms are an enhancement over standard multi-pathing.

You have to admit, though, the SPEC numbers are pretty amazing, and they absolutely blow away just about everything else out there. A single VG8 blade appears to outperform your current high-end multi-node NAS devices.

You and your team have some explaining to do. And complaining bitterly about the use of RAID 1 doesn't strike me as a particularly useful approach.

My impression is that getting good SPEC results has more to do with processor performance and I/O efficiency of the NAS head, and less to do with the backing data store.

Also interesting is that only a modest amount of 1GB ethernet was needed to achieve those results.

-- Chuck

nate

Chuck your thought on the good SPEC results is right - just look at the Avere results for proof of this.

Though back end storage can contribute a lot. Back when Exanet was still around one of their big customers demonstrated a 200% increase in SPEC SFS performance using "better" storage vs the LSI stuff the NAS typically shipped with. Same number of disks, same number of NAS heads, double the performance.

If SPEC forced disclosure of the cost of the solutions tested that would be great, I think overall it's a good benchmark(mainly because it provides a level playing field to compare systems, not everyone has the ability to bring everything in house to test).

The V-MAX certainly is a pretty impressive system, 256GB of cache! wow..

Chuck Hollis

Hi Nate

I think as technology evolves, the bottleneck moves from place to place -- back-end, NAS head, network, etc. What I took away from this was that a smokin' multicore CPU (properly leveraged) can make a big difference.

The classic debate in benchmarking is the right mix between optimizing for performance and optimizing for cost. I think the team picked a point somewhere in the middle of the continuum. EMC has both more expensive and performance backends (imaging all flash, for example) and cost-effective ones.

You're right, the VMAX has a large, shared non-volatile cache -- it's one of the hallmarks of high-end enterprise storage: those big caches can soak up all sorts of transient I/O bursts very effectively. As the capacity of the Symmetrix has increased dramatically, so has the cache -- the ratio has stayed largely constant over the years.

Thanks for the comment ...

-- Chuck

Vaughn Stewart

I think you made my point. While the numbers are large one has to ask is the the test bed used to obtain the numbers either...

A. Realistic and likely to be deployed

B. Unrealistic and unlikely to be deployed

For most I believe the answer is B. As such EMC should be commended for achieving the performance numbers which they posted and they should provide performance numbers obtained from a real-world configuration likely to be implemented by the majority of EMC customers.

While the results in less glamorous results it adds credibility that EMC is more than a storage marketing company.

Again, nice numbers, just seeking substance.

Chuck Hollis

Vaughn

You're sounding like a broken record in a digital age.

The configuration *is* realistic for some customers. Just because your company can't support a RAID 1 config doesn't mean isn't realistic for some customers.

And I think your focus on the back-end is a rather clumsy misdirection on the real issue -- your NAS performance is way behind the curve.

Not by a little, but by a whole lot.

Looking forward to a more meaningful response in the near future.

-- Chuck

Mark Janvier

If you want to test a new product like the VG8, it seems perfectly normal to me that you try to make the new product the weakest link in the performance chain. Otherwise you will not be able to test the new gear's true ability. Discussing what configuration is used in other parts of the chain is useless, since the test was made to see what the new product is capable of. And it seems the new product is capable of a lot, if placed in the correct surroundings. Thus proving the initial point of Chuck’s blog: when using Intel technology you can scale performance together with Intel.

Discussing if the configuration used in this particular test is practical or real-world or not, again seems useless. For some it might be for others it might not.

Mark Janvier – TC @ EMC

Tim

Ok so what I like most about this is the fact that if you do a normal enterprise half loaded ~800 to 1200 spindle 4 head config, with a nice FAST balance, you will have great performance and capacity for both NAS & SAN.

Visiotech

Rules of thumb. Take any benchmarks metric and divide it the number of disk used during the tests. You'll be surprise to see they all starts to appear the nearly the same (+- 20%).

This way you have a realistic measure or apples vs apples. More disk you have more likely you have higher throughput, IOPS etc.

Once you hit the physical limitation of the disk then you start to see how the controllers are good enough to hide disk and backend ports weakness. Strange that most vendors do not show where the chart plateau appears.

These benchmarks never reveal the truth about any disk storage. They are a point in time on the chart with careful setup lab environments. Not real customers life workload of installation. It is well known that disk performance degrade as file systems age. This is mainly due to the nature of fragmentation as well as random IO caused by several shares.

There is other benchmarks that EMC never did. Storage Performance at storageperformance.org. I also do not endorse any of these benchmarks except SPC Benchmark 3BR who will be release soon. At least this one is much closer to customer real life situation.

Chuck Hollis

Hi Visio

While I appreciate the comment, there are a number of points that I think deserve clarification.

".. divide by the number of spindles ..."

Yes, we've all seen benchmark results that follow that rule of thumb, but keep in mind (a) this does not apply to use of flash as either storage device and/or cache, and (b) at some point, the controller / head / network becomes the limiting factor, hence the large number of spindles in this test.

" ... physical limitations of the disk ..."

Although most disk components are similar, how they're used by the storage array vendor can vary widely: algorithms, back-end loop implementations, cache, etc.

That's another reason we need lots of spindles and a crushing workload -- otherwise, the difference aren't readily apparent in more modest setups.

" ... benchmarks never reveal the truth ..."

Couldn't agree more -- there's no good substitute for real workloads with real operational procedures, e.g. snaps, backups, replication, disk recoveries, etc. Ideally, we'd see multiple workloads on the same storage array, filled to capacity, and run for many weeks.

" ... disk performance degrades over time ... "

Mostly true for environments that defer writes and attempt to come back and defrag later, like WAFL. The degree and rate of degradation varies widely.

" ... other benchmarks that EMC never did ..."

We've never endorsed nor used the SPC test. We haven't liked the tests nor the methodology nor the governing processes. However, there's always room for change -- we'll keep an eye on the 3BR benchmark -- thanks!

-- Chuck

The comments to this entry are closed.

Chuck Hollis


  • Chuck Hollis
    Chief Strategist, VMware SAS BU
    @chuckhollis

    Chuck has recently joined VMware in a new role, and is quite enthused!

    Previously, he was with EMC for 18 years, most of them great.

    He enjoys speaking to customer and industry audiences about a variety of technology topics, and -- of course -- enjoys blogging.

    Chuck lives in Holliston, MA with his wife, three kids and four dogs when he's not travelling. In his spare time, Chuck is working on his second career as an aging rock musician.

    Warning: do not buy him a drink when there is a piano nearby.
Enter your Email:
Preview | Powered by FeedBlitz

General Housekeeping

  • Frequency of Updates
    I try and write something new 1-2 times per week; less if I'm travelling, more if I'm in the office. Hopefully you'll find the frequency about right!
  • Comments and Feedback
    All courteous comments welcome. TypePad occasionally puts comments into the spam folder, but I'll fish them out. Thanks!