How The EMC Disk Library Came To Be

EMC offered up a press release today reflecting on the market success we've enjoyed with the EMC Disk Library (EDL).

I felt a special bit of satisfaction when I saw this.  The EDL was one of my special projects way back when.

Some people think that new product development is largely an autonomous, faceless process.

It's not.  Sure, there's plenty of process and data.  But, in most cases, there are a few people who really believe in a new product, and are willing to go to great lengths to see it happen.

And I was one of those people for the EDL.

Parallel ATA Drives Become Viable For Enterprise Storage in 2003

It's always hard recreating context from the past -- technology moves so quickly that people forget and assume that things always were the way they are now.

Back in 2003, I was the product marketing guy for EMC's storage group.  We were hotly debating the pros and cons of ATA disks in the CLARiiON.  The engineering team had figured out a clean way (using a clever bridging adapter) to get parallel ATA disk drives to work seamlessly in a FC-based CLARiiON.

It looked that we would have a nice lead in the marketplace in offering pATA disk economics in an enterprise-class FC array such as the CLARiiON.  The cost per megabyte was extremely attractive -- and, if they were used in the right way -- it was easy to see that it'd be a pretty popular offering.

The product marketing team was wrapped around the flagpole a bit trying to figure out the pros and cons of this innovation. 

Sure, there was now the ability to provide lower cost (and lower performing) capacity mixed and matched as part of a CLARiiON, but there were concerns. 

AS an example, what if people put the wrong kind of information on it, and had sub-optimal performance?  (Turned out this didn't happen too often, most people understood that the pATA disks didn't offer the level of performance of the FC drives, and, if they made a mistake, it only happened once!).

So we spent a lot of time thinking about potential use cases for this decidedly different storage medium.

Thinkg About Use Cases

The obvious use case is to use them for data that's not frequently accessed -- archives, for example.  We had seen success using parallel ATA drives in EMC's Centera, and we knew the access patterns well.  So that was one.

But as we thought about using pATA drives for backups, it became obvious that there was no one "right answer" to how they might be used.

There were certain backup packages that knew how to use disk as a target, e.g. configure a big file system, and point your backup app at it.  But, as we talked to users, we realized that this was nice, but was hardly compelling.  Most backup-to-disk to a file system was bottlenecked by the LAN used to reach it, and we knew that pATA drives brought speed to the party, as compared to tape.

There was another school of thought that we could just use snaps and clones to back up applications.  Sure, you had to think through just how you did it, as the performance of the pATA drives could become an issue if you weren't a bit careful.  And there was real work involved getting all of this set up in large, complex environments.

But I kept thinking -- what about direct replacement of tape?  What about being able to use these pATA drives as a direct tape substitute?  No muss, no fuss?

Virtual Tape Libraries vs. Disk Libraries

As many of you know, VTLs have been around for a while.  They use disk as a cache -- they buffer the incoming backup streams, do some housekeeping and stacking, then turn around and write tape efficiently.  When you go to restore, you're usually coming back off of tape, unless the backup image in question is sitting in the disk cache.

Now, there is nothing wrong with the VTL approach, but it was conceived in a time when disks were horribly expensive.  It was also pretty clear to many of us that disks were going to be a whole lot cheaper in the near future, and this fundamental assumption wouldn't be valid for much longer.

I kept thinking in terms of disk as a direct target for a backup application.  No modifications to the backup application.  Native speed of sequential disks for both backup and restore.  Tape positioned as a backup to the backup.  Use the strengths of the underlying array (e.g. CLARiiON) for performance, availability, management, etc.

We ended up calling the concept a "disk library" to differentiate from the VTLs that had come before it.  It was a different value proposition and offering, based on the emergence of lower-cost disk media. 

But There Were Concerns

Even though parallel ATA disks brought the costs way down, it was fair to say that for most use cases, backing up to disks was more expensive than backing up to tape.  That caused significant mental consternation for a lot of people who thought that customers just wouldn't pay more for a disk-based solution.

There was also arguments that snaps were better, that existing backup apps could write to file systems, and so on.

Now, I lined up completely on the other side of these discussions.

I knew that people were using incremental backups heavily, and that restore times could be problematic from tape.  Putting all those incrementals on disk meant that restore times would be blazingly fast.  And restore times can be pretty important when you need them ;-)

I also knew that many IT shops had spent a lot of time and effort getting their backup applications working well.  The idea of ripping and replacing all of that effort and re-doing it with snaps and subsequent tape backups was going to be an awful hard sell.

I wanted something that was a drop-in emulation for existing tape libraries. 

Most importantly, I knew that people didn't like tape.  Give them a half-decent way to use less of it, and they'd seriously consider it.

The Test Marketing Was Positive

Now, lest you think that we were overly scientific in our test marketing, we weren't.  It consisted mostly of talking to people. 

As an example, our Executive Briefing Center in Hopkinton gets about 5-7 customer groups on an average day -- it's pretty easy to occasionally get a few minutes with a customer, present your thinking, and get their unfiltered feedback.  I find this sort of face-to-face market research much better than any other approach.

We also were able to talk to sales reps and technical presales people, and get their view.  Now, I'm not saying that everyone thought the idea was obvious, but there was a kernel of interest there from all sorts of folks -- enough to encourage us.

Andrei Comes Through

I clearly remember a few heated discussions with the CLARiiON team over our divergent views.  It was all good discussion, as we are all passionate people, especially when it comes to topics such as these.

Andrei Shishov had a group that was looking at this sort of thing, and he came up with an approach for a tape library emulation that we could integrate in front of our CLARiiON with pATA drives.

He came up with a very clever solution, and was able to do some significant value-added integration around what was generically available in the market.  And his team was able to deliver it very, very quickly.  Andrei and his team deserve major credit for seeing the potential, and coming up with something that was pretty cool at the time.

He also realized that qualification of the emulations was going to be really important.  Part of the value proposition of an EDL was "just drop it in, nothing changes, easy to configure, etc." so a lot of work went into assuring that customers had that sort of experience.

We Launch The Product in 2004

I remember working on the launch.  The idea of a disk library (as opposed to a VTL) was relatively new, and anytime you're launching something kind of new, it's extra work.  You have to explain the concept, how it's different than other things, and then justify its benefits.

I think it's safe to say that this wasn't Really Big News in the industry when we announced it.  Sure, we got a few press pickups, but almost nobody really got what we were doing here.  Lots of press focus on the higher cost of disks as opposed to tape, rather than the dramatic performance improvement, and that was that.

I, for one, was not discouraged.  I had met customers that wanted this sort of thing, so I knew we had something here -- it was just going to take a while for the story to play out.

Getting The Story Right For Customers

Sean Kinney was the product marketing manager for the Disk Library at the time, and he did exceptional work. 

His approach to communicating to customers was simple (1) focus on the real-world performance in backup and restore as compared to tape, (2) be honest about the cost differentials -- sure it was more expensive, here's how much, and (3) emphasize the drop-in nature of the product -- nothing changes in your backup environment.

Right away, we had significant interest from customers.  Now, we're a bit fortunate that we tend to sell storage to people with big, honkin' applications, so we knew who to talk to, and what kinds of problems they had.  But although customer adoption was good, it wasn't exactly explosive.

I think it was mostly giving the concept of a disk library enough time so that everyone could get comfortable with the idea of using disk as a direct replacement for tape.  And, every quarter, we ended up seeing more satisfied customers.

As Forrestor points out, we've been quietly successful since 2004, taking the #1 position by their estimation.  It's turned out to be a bigger market than most people realize.

Speed Vs. Cost

As conceived, the EDL is all about performance -- yes, backups are faster, but restores (especially incremental ones) are blazingly faster.  The product philosophy is to deliver disk levels of performance at reasonable costs.  And it does that.

There's also a powerful argument around reliability and availability, as well as simpler management, and -- not to mention -- not having to worry about tapes sloshing around the landscape, if you so choose.

With the advent of target-side data dedupe (e.g. DataDomain, NetApp, and a few others), there's another tradeoff available -- you can potentially get lower media costs through dedupe, but at some impact to performance.  The impact shows up in different places depending on exactly what part of the process you're discussing, but -- generically speaking -- there's no free lunch here.

We've done our fair share of head-to-head bakeoffs in front of customers using real-world applications, and I think it's safe to say that we're in no danger of losing our performance lead here.

But not everyone who wants to use less tape needs blazing performance.  Even a modest performance boost (as compared to tape) may be enough to justify the cost premiums associated with disk as a backup target.  So, I'd expect we'll see more flavors of these tradeoffs in the future from EMC and other vendors.

Platforms Vs. Features

One of the reasons I think that the EDL has been successful is the synergy with the rest of EMC's business.

Take interoperability testing.  EMC (through our eLab) still is the de-facto standard for multi-vendor interoperability qualification and one-throat-to-choke support.  We were able to take what we had done in the SAN world, and apply it to the more focused world of tape library emulation and backup application support.  Customers knew how we approached the topic, and we got the benefit of the doubt in this regard.

Or platform enhancements.  EDL uses either a CLARiiON or a DMX as its storage target, so when we announce a faster box, or bigger drives, or more bandwidth, or whatever -- the entire EDL line gets the benefit of that improvement.  Since our storage arrays span the range from small and cute to large and intimidating, so does the EDL line.  Basically, the EDL gets a bunch of regular platform enhancements for essentially "free" -- they're a normal part of our storage array enhancements. 

Again, hard for other vendors with smaller offerings to match this.

Other parts of the portfolio can potentially help out as well, like RSA encryption and key management.  Or NetWorker integration between the backup catalog that EDL sees (moving backup images from disk to tape) and the broader one that admins see.  Or remote replication between EDLs.

Although the EMC Disk Library product can stand on its own, it doesn't have to -- it gets the benefit of the entire EMC portfolio, and I think that's tough for others to compete with.

A Bit Of Personal Satisfaction Here

I tend to get passionate about product discussions.  I can't help it, it's part of my DNA.  Even though I don't have direct product responsibility in my current role, I just can't resist kibbitzing from time to time.

When I reflect back on the EDL, I take a bit of personal satisfaction that I was able to help convince others that there was a cool market here, one that wasn't being well served by existing offerings, and that EMC had a clean shot at offering something that was unique for its time.

And, thankfully, I was able to convince others to invest their significant time and effort in attacking a market that largely didn't exist at the time.  At the time, it was a heckuva gamble.

It's one of the reasons I love working at EMC -- we're not afraid to try something new.  Kudos to our management team for funding the experiment, because not all of them work out.

I remember declaring a small moral victory when we had reached our 100th customer. 

It's nice to see we're at 1,100+ customers, and still going strong.

Comments

[Ronald]

Chuck, the EDL does a great job but it taught me a moerse lesson. Solutions are about products, partners, people and processes. At first bump, I vastly underestimated the complexity of the solution (and I think so did EMC!)

[Chuck Hollis]

Hi Ronald -- thanks for the comment. But now you've got me interested -- what was the story for you? Thanks.

[Ronald]

My story is more like a novel. Something that was meant to take a few weeks stretched to a year. I'll post a few chapters some day over at my blog, so as not to fill yours with my diatribe. However, we did eventually nail that backup window target and never again will I underestimate the complexities in backup. :-)

[Jon Tomlinson]

We have implemented several CDL's and found it a challenge as well. We are an Outsourcer, and as such operate a medium sized environment that’s leveraged for all of our customers. Here’s my thing. The CDL works as you said. Drop it in and it works. The backup software worked just as it said in the glossy. But there was no integration. Tape is still a factor of course and now we had to find a way to efficiently move images to tape. It was a challenge. Now the new software release should ease some pain. But SUN's idea of having the VTL manage the movement to tape (unbeknownst to the software) appears to have some merit as well. All in all it was the right choice, but certainly easier for a small shop than a large one.

[Chuck Hollis]

Hi Jon -- thanks for the comments.

We're about to go down a rat hole here, so hang on.

The underlying problem is that your backup software wants to know where things are at all times, e.g which backup images live on which piece of backup media, whether that is a real tape cartridge, or an emulated one.

Everyhing works well when backup images are sent to disk, and stay there. But many people want to move those images again to tape, ideally using the disk library or VTL to do that.

VTLs provide that transparency. When the backup application asks for a tape image, the VTL says "here it is", regardless of its physical location.

Disk libraries don't work that way -- the disk library independently moves images to tape, and -- unless other steps are taken -- the backup management software doesn't know that the backup image has been moved.

EMC has done several integrations with popular backup packages to run a small media-manager on the disk library itself.

In this scenario, when it's time to move from disk to tape (or back), this can be initiated and controlled from the backup application (if desired), but -- at a minimum -- any move is communicated back to the backup management application so it always has an accurate picture of where things really are.

We've done that for Networker and a few other packages. If you're really interested, you should contact your EMC team to find out what the latest is.

And, of course, nothing is perfect. VTLs have pros and cons as well.

Thanks for writing!

[mgbrit]

Chuck, I know EMC still uses the Falconstor IP for the front-end tape emulation. EMC purchased some other tape emulation IP a couple of years ago - whatever happened to that?

[Chuck Hollis]

Hi mgbrit ...

You know, I had forgotten all about that acquisition -- can't even remember the name of the company or the technology.

I'm not close to the product team anymore, so I don't know if they're still using Falconstor as a base, or if they added some of the acquired technology, or something else entirely.

Sorry I couldn't help ...

[Ronald]

Chuck, hope your trip to Europe has been fun. I have writen my diatribe over at http://thinkingproblemmanagement.blogspot.com/2008/02/backup-window-saga-things-are-not-as.html

You comments on 22/2 are spot on, but it is not possible to go tapeless as I think the auditors and regulators would have heart attacks. Besides, being the cynic that I am, the more fail safes the better.

BTW: Please award your man in France a medal of honor.