From Lightning To Thunder

We all know that thunder closely follows a big lightning strike, and with today's unveiling of Project Lightning (now VFCache), there's thunder as well, as in Project Thunder.

To get a real appreciation for what this is all about, I'd strongly encourage you to read this background post first, and then my overview of VFCache.

Now, with all of that out of the way, let's dive into this interesting technology preview.

Making It Simple

VFCache is all about making server-based flash storage more usable, more efficient, more performant, etc.  Plenty to consider there.

But -- wait.  Step back a moment.  What's really going on here, and where will it lead?

Here's a likely scenario.  

Customer needs more performance for, say, a critical Oracle application.  Decides to use VFCache to dramatically improve performance without having to bring in an Oracle Exa-whatever box.  Likes the results, and wants to do more.

Buys some more VFCache for other apps.  And maybe some more. 

At some point, a light bulb goes off, and the customer thinks "gee, maybe I should be thinking of all that server-based flash as a shared resource for all my servers".

Err, that's exactly what happened with EMC's Symmetrix back in 1995: people realized that enterprise storage (especially the good stuff) ought to be a centrally-managed, shared and pooled resource vs. bound to specific servers.  We did it with point-to-point SCSI busses back then (!), but there ain't all that much of a difference if you think about it.

Performance-enabling storage technology will always be a scarce and comparatively expensive resource.  And you'll always want to optimize your usage of it across all potential consumers of it.  We've seen this movie before.

Will History Repeat Itself?

There's no reason to assume it won't -- at least from my perspective.

In a nutshell, Project Thunder is a pure-flash storage array, intended to be clustered using server-area network technology.  In one sense, it looks a lot like a SAN, except with different technologies.

On the hardware side, not a lot of big surprises.  

Familiar form-factors of intelligent storage modules that can be quickly and dynamically assembled into a scale-out architecture, drawing heavily from our experience with Isilon and others.

The ability to support many terabytes in a single module, and -- of course -- much, much more through aggregation of multiple modules into a single, transparent "pool".

Gobs of bandwidth, thanks to a server area network (think Infiniband, if that helps).  We haven't talked much about it, but there's been a team at EMC intently focused on RDMA optimization around storage protocols, and you'll undoubtedly see some of that work here as well.

Yes, it will be another EMC storage platform -- let's get that out of the way right now, shall we?  

But it will interoperate nicely with our more traditional FAST-enabled arrays, the new VFCache, etc.  From our perspective, it's just another storage platform -- albeit a very cool one :)

What I find important is that we'll largely be able to take our entire, massive storage ecosystem that's grown up in and around the array -- and make it available on this new platform before too long.

A variety of management models, depending on your needs.  Killer integration with VMware and other virtualization environments.  All manner of storage functionality: tiering, replication, dedupe/compression, VPLEX distance, etc. etc.  Security.  Deep integration and qualification with popular servers, databases, operating systems, applications, etc.

Project Thunder is a shared storage array.  We know how to do storage arrays.

Indeed, the mind boggles at the potential of a VCE Vblock based entirely on Project Thunder.  Folks, we'll likely have a new, incredible "lust toy" for the advanced infrastructure crowd before too long :)

Innovation At EMC?

Our detractors occasionally tried to paint EMC as old, stodgy, slow and hide-bound in our thinking.  While there's always room for cool innovation just about anywhere in our industry, I don't think that's an entirely accurate representation of what goes on at EMC behind the firewall, so to speak.

Case in point: the now-visible Project Lightning and Project Thunder.  Best of luck to any competitor -- large or small -- who decides to go head-to-head with us.

Here's what happened: we realized -- early on -- that there was more to this flash storage stuff than simply making our existing arrays go faster.  It clearly opened up entirely new ways to think about storage -- we could all easily see that.

We also realized that our existing product groups were (justifiably) mostly focused on their existing products and customers.  To get where we wanted to go quickly, we sort of needed a clean sheet of paper, but wanted to draw heavily on the extensive resources of the broader EMC.

The result was that we formed a flash storage division whose formal start date was April 1, 2011.  The team was comprised of some familiar faces at EMC in new roles, plus some fresh yet experienced faces from elsewhere in the industry.  They put their heads down and got busy, which is how we like to do things here at EMC.

It is now less than one year later, and we have the first shipping product salvo in a continuing barrage of rather distinctive technology announcements during 2012 and beyond.  In the process, we've essentially created a new storage group at EMC with their own technology, value proposition, etc. -- but with strong and recognizable family DNA.  

As we did with Atmos, VPLEX, Centera and so on.

Here's the interesting part: despite Lightning and Thunder's innovations, it's just one of many incredibly cool things we're working on here at EMC.  There's so much more to come.

And some people wonder why I like to work here so much :)

Comments

[Carlos Soares]

I find it curiously ironic that EMC chose Lightning and Thunder as project names for these products. Lightning and Thunder were the names for the Hitachi high-end and mid-range modular products that transformed Hitachi Data Systems to a storage company and an EMC competitor, as seen in the press releases below:
http://www.hds.com/corporate/press-analyst-center/press-releases/2000/gl000626c.html
http://www.hds.com/corporate/press-analyst-center/press-releases/2001/gl010123.html
I know this history well, as I named the Hitachi products.
FYI: I left HDS in 2006. Prior to that I worked at Data General's AViiON and CLARiiON divisions.

[Chuck Hollis]

Hi Carlos - too funny! I remember that now ...

I don't think anyone involved here made the connection. Good names you came up with at the time ...

Thanks!