There are two ways to bring a new technology to market.
One way is to give people a better product to do exactly what they're doing today: no process change required.
Another way is to give people a better way to do things: substantial process change required.
Meaningful breakthroughs in technology almost always involve process change. The downside is that any substantive process change takes significant time and effort.
There's no shortcut that I'm aware of.
In our world, EMC Atmos is an excellent example of innovative technology that demands both a mindset and process change. The lessons we've learned about how to go about introducing new technology (and new process!) to large audiences is instructive -- not only to other technology companies like us, but to any IT organization looking to introduce new ways of doing things into their ranks.
As technologists, we obviously have to learn to get good at getting people to actually embrace what we come up with. Otherwise, the stuff just sits on the shelf :)
The Story Of Atmos
At EMC, "cloud" is not a new concept -- it seems we've been studying and debating the impact for many, many years. One of the first -- and most heated -- debates was around our core franchise of storage: what would cloud do to storage, and how could we get ahead of the curve?
Early on, we realized that cloud -- in its most advanced state -- was fundamentally different than the storage we had in the current portfolio.
Cloud storage would be inherently geographically distributed, for example. It would ideally deal in objects wrapped in rich metadata, and not the more traditional files and blocks. It would be inherently multi-tenant and multi-policy. It would be software, not hardware. It would have to scale in ways that were hard to visualize. And more.
We made the decision to spin up a new technology group (led by Mike Feinberg) to take a clean sheet of paper and build a legacy-free storage platform for the cloud. The result was Atmos -- a storage software platform that ran on commodity hardware, purpose built for supporting cloud applications.
When we announced Atmos at EMC World in 2008, there was a collective head-scratching by most in the storage industry. Analysts generally had a hard time figuring out what the heck we were doing; competitors had a frat-boy field day lobbing insults our way.
What ultimately lay ahead was a monumental challenge: convincing people to develop hundreds of applications using the new paradigm, and creating a thriving ecosystem of compatible service providers to support them.
Now -- three years later -- I can safely say that we've achieved most of our goals.
Atmos has carved out a unique and growing segment of the rapidly expanding cloud storage marketplace, and shows no signs of slowing down anytime soon.
The Breathtaking Elegance Of Atmos
To a bona fide storage geek like me, something like Atmos is achingly elegant to behold.
Gone are the familiar file and block interfaces; instead data is handled in objects, each wrapped in rich metadata, with simple applications that use RESTful APIs to store, retrieve, query, or modify aspects.
From an architectural perspective, scale-out really means geographical scale: multiple distributed Atmos instances are far more powerful, more resilient, etc. than a single large one. Concepts such as mutli-tenancy, service definition and delivery, associated workflow and policy management are baked in vs. bolted on.
Storage administration is not what you might think: it's about provisioning and monitoring services vs. more traditional tasks. Atmos runs nicely in a VM. And so on.
In one sense, it's what storage aspires to be. To this day, there's really nothing like it, even when considering popular cloud storage services such as Amazon's. Atmos is such a sharp departure from traditional paradigms that most people don't have a convenient reference point to understand what it is, and how it's fundamentally different.
Therein lies just one of the challenges associated with introducing radically different technology.
The Challenge Of Getting People To Develop Applications
Yes, Atmos can expose a traditional file system, but that's sort of like putting a thick blanket over your high-end speakers: all the richness gets lost.
It's not that writing to an Atmos API is difficult -- it's actually much simpler than attempting to work with a file system -- it's just that it's different.
I've attached a before-and-after code example just to give you a sense of how powerful the abstraction is. The problem? It's a new way of doing things: nothing more and nothing less.
So, how do you get people to change their ways?
The first -- and most obvious -- is to remove any and all forms of friction associated with activity.
A good starting point is to create a vibrant community with code samples, working applications -- and people who are comfortable using the new tools to do great things.
Sounds easy, but it takes a while.
The persistence has paid off: the Atmos community has continued to not only thrive and grow, but attract more and more people working with the API.
A second source of potential friction to working with an API is having a back-end service you can call. Here, the team has built up a very wide range of options.
For starters, you can download the Atmos Evaluation Edition, run it in any VM that's handy, and you're off to the races.
If you're looking for something more substantial, you can set up an account with any number of Atmos partners and service providers around the globe and have even more fun if you like.
Then there's making sure there's plenty of convenient bindings from popular development environments into Atmos storage services. For example, there's pretty good integration with CloudFoundry and Spring where Atmos services are easily exposed and consumed.
Finally, there are plenty of nifty existing clients that you can download and try for yourself, including many that are back-ended by Atmos services from our partners.
You might be interested in the AtmosFox client, this Python example, or perhaps a broader view of SDKs.
What About Storage Administrators?
Atmos environments haven't figured out how to entirely eliminate the need for a storage administrator. You don't need a lot of horsepower, but you *do* need a few people who understand the environment and can look after it.
There's very little in the traditional storage administrator experience that comes forward when you've got at Atmos environment in front of you. It's not really about learning, it's more about unlearning everything you know. If you've ever switched from a PC to a Mac or iPad, you're familiar with what unlearning is all about.
However, to really learn something, (unlearn?) you've got to work with a real, live environment. The Atmos team came up with a pretty cool solution, using the EMC vLab. The team is conducting instructor-led training using a split-screen approach.
In one panel: the course content, exercises and instructor. In the other panel, a multi-instance Atmos environment running on vLab.
Instructors, participants and supporting resources can be anywhere in the, errr, cloud. Workspaces are persistent, and can be easily moved to other development or production environments if needed. Since Atmos administrator training is exceptionally easy to consume, more of it gets consumed.
Building A Partner Ecosystem
While we have a considerable number of larger enterprises that use Atmos to deliver cloud storage services internally, the real action is with service providers, which makes a certain sense, as Atmos is designed and intended to be consumed as a service.
These fall roughly into two categories: vertical service providers who have a specific application in mind, and a broader set of service providers who want to offer services to all comers.
This second category deserves some focus -- geographical coverage is incredibly important. Why?
Most people don't want their data leaving the country. Without a local provider of storage services, you're basically hobbled in developing your marketplace.
The Atmos team has done a wonderful job here, helping service providers around the globe stand up Atmos-based storage services that are easy to consume.
That same sort of global reach is also important to an important set of customers who want to deliver their information around the globe, as well -- without the huge startup costs involved in building a global delivery platform!
The numbers starting to get impressive: 30 service providers, 51 locations with over 24 PB between them. It's important to point out that the initial investment to offer an Atmos-based service is quite low: it runs in a VM on almost any hardware, and comes out-of-the-box ready to stand up various SP services.
I've heard stories of standing up ready-to-consume services in as little as a few days.
We've armed our SP partners with the same tools we use ourselves: our Atmos developer community, our training capabilities, etc. but have also added something important: a growing suite of gateways for enterprise customers who are looking to extend their internal resources with external ones.
For a regional SP partner, that represents an interesting source of demand that EMC is potentially generating on their behalf :)
Helping Our Customers
You'd be surprised at the large number of enterprises that are using e Atmos as a key part of their business. Cisco, for example :)
However, paradigm shifts aren’t always easy, especially when you’ve made a massive investment in an older one.
As an example, consider a customer I met a few months ago. They had an enormous amount of content that was being generated, managed and consumed globally. They also had a bewildering tangle of regulations to consider and comply with.
Their business was growing fast; the IT budget wasn’t.
Their existing environment was a classic beast: all the files in traditional file systems, coupled with a relational database containing all the metadata pointing to the files. I didn’t have to point out the limitations of their current setup, they did it for me.
It didn’t take more than 20 minutes for the architects to realize that Atmos was an ideal answer – quickly followed by an hour of pessimism generated from iterating all the organizational challenges associated with using it. We eventually came up with an incremental transition strategy, helped along by many of the components I’ve listed here.
I was pleased to recently hear that their conversion was going quite well, and they’re actually ahead of schedule. Just shows that you really don't know how long something is going to take until you get started :)
BTW, if by now you're interested in attending a webinar to learn more, they're given regularly. Registration link is here.
What's Next?
The answer might be obvious: better support for developing and deploying mobile applications.
The vast majority of mobile applications deal with content: capturing, sharing, distributing, etc. Indeed, the shift to "mobile first" as a preferred platform has been violent and swift.
In addition to creating the various bindings with the tools mobile developers prefer to use, there's something more we think we can do: create entirely virtualized environments of clients, servers and services for both developers and administrators.
Imagine having to develop a demanding service that (a) appears on all manner of devices, (b) needs to scale to enormous dimension, (c) will involve dozens if not hundreds of back-end service entities, and (d) need to have policy control of data due to various regulations.
Having access to an easy-consume environment where you could do all of that at one go -- that would be pretty cool, wouldn't it?
Looking Back
When Atmos was introduced in 2008, there were far more doubters than believers. That's understandable.
We deeply believed that -- eventually -- there'd be a market for cloud-optimized storage services. Maybe we were a bit too early, but we believed in the concept, and were more than willing in investing in the future, even in the face of massive headwinds.
Part of Atmos current success is market-based: the market is finally where we thought it'd be a year or so ago.
OK, so maybe we were a bit ahead of our skis.
Part of the success goes to the Atmos team hanging in there, and continuing to invest in building out the ecosystem around some very compelling and differentiated technology.
But I think the lion's share of the credit deservedly goes to EMC's exec team who saw the potential of Atmos -- and continued to invest when there were many reasons to cut losses and move on to something else, as happens so often in our industry.
If anything, the Atmos experience says a lot about EMC and who we are.
We're a company of very strong beliefs -- and we're willing to invest in them.
Comments