Leading An IT Transformation

Leading a transformational change in any large organization is not for the faint of heart.  

Even more so when the subject of said transformation is a large, entrenched IT organization delivering mission-critical services.  But that is exactly the situation so many IT leaders find themselves in these days.

At our recent IT Leadership council, Sanjay Mirchandani opened with an extended keynote detailing EMC's IT transformation: the "why", the "how" and the outcomes.  

This post is part of an extended sequence recapping all the excellent content at the event.  The back story and overview can be found here.

Why do I find this presentation particularly interesting?  

There are undoubtedly IT organizations out there that are arguably "better" than EMC's own IT.  And I know for a fact that there is a large population of IT organizations that are still running in a tradtional legacy mode and have yet to begin their journey.

What's fascinating to me about this story is the change: how a large IT organization went from one kind of IT organization to another.

Yes, EMC IT supports a large, global technology company, i.e. EMC itself.  And, yes, our situation and experiences might not be an exact match to your situation and experiences.

But, as I've shared the story over and over again, there is plenty to learn for many IT leaders who are looking for clear (and successful!) examples on how a large IT organization can fundamentally change how it does business.

To Begin With

I think it's fair to say that Sanjay is widely seen as a successful business leader at EMC.  Yes, he's worked around IT for most of his career, but had never worked within an enterprise IT organization -- that is, until he became EMC's CIO a while back.  

Hint: if you want to "run IT as a business", what kind of person would you put in charge?  

On stage and in person, he's a charismatic and dynamic character -- a archtypical "agent of change" if there ever was one.

Sanjay always starts with a two snapshots of EMC IT's footprint -- one in 2004, one as it looks in 2011.

A couple of interesting points not readily visible on this slide.  During this period from 2004 to 201l:

• EMC's revenues and # of employees have expanded dramatically
• EMC has made many dozens of acquisitions to enter new markets
• EMC has greatly expanded its business footprint outside of developed economies
• EMC IT has managed on relatively flat-to-down budgets and headcounts
• EMC IT has kept the "managed application" count relatively flat

That alone ought to qualify Sanjay's team for a Nobel Prize in IT Management :)

Yes, the EMC IT organization has access to world-class IT technology (and technologists) as part of EMC. 

But, in some sense, that rather privileged access draws even more attention to the challenges within an IT organization itself.  

For the record, EMC IT must pay for everything it consumes, just like any business.  And, unfortunately, they don't get the benefit of a senior account team to make EMC technology easy to consume.

That being said, EMC IT's challenges should look very familiar: flat-to-declining budgets, dramatic growth of information and users, aging infrastructure that's 100% allocated but not 100% utilized, critical apps that were built in the last decade, a business that's moving incredibly fast and wants their new apps right now, highly customized application stacks everywhere -- sound familiar?

Within EMC IT, they've been running five separate initiatives for several years now.  The particular focus areas might shift over time, but the initiatives themselves don't change.

They are (in no particular order):

(1) Efficiency -- currently, the focus on efficiency is centered around establishing financial transparency across the organization.

(2) Total Customer Experience -- the current focus here is creating a core of next-generation business applications

(3) Workforce Productivity -- EMC is largely comprised of knowledge professionals, so we characteristically invest in their productivity.  The big thing right now is 
mobilized collaborative and social experiences, also dubbed "workplace 3.0"

(4) Architect For The Future -- this is about transforming EMC IT to look like a cloud-powered service provider.  The current focus is fully implementing ITaaS.

(5) IT Proven -- since EMC is in the business of selling technology to progressive enterprises, EMC IT invests heavily in both proving out newer EMC technologies, as well as sharing their experiences with customers and partners.

The interesting point to me is that these five initiative categories (a) nicely reflect EMC's ongoing IT requirements, and (b) don't change much from year to year.  

I should also point out that I believe the real levers in this group are #1 and #4 -- the transformation of EMC IT to a competitive, cloud-based internal service provider with clear financial transparency.  

Do those bits well, and the other initiatives become greatly enabled.

Question: how many of you have a similar chart that stays largely the same from year to year?

The First Phase of the EMC IT Journey

By now, you should be thoroughly fatigued from seeing the three-phase chart we use to describe how most IT organizations transform.  

If you're not familiar with it, we use "degree of application virtualization" as a rough proxy to establish how much progress towards end-state has been made.

The first phase is usually when IT adopts virtualization for the workloads that IT itself controls.  The technology changes, but little else.  Most IT groups that I speak to are well through this phase, and understand the benefits -- mostly in the form of better utilization of server assets.

The steps EMC IT took at this point should look mostly familiar, save for one.

In addition to introducing server virtualization, infrastructure was consolidated, storage tiered (manually at first, and then automatically), disk-based backup was introduced and so on.  

In retrospect, the EMC IT team looks at this time as gaining experience with many of the key technologies they'd be using more critically in the near future.

The choice to standardize on x86 was a very controversial one when it was made.  Remember, this is back in 2007-2008.  Most of EMC's critical workloads were running on big SPARC boxes back then.  Yes, Intel could show a nice roadmap, but convincing the IT rank-and-file that we'd end up being an all-x86 shop going forward was not without its anxieties.

In retrospect, any decision to standardize (where possible) on a single strategic technology turns out to be huge.  

Examples in the EMC IT world include standardizing on x86, standardizing on VMware, standardizing on Cisco, and -- of course -- standardizing on EMC.  Sanjay -- as an IT leader managing a huge portfolio -- is quite vocal that the business benefits of ruthless standardization far outweigh the traditional concerns of picking a single technology or, sometimes, a single vendor.  

I've seen this happen: if you are unfortunate to start whining to Sanjay about the wisdom of having multiple competing vendors, avoiding lock-in, etc. -- it will be a very short meeting indeed.

Indeed, the uncomfortable standardization choices made here ended up paying enormous dividends down the road -- something that's hard for most people to see.

The Second Phase of the EMC IT Journey

In retrospect, just about everyone you meet is using virtualization for their non-mission-critical apps.  

Maybe they haven't been ruthless in their standardization choices as we have, but I think everyone has seen the first part of this movie.

Things got decidedly more interesting when EMC IT began to tackle the business-critical and ultimately mission-critical applications that power EMC's day-to-day business.  

In retrospect, this phase was much less about technology, and much more about people and process.

Sanjay's "highlights" slide makes this look far too easy, I think.  Yes, EMC IT swept the floor of legacy stacks, and replaced them with virtualized, tiered and shared cloud-like platforms and processes.  

Getting the IT team's capabilities to the point where this wasn't the risky proposition it might sound like was the real heavy lifting behind the scenes, which we'll cover later.

A new layer of integrated management and security was designed and deployed.  As Sanjay puts it, when it comes to management and security frameworks, it's not what you add, it's what you end up taking out that really matters.  One measure of success here were the legacy tools (and associated processes!) that could be decommissioned as the new environment went in and become productive.

Dedupe backup went in at this stage (as you might expect), and the important discussion of a next generation data center was started.

Looking back, the idea of building a "legacy-free" data center (technology and process) was now feasible -- the environment and the processes were maturing to the point where enough people could see what the near-term future would bring.  In some sense, having a hard deadline for getting in to the new data center provided a useful forcing function to accelerate multiple programs towards a converged outcome.

At a high level, this phase was all about being able to deliver enterprise-class IT services from a virtualized and shared pool of resources.  

Sanjay has his "lessons learned" that he shares freely.

The first point is rather obvious, but often missed.  The conversations outside of IT are often more important than the conversations inside of IT.  

Our IT group was investing in changing the way they do business to serve our internal customers better.  A frank and open dialogue with those same internal customers is probably the first order of business ...

Standardization and simplification were far more important than elegance or building the "perfect" solution.  A standard and simple platform that could handle 90% of the use cases was far more appealing than a custom and complex environment that could do 100%.

As we'll see in a moment, measurement presents its own challenge, because what you're measuring tends to change as you move from phase to phase.  And, no, there are no generally accepted industry standards or norms to point to when it comes to measurement metrics or methodology -- our EMC IT team had to figure a lot of this out on their own.

One important theme -- baseline your existing environment before you start on each phase.  Without that baseline, you'll never know how much progress you're making towards the end goal.

Sanjay's "virtual first" policy was seen as a bit radical when it was first established -- it's now becoming more of the norm with the progressive IT teams I work with.

And finally, be prepared to make a subtantial and ongoing investment in skills and education.  More on that later.

Sanjay did share some of the measurements they'd seen during the journey.  He's not 100% happy with what was chosen to be measured at the outset, nor is he entirely satisfied that IT measuring the right things these days.  

For example, establishing cost transparency might be thought of as a useful metric, but unless it visibly impacts IT consumption choices, you haven't accomplished your goals.  

More work to be done here, he says.

He does share one chart that illustrates the relationship between IT agility and increasing the proportion of IT value-oriented spend.

On one axis, we have the declining time to provision a service to support a new instance of an application.  You can see it going from 90 days to under a day.  On the other axis, the proportion of IT spend that's going towards new business capabilities vs. keeping the lights on.

We currently think that about 42% of every IT dollar is now going towards new capabilities vs. running existing ones.  And we think far more is possible in the near future.

The next challenge was making these resources easy to consume.

The Third Phase of the EMC IT Journey

When EMC IT began to tackle IT-as-a-service, they needed a simple statement to characterize what the real goal was.  

They ended up with "optimizing IT production for business consumption".

I've grown to like that characterization, because it shifts the focus away from "what's good for IT" clearly towards "what's good for the business".  As you might realize, they're not always the same thing :)

If EMC IT was going to compete with external IT service providers, they had to organize and operate like IT service providers.

Perhaps the deepest discussion was around the organizational strategy to achieve this.

Personal note: in my day-to-day interactions, it is the rare IT group that I see is well-organized to achieve their objectives.  Transforming to a competitive internal IT service provider is hard enough; I would argue it is damned near impossible unless you take a serious look on at key functions and how they're structured.

And this is the 'money slide" that drove all the discussion.  Let's take a quick tour -- it's very instructive I think.

Group (1) is the all-important interface to internal customers, labeled here as "go to market" group.  

There's an overall strategy around what services should be offered, a service catalog that is the instantiation of your wares to sell, and a pricing/chargeback/showback function.

This is your internal IT sales force.  Their mission is to drive the consumption of the services delivered by internal IT.  They are not asked to ration IT consumption, or argue with people around what they "really" need.  They have an inventory, and their job is to move it.

Note: while this might seem foreign and alien to most traditional IT professionals, it turns out that one of the things that successful IT service providers have that most enterprise IT organizations don't have is a go-to-market function.

Group (2) are your "product managers".  These are the people who "own the offer".  They're trying to figure out how to improve existing services to drive consumption, and offer new ones that people might want.  

Like any product (or service) manager, they're looking closely at what the sales force is doing, meeting directly with key users, figuring out costs associated with delivering an offering, benchmarking themselves against competitive alternatives, introducing new capabilities into their marketplace -- a set of disciplines well understood outside of IT, but rarely practiced within IT.

Group (3) are your "service delivery managers".  These are the people who make sure that customers and stakeholders are happy with the service delivered.  If not, they understand why that is, and hopefully what to do about it.   

They understand how the services are being consumed, can forecast medium-term aggregate demand, and can provide good metrics on quality-of-service.

Group (4) are your "financial managers".  They understand what it costs to deliver a service -- all in.  They make calls on pricing and costing policies (fully allocated or as consumed?), juggle capex and opex, keep a sharp eye on utilization, and so on.

Keep in mind that "costing" is not the same as "pricing.  Smart business people use pricing to drive positive behaviors :)

Group (5) are your "trust brokers".  We're not just talking just security and GRC (governance, risk and compliance) here -- we're talking a much broader set of discussions around availability, business continuity, being able to handle peak workloads, and so on.

What risks exist around IT service delivery, and how are those risks identified, managed and exposed directly to the stakeholders who care about such things?  Same as any enterprise-class IT service provider would have to do.

Group (6) are your "skill builders".  You just can't underestimate the continuing demand for new skills and roles in these environments, especially at the outset.  Rather than leave individual functions to fend for themselves (as is the default), EMC IT is investing in an ongoing set of competency enhancement measures as a centralized function -- leveraging new role descriptions, new certifications, cross-skill rotations, new collaboration models -- a very comprehensive approach

Finally, you need a place to do all this great stuff -- the cloud platform itself.

Now, please consider for just a moment how much real estate on this slide is around new functions and processes, and just how little is devoted to the technology itself.  You'll gain an appreciation for why I have to force myself from rolling my eyes when someone presents me with their architecture for an all-singing, all-dancing "cloud" that they want to build.

You'll also perhaps understand why I'm such an ardent fan of pre-integrated infrastructure concepts like Vblocks.  There's a lot of heavy organizational lifting to do here -- why not focus the team on the hard value-creating tasks vs. essentially re-inventing the wheel?

The specific "service buckets" the team is implementing isn't all that surprising.  EMC IT believed (as I do) that it all starts with getting the infrastructure right.  Unless you can deliver infrastructure components (e.g. storage) and infrastructure services (e.g. a resilient virtual machine) effectively, there's no higher level construct that's going to hide the underlying complexity.

Indeed, we found that the design patterns and operational flows used to do IaaS also were largely repeatable for PaaS, SaaS, et. al.

Note the next-to-top layer -- user interface as a service.  If you dig out earler versions of this chart, you'd see this labelled "desktop as a service".  

Now, virtual desktops are seen as just one choice of user experience.  "Choice commputing" refers to folks who bring Macs into the office, or perhaps our Linux-centric developers.  Mobility goes even further -- assume a fully mobilized device that might not have persistent connectivity -- and certainly not provisioned by IT!

Whether or not you like EMC IT's charts is largely irrelevant -- I'd argue that any decently-sized IT function should be able to put up a chart that looks sort of like this one -- with a clear layering of the existing or to-be-built services to be consumed by either other IT functions, or -- occasionally -- by users directly.

In communicating with business consumers of IT, we've found it very useful to use terms and concepts that they're already familiar with, "hosting" as an example.  Most people are familiar with what a hosting provider does.  

Although there are some key differences in the services provided by EMC IT vs. what you'd typically find in a garden-variety 
hosting provider, the adoption of "commercial terms" has been helpful.

From that perspective, here is how EMC IT looks as an internal hosting provider.

From left to right, EMC does a lot of R+D, so "lab as a service" (composed of multiple internal IT service elements) directly addresses the needs of EMC's many thousands of engineers around the globe.

"Cloud9" refers to our self-service, no-questions-asked IaaS offering -- only available for transient requirements under 30 days.  If you need a persistent version, or something with availability, performance, etc., there's a broader range of IaaS services that can be "contracted" for.

Next over, you'll see the "remote office bundle".  EMC does business in a lot of smaller locations around the globe: sales, support, development, etc.  EMC IT has packaged up an all-inclusive service that enables these sites to get up and running quickly with the things they're likely to need, e.g. file share, backup, etc.

There are other examples shown here, and many more that are on other "customer-facing" charts, but the over-arching principle is the same: each service is designed around the people who are consuming it, rather than the IT professionals that built it.  

The services are optimized for consumption, not production.

The other key point that might not be entirely obvious is the decoupling of the service from the underlying infrastructure.  For example, we use Vblocks extensively here at EMC IT.  A given Vblock might be used to support any or all of the service offerings -- lab-as-a-service, virtual desktops, 
advanced analytics, hosting, etc.  The front-end service might change, the back end doesn't.  

This is in sharp contrast to the more traditional practice of building individual (and prseumably optimzed) stacks for each and every service.

One particularly illustrative example is the "free" service (Cloud9).  One view is that, because it's "free", it should be built and operated as cheaply as possible.  The broader view is that Cloud9 is simply an intake function for more challenging and persistent workloads, and that the cost and complexity of standing up a completely different and purpose-built cost-optimized environment means that EMC IT will "lose" a little money on this specific service, but gain it back again in other areas.  

Price isn't always equal to cost :)

We'll get into a deeper discussion later on around the organizational and functional models EMC IT had to implement to get to ITaaS, but for now, I'd like to focus on the financial transparency aspect.

Gaining clear and unfettered financial transparency is turning out to be the big lever that moves the world.  Sanjay sees it, and I see it as well.

The ability for EMC IT to expose back realistic unit-based pricing means that -- for the first time -- business consumers can make intelligent decisions around choices.  EMC 

IT gets clear market signals as to what people want, and what they're willing to pay for.  EMC IT unit costs, quality, etc. can be compared against external options.  

Folks, for the first time, IT services at EMC are becoming an open marketplace with clear signals and feedback loops that optimize both production and consumption of IT 
services.  

All through the simple magic of clear pricing.

Hallelujah!  It's Economics 101 all over again ...

More importantly, the EMC IT team has joined up with our finance people and an external consultancy, and will shortly release what I consider a ground-breaking treatise on how to achieve financial transparency in an enterprise IT model.

 I'll make sure to make it very visible here once it's released.

When we look back at this several years from now, I betting that this discussion ends up being more important than all the technology stuff ...

The Durham Data Center

Many IT vendors are justifiably proud of their new data centers, and -- at EMC -- we're no exception.

Sanjay likes to point out that the entire, massive Durham data center runs on a single data center operating system -- Vsphere.  Vblock reference architectures are used throughout.  

The data center itself is tightly federated with other EMC IT data centers.  We're increasingly doing active-active workload balancing as we get more comfortable with the operational side of 
moving applications and data sets around.  Not suprisingly, we've got zero RPO and near-zero RTO for availability. 

Just like you read about :)

The design patterns and choice of technologies are moderately interesting, I find.  Understanding the operational processes behind all the technology I find far more interesting.  And how one goes about creating the business case to invest in a legacy-free data center -- perhaps the most fascinating of all.

What Big Data Means To EMC's Business

It should be no secret that EMC is investing feverishly in technologies and skills to empower our customers to exploit the mind-bending opportunities around big data.

That being said, big data can help EMC itself.  

And Sanjay's IT team is assuming part of the responsibility of identifying those emerging opportunities, and helping to make them competitive advantages sooner than later.

From the EMC IT perspective, big data is a very different beast indeed than traditional data warehousing and business reporting.

Sanjay uses a few illustrative charts to share "what's different".

The first are characteristics of the data itself -- familiar stuff: 

Volume -- yes, there's a lot of it.  You should be thinking in the multi-petabyte range: if not at the outset, then certainly not long, especially if you're successful.

Velocity -- the data wants to move fast.  Going from ingest to insight should be a quick as humanly possible.  Weeks and months becomes days, hours or even minutes.

Variety -- there seems to be no limit on the number of different data types people want to ingest and analyze.  Continual re-interpretation of existing data sets is becoming 
the norm as well -- same data, different lens.

Complexity -- unfortunately, there are precious few standards on how data is represented, stored, interpreted, categorized, etc.  

If that's the "macro" view of what makes "big data" data different, there's also a cooresponding value-chain view that's worthwhile.

Instead of selectively sampling internal data sources, we're now accessing not only *all* the data within the enterprise, but as much as humanly possible *outside* the 
enterprise.

Instead of analyzing what happened in the past, the entire focus is to help to predict what's likely to happen in the future -- clear scenarios with probabilistic weighting.

The corollary, of course, is the freshness of the data.  As an example, I occasionally get sent a detailed analysis of what happened in EMC's business last quarter.  I usually get this report about six weeks into the new quarter.  Frankly, these days I don't even bother to open the document -- it's just not that useful, thanks.

The final -- and perhaps most daunting challenge -- is to re-engineer core business processes to leverage real-time analytical insight to improve effectiveness.

Our first use case?  What most would call "corporate quality", what we at EMC call "TCE", or Total Customer Experience.  A while back, I wrote an extended post on "How TCE Changed EMC".  Towards the end, I hint a bit about the responsible business group taking the next step and using big data 
techniques to re-engineer our core quality processes.

From the EMC IT side, that's the "killer use case" we're using to invest in creating a new big data capability.

One of the things you'll hear from big data practitioners is to focus on big, hairy questions that can really move the needle.  Well, at EMC, customer experience and the supporting corporate quality initiatives is just such a use case.  We long have been able to precisely correlate things like NPS (net promoter scores) and long-term buying behaviors.  We get it -- at least at a macro level.

The laser-focused "use case within a use case" here is disk drive availability.  Disk drives -- at least in their current incarnation -- are mechanical devices.  They can fail.  When they fail in predictable ways and understandable ways, we're well prepared.  When they start failing in unpredictable and unfamiliar ways, that's a big deal to us and potentially our customers.  

Not that any disk drive manufacturer can occasionally "lose the recipe", but it has been known to happen.  When a particular flavor, vintage or variety of disk drive starts to fail en masse, that can be a fairly significant event at EMC.  We care greatly about detecting it early, getting to root cause as quickly as possible, figuring out who might be impacted, and immediately begin remediation efforts if required.

You don't want to wait for the Quarterly Quality Review to figure out that you might have a problem :)

EMC IT provided a big data environment that does just that -- correlates hundreds of millions of real-time data events into understandable and usable data that our quality and customer support engineers can analyze (and get alerted on!) in near-real time.

A humble starting point, to be sure, but I think it's an interesting example of how EMC is learning to master big data internally to dramatically improve some of our most critical business 
processes.

Sanjay's Final Thoughts

Like most of us here at EMC, Sanjay is incredibly optimistic about the future of IT and the IT profession in general.  

He sees that more and more business and organizations are putting information at the center of their models -- and that's where IT can play an incredibly strategic role.  Like most important things in life, getting there isn't 
easy, simply or quick -- but it's a journey that's worth embarking on.
His key takeaways reflect that longer-term view.

Cloud computing is an opportunity for IT, and not a threat as is so widely stated.  

In his world, virtualized mission-critical applications are the norm, and not something new and exotic.  He can't imagine running his IT business any other way.

He fully anticipates that -- over time -- most enteprise IT organizations will use a combination of internal and external resources to deliver services.  IT moves from being a builder of everything to a builder-broker, as he puts it.  That's hybrid cloud.

And, beyond that, he sees the next big opportunity wave in helping businesses -- including EMC, of course -- make the most of our big data resources to transform the way we 
do business.

A very optimistic picture indeed ...

Comments

[Richard Black]

Chuck, facinating post, thanks for sharing. Is there any way to print this story neatly please? Any chance you could share the actual slides? R.

[Chuck Hollis]

Hi Richard

The PDFs can be found at

http://www.emc.com/itleadershipcouncil

If you'd like to see them on your iPad, there's a neat application that can be found at

http://www.emc.com/apps/folio.htm

If you'd like the original PPTs, please drop me your email address at either @chuckhollis or chuck dot hollis at emc dot com.

As far as reformatting the material to look more like a white paper, that's a fair amount of work. I know there are people back here who are working in that direction, but I don't have any specifics.

Glad you enjoyed the content and topics!

-- Chuck

[Ken Carroll]

I forgot to add this to my post above on the 13X cost increase story. Not only did the development cost go up by an order of magnitude (i.e. 13X), but the operational and maintenance costs jumped as well. The final version had at least twice as much code, was harder to enhance, required four teams for maintenance and operation rather than two, and introduced more moving parts into production.

The team involved was forced to do this by ruthless application of mandatory standards which had been put in place by inadequately experienced people who had no knowledge the the quite important requirements & context of this important application.

It is but one of many real examples.