SOA and Information Infrastructure

IT is an industry that runs on Big Ideas. 

And, without a doubt, one of the biggest ones today is SOA -- service-oriented architectures.

But most of the discussion seems to be on why this is such a good idea for building the next generation of applications.  Not much discussion is focused on what this will mean for the infrastructure guys -- and how this will change their world in a very big way.

Today, I'm going to wade in and offer some thoughts as to why some fundamental approaches to IT infrastructure are going to need to change in a SOA world, and -- not surprisingly -- what EMC is doing to help people get ready.

So, once again, what is this SOA thing?

I tend to oversimplify, so no flames here, please. 

At a high level, it's the use of an enterprise services bus (ESB) to act as a broker between service providers (e.g. customer information database) and service consumers (e.g. a CRM application).  Without waxing too philisophical, it requires an IT organizations to precisely break down their application requirements into front-ends and back-ends.

The idea is that IT applications should be modular.  Need a new application?  It's nothing more than a composition of services presented in a new way.  Need to upgrade your back-end customer database?  Go right ahead, nothing else should be affected if you've done it right. 

It's the stuff that IT architects dream about at night -- everything modular, everything architected.

Another way to cut it is by looking at the evolution of corporate business applications. 

Phase One was monolithic applications that served a single business purpose, with users supplying any cross-domain integration (spreadsheets, reports, cut-and-paste, etc.). 

Lots of people still live in this world.  Most of the information integration burden rests on the consumers of information, and that ends up being error-prone and expensive.

Phase Two was the building of ad-hoc bridges (file transfers, API calls, web services, etc.) that did some sort of cross-application integration, but in kind of an after-the-fact, gee-we-hadn't-really thought-about-that sort of way. 

Most people live in this world, where the IT guys are building and maintaining an ever-growing portfolio of cross-domain information feeds.  If you try and upgrade any portion, most of the time is figuring out how to avoid breaking everything else.  And, yes, end users still end up spending a lot of time integrating (and verifying!) their own information.

Phase Two And A Half was the "let's put everything in a humungous database" era.  You've probably heard this one -- we'll just build a single, ginormous database and run the business off of that.  Well, most of you can probably list of a half-dozen reasons why that isn't realistic, which is -- ahem --- why very few people are seriously pursuing that course today.  Just like a branch of evolution that didn't get too far, this one died an early death.

Phase Three is SOA -- it's well-defined information and service interfaces between service providers and service consumers.  One version of "truth" by function, via multiple well-architected information sources.  The use of a services bus to coordinate traffic between providers and consumers.  New services can be snapped on to the bus with a minimum of effort.  New applications are nothing more than compositions of existing services.

Very few people live in this world, but a lot of people are moving in that direction.  Some people are thinking "best of breed", others are considering living within a vendor-provided stack (think SAP NetWeaver or Oracle Fusion), but -- even then -- you'll be living in a SOA-ish world.

I believe that SOA is a better world -- if you can get there.  There are few IT organizations I've met that can grab this bull by the horns today, and put the disciplines and methodologies in place to get to SOA over the next few years.  But you should try.

If you can get to that world, your application investments should be much easier.  Easier to enhance, easier to upgrade, easier to extend, better data quality, IT is now more responsive to new business requirements, you're free to add functionality from different vendors, and so on and so forth.  Just getting to that world from an applications perspective will be the single largest application investment that IT organizations will ever consider, but it is very strategic in nature.

But that same idealized SOA world will create an entirely new set of challenges from the folks who have to run the infrastructure.  A lot of what got put in place for the first two phases will have to be ripped and replaced to support SOA. 

And that's a very interesting discussion that's worth starting here ... sorry, IBM, but there's some heavy lifting here that's worth exploring.

Resource Management in a SOA World

Here, resource management refers to the computing resources (CPU, network, storage, etc.) that support applications.  And a pretty healthy discipline has grown up around sizing applications, monitoring their growth, and taking appropriate action. 

How does this change in a SOA world?  Well, the whole concept of "application" is up for grabs.  Everything talks to everything else.  There's very little ability to sub-segment part of the landscape in isolation, and optimize it.  You'll either need to optimize everything, or nothing.

As an example, I'd guess that resource sizing using traditional approaches will be near-impossible.  Imagine a back-end database server.  Since everyone is free to use it, there's no telling exactly how much traffic will be generated, or what the ultimate impact of slow response time might be (other than "not good"). 

One moment you've got 10 requests, the next moment you'll get 10,000.

You'll either be forced to use "worst case sizing" for servers, storage, etc. -- or, (ideally) you'd need the ability to monitor the situation, and add more crunch should you need it, and take it away if it's no longer needed.

So here are some things to think about ...

• Think dynamically re-sized VMware containers that can non-disruptively grow to address more processors and memory, and then give it back.
   
• Think service level monitoring wrapped around each side of the ESB that can detect when things aren't moving fast enough, and more performance is needed (Smarts).
   
• Think storage arrays that can accept dynamic QoS (quality of service) inputs from a performance level manager, and re-optimize storage I/O service levels on the fly.  Think of networks that can do the same thing.

So, very little of this technology exists in complete form today, but many of the elements live in EMC's portfolio.  I think they'll be very interesting, because the (ugly) alternative is to simply over-provision everything by several orders of magnitude, and that won't be too appealing.

Problem Resolution in a SOA World

So, you're living in a SOA world, and a user calls up and says their application isn't running -- "service not available". 

Let's see, this particular application calls a half-dozen other things, which in turn can call several dozen other things, and there's a whole mesh of servers, network elements, databases, storage arrays, and who knows what else.  Yuch.

Lessee.  First, check all the servers.  OK, let's check all the databases.  Now, let's check all the network devices.  Maybe it's some storage thing.  (tick tock, tick tock, tick tock).  Maybe we'll have a big conference call, and everyone can look at their monitoring screens together.

Traditional agents and frameworks (with lots of homegrown code) just won't do it.  Why?  Their referential model is standalone applications with standalone user communities.  They just aren't architected to understand the SOA model, and the fact that relationships matter more than the things themselves.  I wish you luck on this approach.  You just can't write enough script code to cover this architectural mismatch.

Maybe you're thinking you'll just monitor the ESB to figure out what's going on.  But the ESB doesn't see anything above it, or anything below it.  In reality, it's just another thingie that needs to be monitored.

You'll need two core capabilities here:

• the ability to dynamically show -- in real time -- the precise configuration of your environment, including all application relationships. 
  
  As experienced support people know, it all starts with knowing what's on the floor, and the excel spreadsheet is always out-of-date.  Call it a CMDB, call it what you will, but unless you've got a real-time view, you're hosed.
   
• the ability to correlate -- in real-time -- all the existing error and status messages using a model built with the above technology, so you can quickly isolate the real problem and fix it. 
  
  And we believe that model-based technology is the right ticket, because in a SOA world, the relationships between things matter more than the things themselves.

In the EMC technology portfolio, these products are nLayers (now Smarts Application Discovery Manager) and Smarts respectively.  They're used today primarily where you have business process that are built around applications talking to other applications -- which is a pretty good working description of SOA.

Business Continuity and Recovery in a SOA World

Most discussion around backup, recovery, replication and the like is centered around protecting important applications.  But what happens when we lose the concept of a discrete application in the SOA world? 

We have to protect all the participants in the SOA-plex, and the relationships and interactions won't be obvious, because everyone will be using everything -- that's what it's about.

Imagine if three interrelated databases are backed up to discrete, but separate recovery points, and you have to recover.  One gets recovered to 1PM, the second to 2PM, and so on.  And, of course, you've got real-time business processes that use all of them.

The term I use is "data salad" -- there's no single point of consistency.  You'll need to think about the problem very differently.

For years, EMC has been promoting the use of consistency technology ("congroups", in the vernacular) for backup, replication, etc.  The idea is that all the participants are recoverable to the exact same point in time -- there's no way that one can get ahead of the others. 

It's found a lot of traction in certain financial institutions, but I would think that the technology becomes far more interesting as we encounter SOA applications that are routinely updating multiple, synchronized databases.

Information Management in a SOA World

We've talked in other posts how IT will have to be responsible for the information it manages: cost-optimizing it, protecting it against loss and inappropriate use, creating new value from in, making sure it's retained long enough, etc.

Those requirements don't go away in a SOA world.  But, instead of having some application-specific piles of information you can tackle (e.g. email, SAP, etc.), most of the interesting information will be sitting behind some well-behaved information service (think database) on the back end of an ESB.

You'll need a capability to manage the metadata associated with each of these information elements, what are they, what do they mean, what's the sensitivity, when should it be archived, etc.

And the fact that Documentum can virtualize the metadata in existing databases and repositories should help the SOA pioneers get to the promised land, but still have a way to understand how the information should be managed in an aggregate sense.

Collaborative BPM in a SOA World

Lots of discussion around business process management (BPM) these days, and one of the advantages of SOA is that it's (relatively) easy to define new business processes by simply orchestrating existing services in new ways.  I've seen some of these demos, and it's pretty heady stuff.

But most of the BPM discussion is around transactional business processes: ordering a new widget, or processing a new customer.  The more interesting aspect of BPM is the collaborative angle, where it's a mix of transactional events, and smart people evaluating / discussing the information, and making a recommendation.

Imagine loan processing -- there's always humans involved (or should be!).  Or people trying to figure out next year's marketing plan.  Or a bunch of researchers trying to figure out what the most recent drug trials mean.

As more and more of our workforce become knowledge workers (rather than rote processors), the area of collaborative BPM is where I believe we'll see more value coming from BPM methodologies.

Clearly, I think that Documentum excels at this sort of collaborative BPM in a way that traditional BPM approaches don't try and address.  The ability to combine transactional and collaborative activities in an overall managed workflow is just the ticket for many industries, and I'm spotting new ones each week.

Back to SOA -- in a SOA environment, the "user oriented information integration" we saw so much in Phase One and Phase Two becomes collaborative BPM, in my humble opinion.  And your collaborative BPM tool ought to be very comfortable in getting information from a SOA environment, and driving additional workflow back into the SOA environment.  And, yes, Documentum does both well today.

And there's more ...

We haven't talked about test and dev environments in SOA need to be dealt with -- think about that for a moment if you want your head to hurt. 

Or how the security model will need to fundamentally change and evolve. 

Or how SOA environments will need to be bridged across corporate boundaries to bind business partners closer together, and the need for trusted third parties to authenticate transactions across boundaries.

Or probably a couple of dozen other topics I haven't really thought about yet.

So, yes, SOA is big -- really big -- but not in ways you may have imagined.  And if the infrastructure doesn't support the new demands of SOA, it's all going to come crashing down. 

I remember back to the mid 1990s when we saw the first big SAP applications come online, and the infrastructure crumbled around the new requirements. 

It happened again when corporations started going online, and -- for the first time -- exposed business processes to the outside world.

Hopefully, we can learn from our past mistakes, and help get ahead of the new SOA infrastructure requirements that are undoubtedly out there.