Information Infrastructure for SOA

I'm called on frequently to speak about a variety of topics, and -- recently -- there's been a renewed interest in infrastructure associated with SOA.

You probably know about SOA -- service-oriented architectures.  It's a (relatively) new way of thinking about how applications are designed, developed and deployed.  .

But, like anything else, anything big and hairy (like SOA) has serious infrastructure implications. 

And SOA -- at scale -- is no exception.

You Do Know About SOA, Don't You?

Rather than drag you through an impossibly over-simplified description of SOA (what it is, how it works, why it's important, etc.), I'll simply refer you to this nice Wikipedia writeup.

It's not simply a buzzword anymore.  People are really doing it, whether through their own internal application development, or through integrated suites such as SAP's Netweaver, or Oracle's Fusion.

We're seeing more than a few large-scale SOA applications make it into production.  And, of course, as a result, we're being asked to address some interesting new infrastructure issues.

The Nature Of The Beast

In a SOA world, the concept of a traditional, monolithic application simply disappears.  Business processes are composed of application services, which run in a variety of places and serve multiple masters.  Not to over-simplify, but the phrase "everything talks to everything" captures the spirit of the discussion.

So, as a result, fundamental ways of approaching infrastructure in the monolithic world just aren't working as well as people would like.

Let me offer you a survey of seven common issues we hear about, and what we're doing about them.

Issue 1:  Development and Test Environments

I think it's safe to say that developing and testing integrated SOA applications is different -- and more difficult -- than traditional ones.

Sure, the individual application elements might be easier to develop and test, but then there's that big, hairy end-to-end testing where it all comes together in one gigantic beast.

You're probably aware of the benefits that VMware's Lab Manager brings to development environments.  Developers can easily create virtual machine test beds to do their work.  Lab Manager bring higher-level process and object control to the basic capability of creating virtual machines for test and dev.

In a SOA environment, we're seeing more people give serious thoughts to their test beds -- they're putting in entire environments of VMware Lab Manager, file-oriented storage such as EMC's Celerra, and creating a complex, multi-instance environment that's not only far faster to set up and tear down, but uses a fraction of the physical hardware that would be needed without virtualization.

Once again, most people are attracted to the "less hardware" part of the equation, but the real value is developers who can get their work done in a fraction of the time, build better complex test harnesses, etc. 

Getting to good happens will likely happen far faster using these virtualized test beds.

Issue 2:  Agentless Discovery (and Compliance)

Imagine you've just been handed a project, titled "Move Our Company To SOA".   Lucky you.

One subtle aspect of this journey will be the need for discovery at every step of the journey.

Let's start at the beginning -- do you have a 100% accurate map of all your application information flows?  A complete inventory (with versions and config information) of all your servers, operating systems, databases, networks, storage, etc.?   Can someone tell you how it all plugs together, and how all the different pieces interact and support each other?

If you're like most people, the answer is probably "no". 

But I'd offer that you're facing an enormous expense (and risk) if you don't have a near-100% clean picture of that before you start.  And that right there is a big, expensive problem.

And then there's configuration drift.  Any project this size will take a while.  And, of course, the environment is being modified while you're working on your SOA beast.  New applications are added.  Versions and configurations change.  New information flows are created. 

How do you keep an eye on the old beast while you're working on the new beast?

The same problem exists in the new SOA environment as well -- is everyone adhering to the guidelines?  Following change control for introduction of new components, versions or configurations? 

Finally, how do you ensure that your new SOA environment is compliant with the policies and procedures you've defined?  Part of the beauty of SOA is that things can be plugged in fairly easily, which leads to lots of different people doing exactly that.

Being able to have a near-real-time view of exactly what you have in your old environment, and your new environment, spot changes that have been made, and to do so in a non-invasive, agentless manner I would think would be essentially a life-saver, especially for large, hairy, never-ending migrations.

I think that's why we're seeing so much Smarts ADM go into these kinds of environments.  Not only does it do a good job of helping with these problems, it builds a model of the IT resources and applications that's very useful in the next topic, which is ...

Issue 3:  End-to-end Service Delivery

So, when a user calls today and says "something's not working in the business process", what do you do today?  You go check a bunch of stuff, and try to figure out where the problem is.  Networks?  Databases?  Servers?  Storages?  Applications?  Desktops?   Lots of places to go look, right?

Now, imagine that problem, only multiplied by an order of magnitude in a SOA environment.  You've built these elegant business processes, composed of abstracted services, that run in a flexible manner.  The only problem is, you don't really understand how it's all connected, or how to quickly find a problem if you need to.

Or if anyone has changed anything without telling anyone …

Enter model-based management.

As an example, Smarts can detect the application traffic, and progressively map the underlying supporting logical and physical components, and how they relate.  This helps in two ways: point at a business process, and you can see all the actors that support it, or pick an arbitrary component (e.g. a database) and you can see everything that uses it, and everything it depends on.

It uses the configuration information gathered through discovery to build a very sophisticated model, and then can work with existing monitoring tools to correlate their inputs, and tell you exactly where the problem might be.

Now, this is nice enough in your ordinary spaghetti traditional environment, but in a SOA environment, I’d offer you're going to need something very much like this.

Your business users probably won't tolerate worse service levels in the new SOA environment, will they?

Issue 4:  Backup, Recovery and Information Protection

In the one-application / one-database / one-storage-array world, these issues were pretty well understood.

You quiesced or froze the database.  You backed it up (or replicated it) so it was consistent at a point in time.  And if you didn't do that, you weren't quite sure what you might have when you needed to do a restore.

Well, it can be a bit more complicated in the SOA world.  A transactional update might involve multiple databases, and multiple servers and multiple storage arrays.  All woven together in an interrelated business transaction.

How do you get a consistent view of all these images at a single point in time, so you can back it up or replicate it, and know that you'll be able to recover so all your databases aren't out of whack?  Imagine if you had to do a recovery, and all the databases were a bit out-of-sync with each other? 

Imagine the havoc created.

It's a difficult technical problem, but -- fortunately -- one that EMC addressed quite a while back.  It first cropped up on mainframes years ago (everything happens first on a mainframe, doesn't it?), but the general techniques work well for distributed servers, and – especially – SOA.

EMC's replication products support a feature called "consistency groups".  You define a set of databases that have to be logically consistent at all times.  When it comes time to snap, copy, etc. you use the array features to "freeze" a point-in-time image (across multiple storage arrays if need be), do your copy, and let everyone proceed.  It sounds complicated, but it's really pretty easy.   All the relevant information gets protected at the same point-in-time.

And, surprisingly, no one else appears to have solved this particular problem as cleanly.

Issue 5:  Virtualized Resources

Capacity planning is very difficult in the traditional application world.  Now, consider if you will, the challenge of capacity planning in the SOA world.

You've got database and application servers everywhere.  They're used by lots and lots of different application and business processes.  If you're trying to figure out "how much stuff do I need" at the back end, it's going to be a whole lot of guessing.

Generally speaking, people tend to be conservative in sizing these things.  So, I've tended to notice a preponderance of overconfigured servers, networks, storage, etc., just in case.

Or, conversely, no one really had a clear idea on just how busy this particular server / database / storage would be, and they've got a different kind of problem to face.

Virtualization technology -- in different forms -- can make a big impact in certain areas.

A good starting point is ESX 3.5 and the new features in DRS that can do load levelling.  It creates a pool of virtual server  images, and if you're not getting the service level you need from one virtual server or another, it can move things around (CPU and memory) to make the situation better -- non-disruptively.

The same general capability exists with SAN-based virtualization (e.g. EMC's Invista) that can do non-disruptive moves across  arrays (EMC and otherwise) if needed.  There's also features within the array to do service level tiering as well, e.g. CLARiiON's QoS manager among others.

It should be pointed out, though, that invoking these load-levelling features is not fully automated across the board.  It'd be nice to set some overall performance objectives across the distributed, virtualized environment for specific business processes, and have all the players automatically compensate for shifting workloads, but we don't live in that world -- yet.  But soon, I think.

Finally, EMC storage supports the notion of virtual provisioning, where you can allocate large logical storage capacity, but not physically consume it until it's needed.

Issue 6:  Information Security

Many traditional security models break in the SOA world. 

As an example, if every database / application has its own ACL, that won't be very conducive to programmatic access from higher level composed application services.  Similarly, if your model is "hey, they logged into the network, so they're OK", that won't meet the needs of having some information a bit more secure than others.

We think that the new model will require a more enforcement points (endpoint, network, ESB, application, database, information), a centralized means of defining which users/applications can get to which database records/information under which circumstances, all wrapped in an auditing and reporting capability.

And, not surprisingly, big portions of the RSA portfolio address these needs. 

From different flavors of authentication, to encryption and key management, to information rights management and data loss prevention, and security event information management -- the components are architecturally designed to work together to create the new security paradigm required for SOA.

From the conversations I've had with customers, I have to admit that they're not thinking much about security in their new SOA world.  Maybe they see it as too hard, or are worried about slowing down their project. 

I'd offer that -- with a bit of forethought -- they could end up with a security architecture for SOA that's much more transparent,  and doesn't get in the way of running or growing the business.

Issue 7:  Enterprise Content and Collaborative Workflows

I've sat through more than a few customer presentations of what they're doing with SOA.  And, almost always, I'm seeing the same thing.  Transactional information.  Rigid, linear workflows.

Sure, you need to start somewhere, but here's what I usually offer as food for thought:

First, more and more of the high-value information we're seeing is not transactional -- it's content.  Scanned images, emails, documents, voice, video, reports, spreadsheets, etc.  Just thinking in terms of transactional, database-oriented information probably isn't a 100% accurate view of all the information that needs to participate in a SOA environment, if not now, well, certainly in the future. 

Even relatively unexciting processes like claims processing, or customer service, or human resources have an amazing amount of content associated with them.

Second, most of the workflows I see described look like rigid flowcharts.  That's a bit out-of-sync with reality, I think.

Look, we're all becoming knowledge workers.  We need workflows that include concepts like discussion, review, collaboration, consensus, etc. -- captured and described as part of the SOA environment.

Once you start thinking about the needs of knowledge workers, other requirements pop up, like enterprise search, personal information portals, and the like.

I've seen that Documentum (and all of its components, presented as SOA services) can bring an enormous amount of value to most SOA environments by addressing the non-transactional, non-structured-data, non-linear-workflow part of the problem. 

And help address the needs of your high-value knowledge workers.

Information Infrastructure for SOA

Building and deploying SOA applications is no walk in the park. 

It's challenging, and it takes a long time.  Nevertheless, the benefits are so compelling that more people than ever before are moving in that direction with a serious, strategic intent.

I think the parallel set of topics -- information infrastructure for SOA -- has to be part of the discussion.

• A good information infrastructure should help in migrating to SOA.
   
• A good information infrastructure should help to keep SOA processes running smoothly with a minimum of fuss.
   
• A good information infrastructure should allow flexible change to its components without disrupting the bigger picture.
   
• A good information infrastructure should be able to effectively backup and recover its information in a consistent fashion.
   
• A good information infrastructure should efficiently use resources, and react dynamically to shifting workloads.
   
• A good information infrastructure should have security designed in, and not added as an afterthought.
   
• A good information infrastructureshould be able to handle all types of information, all types of workflows, and give some thought to the knowledge worker, in addition to the transactional worker.

But I think that the most telling observation is that – when discussing SOA – the infrastructure guys usually aren’t in the room. 

And, well, maybe they should be.

What do you think?

Comments

[Ronald]

Methinks you understate the biggger and hairy legacy information mindfield. Legacy is "layers upon layers of technology on top of technology are the end result, and an architecture that is inflexible, static, fragile, and thus difficult to change along with the business requirements. This is the norm, not the exception."

The above quote comes from http://www.zapthink.com/news.html?id=2518 Worth a read!

[Chuck Hollis]

Hi Ronald -- couldn't agree more.

My favorite quip is "we shouldn't call it IT architecture, a better term is archaeology -- layers and layers of historical remnants, with new roads built on top."

[Alex]

Thanks for interesting clause!
You can specially write for the Russian readers about SOA?
(On a site or in magazine)
Are always open for cooperation.

About ERPNEWS http://erpnews.ru/doc2824.html