The pattern is familiar: take any critical business question, and use big data analytics to come up with better predictive models. The effect seems near-universal across a dizzying array of disciplines: marketing, customer service, finance, manufacturing -- and even IT.
What amazes me is that more IT groups aren't hiving off a small pool of resources to start to learn to apply big data analytics to *their* important business questions: not only for their own benefit, but to gain the domain expertise that the business will inevitable be asking for.
It's not expensive. It's not particularly difficult. The handful of people I meet who are actually doing some of this are finding the glimmerings of great potential, and appear to be having great fun in the process.
And there is obviously no shortage of interesting questions that need to be answered in the IT world ...
Life In IT Ain't Perfect
Certainly life in the IT world would be great if (a) requirements were well-understood and never ever changed, and (b) things worked predictably. Pay raises for everyone would be nice, too ... but none of that is likely to happen soon.
In the meantime, we're faced with the need to predict the unpredictable: surprise requirements, new threats and risks, things breaking in unforeseen ways, etc.
Plenty of high-stakes business processes are to be found in the IT function, with all sorts of interesting business questions to be asked in virtually every IT discipline: operations, security, forecasting, internal customer demand and more.
Or, if you prefer, look at it this way: (a) IT aspires to run like a business, (b) progressive business people are rapidly adopting big data analytics -- ergo, it is inevitable to me that progressive IT organizations will eventually use these techniques in almost all aspects of their operations.
Just like any competitive business.
The Data Is At Hand
Now, let's come at it from a different direction -- how much operational data does IT potentially have access to? Think about your environment in its totality: all the logs, counters, traces, emails, feedback forms, web access logs -- easily hundreds of terabytes or perhaps hundreds of petabytes.
All rich, all diverse -- and all uncorrelated: a virtual treasure trove of raw data just waiting to be exploited.
Have We Reached The Limits Of Traditional Approaches?
Most businesses rely heavily on packaged applications. For the stuff that's unique, you integrate and roll your own -- that's where the unique business know-how resides. Now harvest all those data sources, add more external data -- and even more value is created.
Let's apply that same model to the IT function. Most everyone relies on packaged tools and frameworks. For the stuff that's unique to your operation, you integrate and roll your own -- that's where the unique IT/business knowledge resides.
What's now left to do is harvest all those data sources, apply analytics -- and move to the next level.
Some Quick Examples From My Travels
There's a small team within EMC IT that's running an interesting experiment: can big data analytics predict potential app delivery issues better than a traditional approach? The experiment is being run against our internal Exchange farm -- well architected and well monitored using the best available approaches.
But still problems are encountered, and being denied access to instantaneous email services on any device of our choosing -- well, the angry mob forms very quickly here.
The big data approach comes at it differently. Gather and monitor as much "fingerprint" data as possible: counters, logs, traces -- everything you can get your hands on. Use pattern matching to discern when the observed fingerprint is starting to meaningfully deviate from the predicted one, and raise an alert.
Note that the goal here is simple correlation -- causation is left for later.
Even with the limited data sources and simple models being used, the approach is already doing a better job of raising a meaningful alert well before there's any traditional alert raised through existing means. More effort is required, but there's something obviously there.
Another example comes from the security world. Most security teams are sitting on vast mountains of potentially relevant data -- but how to make sense of it in a timely manner? Sure, there are great packaged solutions that gather, analyze and make recommendations -- but they tend to look like point solutions compared to the challenges at hand.
Can more progress be made by creating a number of predictive analytical apps using a big data approach?
Our in-house security team is testing that hypothesis, and coming up with capabilities that can't be found in the marketplace -- yet. Our business is somewhat unique, which means our security requirements are somewhat unique. Packaged solutions and platforms can only get you so far ...
A third example? At the end of this year, many IT groups will have to forecast how much capacity they'll need next year: compute, storage, networking.
The traditional methodology (have a hunch, provision a bunch) only gets you so far. Sure, you can make some pretty graphs showing historicals, but there's a potentially better approach in mashing up diverse data sources: economic factors, business forecasts, proliferation of mobile devices, utilization rates, etc. -- and backtesting to previous forecasting exercises.
And I've met a few people who have started to do just that -- and getting far better at capacity forecasting as a result.
The data is there -- do we know how to use it?
I bet my very smart readers could share another dozen examples from the IT realm where better predictive models would make a huge impact on IT operations. The technology is there, the data is there ... and the problems are certainly there!
Making Matters Worse ...
One of the traditional roles of many IT groups is to be "first in" on whatever new technology is coming around: social, mobile, videoconferencing, etc. The idea is that the IT group gains experience to help the rest of the organization exploit the technology effectively.
Or maybe it's an excuse to buy the latest toys :)
I think it's a given that a majority of businesses will be exploiting big data analytics over the next few years -- if they aren't already doing it on their own without IT. Being able to go to the business and say "hey, we've invested in learning about how all this stuff really works, and here's what we can bring to the table" -- well, that wouldn't be a bad thing.
We're not talking thousand-node clusters and petascale data farms, either. Many of these experiments are being run on quite modest environments: limited data, limited compute. And if you're a VMware customer, getting started is almost free.
What's required are a few interesting use cases, and a few people working on it. Not a huge investment in the big scheme of things -- but certainly a strategic and innovative one.
Big Data Changes Everything
That's link bait, I know it. It should read "Big Answers Change Everything".
Progressive business leaders everywhere have woken up to the potential at hand, and the race is on to transform their business using the amazing power of predictive analytics.
Why should IT be left out from doing the same?