I've now had the experience about a dozen times.
I'm in front of an IT leadership group. We get to talking about big data analytics.
They stop me and say "We get it. What the hell should we be doing?"
Fair question.
Since I haven't seen any good advice on the subject to date, I thought I'd share what I've been telling them.
Feel free to add your own thoughts in the comment sections, if you'd like.
The Dual Role Of The CIO -- Which Are You?
Is the role of the CIO (or IT organizations in general) to save money, or make money? Or perhaps a bit of both?
There's plenty of discussion around the former topic: saving money.
Life is a never-ending treadmill of improved efficiency, improved automation and improved utilization.
Indeed, one can certainly make a good career as an IT leader by simply showing people how to save money through the judicious and intelligent use of IT. Plenty of demand for that particular skill set, based on what I've observed.
But what happens when the focus shifts to making money? Identifying new sources of revenue and competitive insight? Creating entirely new capabilities for the organization that might be put to use in creative and unexpected ways? Dare I say -- innovation?
And that's the first intellectual hurdle for the CIO or any IT leader -- is this about saving money, or making money?
The discussion is a particular relevant one. Given that most IT organizations work for the CFOs -- and that most CFOs tend to focus on saving money -- your audience for this particular proposition might not be your boss.
It might be a set of stakeholders elsewhere in the business.
Understanding The Big Deal About Big Data Analytics
Here's the deal, plain and simple.
Business leaders in competitive industries are waking up to the amazing power of big data analytics and the predictive models they generate. The race is now on to build these capabilities, and begin to harness their compelling insights.
Progressive IT organizations often have a choice to make: do they simply react to the new demands when presented, or do they lead the charge?
If you're tempted to simply wait until someone comes knocking on IT's door with a nice set of requirements, consider that -- in many industries -- a completely separate IT function has been created to support this type of work. And it doesn't report to corporate IT. Or they go outside and use external IT service providers.
As one well-known example, in oil & gas, you'll typically find two distinct IT functions: upstream and downstream. In big pharma, the same sort of thing. The same is true in many large financial investment firms. The pattern is the same: they couldn't get what they needed from mainstream IT, so they did their own thing.
If instead you're biasing towards "leading the charge", here's what you need to think about.
Step #1 -- Understand How Big Data Analytics Is Different Than Traditional Reporting And Analysis
We all have data warehouses. We all have reporting tools. We all have legions of business analysts who generate reports and fill up our mailboxes.
How is this different?
While this sort of activity isn't any less important going forward, it's not big data analytics, and it's not data science.
We're talking about a new capability built on the foundations of an older one. Just like social collaboration ain't email, big data analytics isn't your father's BI.
- Traditional BI focuses on "what happened". Data science and big data analytics focuses on "what will happen".
- Traditional BI uses limited data sets, cleansed data and simple models. Big data analytics uses many diverse and uncorrelated data sets, prefers raw data and uses mind-bendingly complex predictive models.
- Traditional BI supports causation: what happened, and why do we think it happened? Big data analytics is mostly about correlation: by using multiple unrelated data sources, we've found a wonderful new insight we can't entirely explain.
Becoming a good business analyst in the traditional BI world can be accomplished by many. Becoming a good researcher in a world of big data analytics is very, very difficult indeed.
Data science is quickly becoming a unique and wonderful skill set.
Step #2 -- Go Find Your Sponsors
You're looking for a critical set of business processes that could really move the needle for the business. And you're looking for an empowered executive that is motivated to innovate and invest around those business processes. You'll need the combination of both. Having a good candidate is table stakes, if you can find two or more so much the better.
McKinsey has provided an excellent survey of where big data analytics is being used today, and in the near future.
It's a good starting point to look across your own organization, and see if there's a fit.
The pitch to the sponsors is simple: the power of big data analytics is amazing, we in IT would like to get ahead of this, and we'd like to work with you to do so.
We're not quite sure about what the long-term costs or outcomes will be, but we'd like to invest enough to find out what might be possible. We'll need your help. Are you in or out?
If you find yourself selling too hard, it might be because (a) they aren't quite ready yet, (b) you've picked the wrong people or processes, or -- potentially -- (c) they're up to something important and have decided not to engage with corporate IT resources. Ouch.
Step #3 -- Start Thinking Platform
Big data analytics doesn't thrive unless wildly diverse data sources are really easy to discover, source, manipulate, experiment with, etc.
And that's not really what your data warehouse is doing today; in this world, it's just another source of data.
Most organizations realize that they need a new kind of platform to encourage these new uses of data, something you'll hear described as BI-as-a-service, or (more properly) analytics-as-a-service.
The key difference between one of these BIaaS models is subtle yet incredibly important: it's all about making data easy to consume and experiment with. It's built around the analytic user's needs, and not IT's traditional concerns.
That turns out to be harder than it looks in most situations.
I've included a graphic showing how EMC IT is constructing its BI-as-a-service capability. Note the top-level services -- discovery, visualization, collaboration. Those are the aspects that are most important to the consumers of the service. And it's not a traditional focus in most data warehouse or business reporting environments.
Hint: the EMC IT team has a natural advantage here: they've already reconstructed IT production and delivery to look more like a competitive IT service provider, and less like a traditional IT organization. We also have advanced techonology in-house (e.g. Greenplum UAP) so we didn't have to spend a lot of time debating that aspect :)
For them, they could think in terms of "another service, constructed from existing ones" vs. standing up yet another functional silo within IT.
Step #4 -- You'll Need A Few Magicians
Much has been written about data science and data scientists -- who they are, what they do, and how they're very different from traditional business analysts.
At some point, you'll need access to these rather rare and precious skills, preferrably after (a) there are some interesting business questions to be answered, (b) the data and resources are really easy to get to via your platform, and (c) there's a motivation to actually do something with the insights they inevitably find.
Don't assume have to hire these people as employees -- although we're finding having a few on staff is an incredibly useful thing. Indeed, some of the most amazing stories in data science comes from people who have absolutely no background whatsoever in whatever the topic might be.
They just let the data do the talking :)
Step #5 -- Don't Forget Chargeback
Anecdotal evidence points strongly that these platforms often turn out to be many orders of magnitude more popular than anyone thought. Any resources assigned to the project tend to be instantly and permanently oversubscribed.
Extreme frustration by the business community inevitably results.
Even if you're not doing chargeback in your broader IT environment, consider it a mandatory for BI-as-a-service or analytics-as-a-service. Otherwise, you'll end up rationing the service to interested and motivated users, and that's not a good place to be in. There has to be a funding construct baked into your thinking sooner, rather than later.
Step #6 -- Create A New Governance Function -- And Be Prepared To Use It
If you think about it, in this model you're using information in entirely new ways. You're sourcing it from unexpected places, combining it in interesting ways, and creating powerful insights that are both extremely useful and extremely ticklish.
The focus is different as well. We're accustomed to looking at individual data sets, and not what can happen when they're combined in new and interesting ways.
Saltpeter is pretty innocuous stuff. So is charcoal, and sulfur. Combine them in the right proportion, add a spark -- and you've got a nice little explosion on your hands.
Whatever you're doing today in data governance or information management won't be suited to these new behaviors and use cases. Recognize the challenge up front, and get ahead of it.
A good, lightweight governance function doesn't slow anything down; instead, I've seen many situations where it accelerates adoption as it creates the broad confidence to do things in new ways, knowing that someone is looking out for you.
Step #7 -- Be Prepared To Invest In Learning
Based on the cases I've studies, these exercises end up being prolonged learning experiences for everyone involved: the IT team, the business team, the executive team, and so on.
There's no way to exactly predict how things will turn out. There will be iteration upon iteration. Mistakes will inevitably be made, and valuable lessons learned.
Make sure that everyone on the extended team understands that this is a journey, and not a fixed project deliverable. And don't fall into the ROI trap.
Are You Up For It?
Maybe yes, maybe no. Everyone's situation is different.
But -- as a leader of your company -- I think you owe it to yourself and your team to ask the question: is this something we should be investing in?
And, if you do a bit of homework, and decide "no, that's not for us quite yet" -- I think we've all done our duty. Next topic, please.
But maybe -- just maybe -- you'll find the conditions ripe in your organization for a strategic investment in big data analytics proficiency.
In some ways, usage of such pattern analysis in predicting the future seems to be dangerously close to the oft derogated realm of 'astrology'. Have they also not been finding co-relation between the position of planets and events that could unfold in the future? Before u can get up to say "data mining is more about mining of relevant data", be sure u have checked the hemlines......headline were made several months ago, between their length and economic prosperity of the nation/world.....may be it is time to checkout positions of Saturn/Jupiter...and their kind? If it is all about patterns and one gets wiser after a rationale is 'made apparent'....may be this business of astrology can possibly improve with such modern data-mining techniques? Co-relation between seemingly diverse events seems to be a subject on the horizon?
regards
sudhir.brahma@gmail.com
Posted by: sudhir.brahma@gmail.com | June 09, 2012 at 10:08 PM