I'm seeing a common pattern emerge when anyone sits down to fully contemplate the widespread adoption of big data technology and techniques.
You get sucked into the mind-blowing potential. Like a space traveler wandering too close to a black hole, you get drawn in, and all of the sudden your universe appears very different.
You quickly move beyond the pragmatic: getting better answers to queries, or reducing costs, or being more profitable.
Instead, you start to see the glimmering potential to literally change the world.
And the mind can boggle.
A Brief History Of World-Changing Technology Advances?
Writing, for example, had been around for thousands of years, but things didn't really take off until printing presses became widespread. The generation, dissemination and use of information leapt foreward by orders of magnitude. Libraries, for example, became pretty important.
And humankind took a big step forward.
Consider compute. Machines that could add numbers have been around for quite some time, but it's only in the last few decades that we could pack historical supercomputer-level performance in low-power smartphones, for example.
The world is becoming a very different place as a result.
And let's not forget networks. The ability to transmit information instantaneously at great distances has been around for over a century. It's only in the last decade or so that connectivity has become ubiquitous and transparent for most of what we do.
Once again, the world is now a very different place as a result.
In once sense, the burgeoning big data era is nothing more than the logical evolution and integration of what's come before. Massive amounts of information, compute and bandwidth are the ingredients; the opportunity to exploit all of the above in entirely new ways is the new opportunity.
A Quick Survey Of Big Data Societal Potential
So much of what we do in IT is about pragmatic economic concerns: saving money, making money, competitiveness, efficiency, responsiveness, etc.
Indulge me here, as we take a brief tour of societal topics where big data is showing the bright potential of transforming our world -- above and beyond more traditional economic agendas.
We, as human beings, care greatly about our health: not only our own, but others.
Consider all aspects of health care: identifying new diseases and treatments, measuring their effectivness, delivering services in a cost-effective yet compassionate matter. Double-click a few times, and you'll see aspects of big data behind each and every aspect of health care.
The platform to eliminate many of the maladies that plague us will be built on smart people harnessing the potential of massive amounts of data. Indeed, you'll hear chatter around "health care for one", where we can individually tailor medicines and therapies for specific individuals, vs. large amorphous populations.
The more data, the more we learn. The more we learn, the better we do.
We care about our environment as well.
We'd like to be able, for example, use energy without poisoning our environment. Once again, double-click and you'll find big data behind clean energy research, optimizing transportation, or analyzing smart grids that help us use energy more efficiently, and more.
As I meet these people, I'm trying to make the time to blog their stories. They're fascinating.
We care about educating our children, not only our own, but others.
Indeed, the notion of education itself appears to being re-written. As a parent, I'm highly motivated that my children get educated in the new paradigm, and not the old one. As we take various educational assets and make them freely accessible to all, that's big data. As we understand how different educational approaches compare and contrast over large populations, that's big data.
Indeed, we may before long see "education for one", where specific learning modes are tailored to the individual, rather than forcing every kid to adapt to a standardized model. More big data.
We care about protecting ourselves from others who might do us harm.
Whether it's national security, terrorism, cybercrime, fraud or ordinary street-crime, you'll find an explosive growth of big data approaches to understand and effectively respond to threats: both traditional and newer varieties.
From video to analytics -- protecting ourselves from bad things and bad people is now clearly moving to the big data domain.
We are intensely curious about the world around us.
From exploring the structure of our universe from the smallest scale to the very largest -- there's a bottomless demand for big data to record and analyze even the faintest traces of underlying structures.
Indeed, many branches of scientific research is now built on the assumption that vast amounts of data will need to be captured, stored and analyzed. More disciplines are moving in that direction as we speak.
We, as individuals, want tailored, customized and rich experiences wherever we go.
The more the world understands us individually -- our preferences, our mindsets, our individual contexts -- the more the world can offer us tailored and customized experiences that maximize our enjoyment and minimize our frustrations.
My colleague, Scott Yara -- one of the most compelling big data visionaries -- wrote an excellent perspective on this recently. And I think he's just scratching the surface of what lies beyond.
Data Scientists Are Not Strictly Capitalists
As I meet people who are very deep into this stuff, a common pattern tends to emerge. First, they're usually extremely interesting people with some unique perspectives. Second, they're incredibly excited about what they're working on.
But probe a little bit, and you'll likely find something else as well. They often believe that big data has the potential to change the world, to make it a better place.
And I, for one, am starting to agree with them.
I can see where very large data sets and the attendant infrastructure of storage, compute, and applications require unique and creative approaches.
There are certainly unsolved (or unloved) problems because the tools required have yet to be built or have yet to be applied to the problem.
Similar to the ice age that changed the world, the big data change could follow the same timeline precisely because it is big and complex and slow moving.
What's your feeling about the pace of world change attributable to big data?
An incremental, glacial change or a decade or sooner?
Posted by: Blaine Berger | June 03, 2011 at 10:20 PM